Medicine identification method, checking method, medicine identification device and electronic equipment

Abstract

The invention provides a medicine identification method, a checking method, a medicine identification device and electronic equipment. The method comprises the following steps: acquiring an image sequence in a prescription filling process through a wearable image acquisition device; inputting each frame of image in the image sequence into a pre-trained action recognition model and a pre-trained medicine recognition model to correspondingly obtain a hand detection frame, a medicine detection frame and a medicine category; determining a target action and starting time and ending time of the target action based on the hand detection frame and the medicine detection frame, wherein the target action is an action of filling the medicine by hand; and determining a final drug category identification result based on the drug category identified by the image sequence between the starting time and the ending time of the target action. According to the invention, the target action of the hand-filling medicine is identified according to the action identification, and the identification result of the medicine category is further determined according to the target action, so that the automatic identification of the medicine category is realized, the efficiency of medicine identification is improved, and the accuracy is improved.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a drug identification method, a verification method, a drug identification device and an electronic device. Background Art

[0002] When dispensing medicines in the hospital pharmacy or drugstore, the staff may make mistakes in dispensing medicines, and fail to ensure the safety of the patients' medications. To avoid this, it is necessary to identify and verify the types of medicines dispensed, such as the names of the medicines.

[0003] However, existing drug identification and verification are all handled manually, which is not only inefficient but also prone to errors. Summary of the invention

[0004] In view of this, an object of the present invention is to provide a drug identification method, a verification method, a drug identification device and an electronic device to improve the efficiency and accuracy of drug identification.

[0005] In a first aspect, a drug identification method is provided, the method comprising:

[0006] The image sequence during the medicine picking process is acquired through a wearable image acquisition device;

[0007] Input each frame of the image sequence into the pre-trained action recognition model and the drug recognition model to obtain the hand detection frame, the drug detection frame and the drug category; the drug category includes the drug name category and the drug packaging box category;

[0008] Determine the target action and the start time and end time of the target action based on the hand detection frame and the drug detection frame, where the target action is the action of grabbing the drug with the hand;

[0009] The final drug category recognition result is determined based on the drug category recognized by the image sequence between the start time and the end time of the target action.

[0010] Optionally, before determining the target action and the start time and end time of the target action based on the hand detection frame and the drug detection frame, the method further includes:

[0011] Input the hand detection frame into the target tracking model to obtain the hand tracking sequence;

[0012] When the number of hand tracking sequences is greater than a preset length threshold, the subsequent steps are performed.

[0013] Optionally, determining the target action and the start time and end time of the target action based on the hand detection frame and the drug detection frame includes:

[0014] Obtaining the coordinate information of the hand detection frame corresponding to the last sequence in the hand tracking sequence and the coordinate information of the drug detection frame located in the same frame image as the hand detection frame;

[0015] Determine an overlapping area of ​​the hand detection frame and the drug detection frame based on the coordinate information of the hand detection frame and the drug detection frame corresponding to the last sequence;

[0016] Determine the ratio of the overlapping area to the area of ​​the target detection frame, where the target detection frame is the smaller detection frame between the hand detection frame and the drug detection frame in the same frame image;

[0017] When the ratio of the overlapping area to the area of ​​the target detection frame is greater than a preset ratio threshold, it is determined that the target action is detected; and the timestamp of the image corresponding to the first sequence in the hand tracking sequence is determined as the start time of the target action, and the timestamp of the image corresponding to the last sequence in the hand tracking sequence is determined as the end time of the target action.

[0018] Optionally, before obtaining the coordinate information of the hand detection frame corresponding to the last sequence in the current hand tracking sequence and the coordinate information of the drug detection frame located in the same frame image as the hand detection frame, the method further includes:

[0019] Starting from the second sequence of the hand tracking sequence, determine the size change of the hand detection frame corresponding to each sequence relative to the hand detection frame corresponding to the previous sequence in turn; the size change includes three situations: change, decrease and remain unchanged;

[0020] When the size changes of a consecutive preset number of hand detection frames relative to the previous hand detection frame are all increased, and the increase values ​​are all greater than the preset change threshold, the subsequent steps are executed.

[0021] Optionally, the drug recognition model includes a target recognition sub-model and a feature vector extraction sub-model; inputting each frame of the image sequence into the pre-trained drug recognition model to obtain a drug detection frame and a drug category includes:

[0022] Input each frame of the image sequence into the pre-trained target recognition sub-model to obtain the drug area image;

[0023] Inputting the drug region image into the pre-trained feature vector extraction sub-model to obtain the drug feature vector;

[0024] Based on the similarity matching algorithm, the drug category with the highest similarity match with the drug feature vector is searched in the pre-built drug feature database.

[0025] Optionally, determining the final drug category recognition result based on the drug category recognized by the image sequence between the start time and the end time of the target action includes:

[0026] Obtain the drug category identified by the image sequence between the start time and the end time of the target action;

[0027] Based on the voting mechanism, the drug category that appears most frequently between the start time and the end time of the target action is selected and used as the final drug category recognition result.

[0028] In a second aspect, a drug verification method is provided, the method comprising:

[0029] Identify drug names on prescriptions based on OCR technology;

[0030] Identify the captured drug category based on any one of the methods described in the first aspect, where the drug category includes the drug name;

[0031] Match the captured drug name with the drug name on the prescription;

[0032] If the match is successful, the first instruction is returned, otherwise the second instruction is returned; the first instruction indicates that the medicine is dispensed successfully, and the second instruction indicates that an alarm is issued.

[0033] In a third aspect, a drug identification device is provided, the device comprising:

[0034] An acquisition unit, used for acquiring an image sequence in the medicine picking process through a wearable image acquisition device;

[0035] The recognition unit is used to input each frame of the image sequence into the pre-trained action recognition model and the drug recognition model to obtain a hand detection frame, a drug detection frame and a drug category; the drug category includes a drug name category and a drug packaging box category;

[0036] A first determination unit is used to determine a target action and a start time and an end time of the target action based on a hand detection frame and a drug detection frame, wherein the target action is an action of grabbing a drug by hand;

[0037] The second determination unit is used to determine a final drug category recognition result based on the drug category recognized by the image sequence between the start time and the end time of the target action.

[0038] In a fourth aspect, an electronic device is provided, comprising a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus;

[0039] Memory, used to store computer programs;

[0040] The processor is used to implement the method steps described in any one of the first aspect or the second aspect when executing the program stored in the memory.

[0041] According to a fifth aspect, a computer-readable storage medium is provided, wherein a computer program is stored in the computer-readable storage medium, and when the computer program is executed by a processor, the method steps described in any one of the first aspect or the second aspect are implemented.

[0042] The present invention provides a drug identification method, a verification method, a drug identification device and an electronic device. The method acquires an image sequence in the process of picking up medicines through a wearable image acquisition device; inputs each frame of the image sequence into a pre-trained action recognition model and a drug recognition model to obtain a hand detection frame, a drug detection frame and a drug category; determines a target action and the start time and end time of the target action based on the hand detection frame and the drug detection frame; and determines the final drug category recognition result based on the drug category recognized by the image sequence between the start time and the end time of the target action. The present invention identifies the target action of picking up medicines by hand based on action recognition, and further determines the recognition result of the drug category based on the target action, thereby realizing automatic recognition of drug categories, which not only improves the efficiency of drug recognition, but also improves the accuracy.

[0043] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, preferred embodiments are given below and described in detail with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments are briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without creative work.

[0045] Figure 1 A flow chart of a drug identification method provided by an embodiment of the present invention is shown;

[0046] Figure 2 A flow chart of a drug verification method provided by an embodiment of the present invention is shown;

[0047] Figure 3 A specific implementation flow chart of a drug verification method provided by an embodiment of the present invention is shown;

[0048] Figure 4 A schematic diagram showing the structure of a drug identification device provided by an embodiment of the present invention is shown;

[0049] Figure 5 A schematic structural diagram of an electronic device provided by an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0050] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. The components of the embodiments of the present invention generally described and shown in the drawings here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present invention provided in the drawings is not intended to limit the scope of the claimed invention, but merely represents selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without making creative work belong to the scope of protection of the present invention.

[0051] Considering that the existing drug identification and verification are all handled manually, which is not only inefficient but also prone to errors, based on this, an embodiment of the present invention provides a drug identification method and device, which are described below through embodiments.

[0052] The present invention provides a method for identifying a drug. Figure 1 As shown, the method comprises the following steps:

[0053] Step S101: acquiring an image sequence during the medicine picking process through a wearable image acquisition device.

[0054] In an embodiment of the present invention, the wearable image acquisition device is worn by the person picking up medicines, and the wearing style is at least one of the following: head-mounted, neck-hanging, and binocular.

[0055] Step S102: input each frame of the image sequence into the pre-trained action recognition model and the drug recognition model to obtain a hand detection frame, a drug detection frame and a drug category.

[0056] Drug categories include drug name categories and drug packaging box categories.

[0057] In an embodiment of the present invention, the action recognition model can adopt a deep learning-based model, such as YOLO, SSD, Faster R-CNN, etc. to identify and locate the hand area in the image and output a hand detection frame.

[0058] In a feasible implementation, the drug recognition model can also use a deep learning-based detection model to identify drugs in an image, and output a drug detection frame and the specific name category and packaging box category of the drug. In an example, the packaging box category includes boxed, transparent bottled, opaque bottled, tubular, and plastic belt categories, etc., which are used to assist in identification and play a role in rapid screening and filtering.

[0059] The drug recognition model uses an end-to-end approach. After inputting an image, it directly outputs the drug detection frame and drug category. This end-to-end approach allows for fast and efficient drug recognition.

[0060] In addition, in order to improve the recognition ability of the drug recognition model, some easily confused drugs can be collected as training samples when training the drug recognition model.

[0061] Although the above end-to-end method has high recognition efficiency, the total number of recognized drug categories is fixed. If a new drug category is added, the drug recognition model needs to be retrained.

[0062] Therefore, in another feasible implementation, the drug recognition model includes a target recognition sub-model and a feature vector extraction sub-model; each frame of the image sequence is input into the pre-trained drug recognition model to obtain a drug detection frame and a drug category including:

[0063] Step S102A: input each frame of the image sequence into the pre-trained target recognition sub-model to obtain a drug region image.

[0064] In this step, the target recognition sub-model can also use the above-mentioned deep learning-based detection model to identify drugs, output the coordinate information of the drug detection frame, etc., and use the coordinate information of the drug detection frame to crop out the drug area image.

[0065] Step S102B: input the drug region image into the pre-trained feature vector extraction sub-model to obtain the drug feature vector.

[0066] In this step, a pre-trained convolutional neural network can be used as a feature extraction sub-model. Common choices include VGG, ResNet50, Inception, etc. These models are pre-trained on large-scale datasets and can extract representative visual features.

[0067] The drug feature vector includes the feature vector of the drug name and the feature vector of the drug appearance.

[0068] Step S102C: Based on a similarity matching algorithm, search the pre-built drug feature database for the drug category with the highest similarity match to the drug feature vector.

[0069] In the embodiment of the present invention, commonly used similarity matching algorithms include cosine similarity, Euclidean distance, Manhattan distance, etc. In a specific example, the cosine similarity algorithm can be used for calculation.

[0070] When constructing the drug feature database, a feature extraction sub-model is used to extract feature vectors of all categories of drugs in advance and store them in the drug feature database.

[0071] In practical applications, the cosine similarity algorithm is used to match the feature vectors of the actually collected drugs with the feature vectors in the database one by one through traversal, and the similarity value between the two is calculated. The calculated similarity value is compared with the preset similarity threshold, and the result with the highest similarity is returned.

[0072] This multi-stage approach is used to identify drug categories. Two sub-models need to be trained separately. However, when new types of drugs are added, there is no need to iterate the training model again. Instead, the feature vectors of the new drugs need only be stored in the drug database.

[0073] In actual application, users can choose any method to detect and identify drugs according to their needs.

[0074] Step S103: determining the target action and the start time and end time of the target action based on the hand detection frame and the drug detection frame, wherein the target action is the action of grasping the drug by hand.

[0075] In this step, the overlapping area of ​​the hand detection frame and the drug detection frame can be determined based on the coordinate information of the two. The target action is identified by judging whether there is overlap and the size of the overlapping area, and the start time and end time of the corresponding target action are recorded according to the identified target action.

[0076] Step S104: determining a final drug category recognition result based on the drug category recognized by the image sequence between the start time and the end time of the target action.

[0077] In a feasible implementation, first obtain the drug categories recognized by the image sequence between the start time and the end time of the target action; count the drug categories recognized by each frame of the image in this interval, and based on the voting mechanism, select the category with the most appearances as the final recognition result. The category with the most appearances is the drug category with the largest number in this interval.

[0078] Since the drug categories identified by image recognition are not necessarily accurate, there may be errors in the recognition of a certain frame of image. In this way, a voting mechanism is used to select the category with the most occurrences as the final recognition result, thereby greatly improving the accuracy of drug recognition.

[0079] The present invention recognizes the target action of grasping medicine by hand according to hand action recognition, and further determines the recognition result of the medicine category according to the target action, thereby realizing automatic recognition of the medicine category, which not only improves the efficiency of medicine recognition, but also improves the accuracy.

[0080] Based on the above embodiment, before determining the target action and the start time and end time of the target action based on the hand detection frame and the drug detection frame, the method further includes:

[0081] Step S105: input the hand detection frame into the target tracking model to obtain a hand tracking sequence.

[0082] In the embodiment of the present invention, the commonly used tracking algorithms of the target tracking model include KFC (Kernelized Correlation Filters) which is a correlation filter tracking algorithm, MOSSE (Minimum Output Sum of Squared Error) which is a very fast correlation filter tracking algorithm and ByteTrack tracking algorithm.

[0083] Step S106: When the number of hand tracking sequences is greater than or equal to a preset length threshold, executing subsequent steps.

[0084] In this step, the preset length threshold is, for example, 8. When the number of hand tracking sequences is greater than or equal to 8, the subsequent step of calculating the overlapping area is performed. If it is less than 8, the next frame of image is continuously acquired and the hand tracking sequence is updated until the preset length threshold is met.

[0085] This is because if the hand tracking sequence is too short, it may contain less information, and the data lacks stability and reliability, which is not conducive to the recognition of subsequent hand movements. By setting a preset length threshold as the starting condition for the subsequent steps, the reliability of the data can be improved, thereby improving the accuracy of subsequent target movement recognition.

[0086] Based on the above embodiment, determining the target action and the start time and end time of the target action based on the hand detection frame and the drug detection frame includes the following steps:

[0087] Step S103A: obtaining the coordinate information of the hand detection frame corresponding to the last sequence in the hand tracking sequence and the coordinate information of the drug detection frame located in the same frame image as the hand detection frame.

[0088] In an embodiment of the present invention, the coordinates of the detection frame can be expressed as (x, y, w, h), where x represents the X coordinate of the upper left corner of the detection frame; y represents the Y coordinate of the upper left corner of the detection frame; w represents the width of the detection frame; and h represents the height of the detection frame.

[0089] In an example, (x, y, w, h) = (100, 150, 200, 300) means that the upper left corner of the detection box is located at (100, 150), the width is 200 pixels, and the height is 300 pixels.

[0090] In another embodiment, the coordinates of the detection box may be represented by the coordinates of the lower left corner and the lower right corner.

[0091] Step S103B: determining the overlapping area of ​​the hand detection frame and the drug detection frame based on the coordinate information of the hand detection frame and the drug detection frame corresponding to the last sequence.

[0092] In an example, assuming that the hand detection frame is B1 and the drug detection frame is B2, their coordinates are: B1: (x1_1, y1_1, x2_1, y2_1); B2: (x1_2, y1_2, x2_2, y2_2);

[0093] Among them, (x1_1, y1_1) and (x1_2, y1_2) are the coordinates of the upper left corner of the detection box, and (x2_1, y2_1) and (x2_2, y2_2) are the coordinates of the lower right corner of the detection box.

[0094] The steps to calculate the overlap area are as follows:

[0095] Find the overlap of two detection boxes on the X axis:

[0096] The left edge of the overlap is the maximum of the left edges of the two bounding boxes: left = max(x1_1, x1_2);

[0097] The right edge of the overlap is the minimum of the right edges of the two bounding boxes: right = min(x2_1, x2_2);

[0098] If left>=right, it means that the two bounding boxes do not overlap on the X axis, and the overall overlapping area is 0.

[0099] Similarly, find the overlap of the two bounding boxes on the Y axis:

[0100] The top edge of the overlap is the maximum of the top edges of the two bounding boxes: top = max(y1_1, y1_2);

[0101] The bottom edge of the overlap is the minimum of the bottom edges of the two bounding boxes: bottom = min(y2_1, y2_2);

[0102] If top>=bottom, it means that the two bounding boxes do not overlap on the Y axis, and the overall overlapping area is also 0.

[0103] If the two bounding boxes overlap on both the X and Y axes, the width of the overlapping area is width = right-left, and the height is height = bottom-top, so the overlapping area is area = width*height.

[0104] Step S103C: determine the ratio of the overlapping area to the area of ​​the target detection frame, where the target detection frame is the smaller detection frame between the hand detection frame and the drug detection frame in the same frame image.

[0105] In practice, the area of ​​the medicine may be larger than that of the hand, or the area of ​​the hand may be larger than that of the medicine. Taking the detection frame with the smallest area of ​​the two as the benchmark, calculating the proportion of the overlapping area can be used to determine whether the hand has grasped the medicine, which has a higher accuracy rate.

[0106] Step S103D: When the ratio of the overlapping area to the area of ​​the target detection frame is greater than a preset ratio threshold, it is determined that the target action is detected; and the timestamp of the image corresponding to the first sequence in the hand tracking sequence is determined as the start time of the target action, and the timestamp of the image corresponding to the last sequence in the hand tracking sequence is determined as the end time of the target action.

[0107] In this step, the larger the proportion of the overlapping area, the greater the probability of catching the medicine. According to experience, the preset proportion threshold is set to 0.1.

[0108] Among them, when the ratio of the overlapping area to the area of ​​the target detection frame is less than the preset ratio threshold, a new image is recaptured, the detection frame in the new image is identified, and the detection frame is input into the target tracking model to update the hand tracking sequence. The overlapping area and the ratio of the overlapping area to the area of ​​the target detection frame are recalculated using the coordinate information of the detection frame and the drug detection frame corresponding to the last sequence of the updated hand tracking sequence. If it is less than, the hand tracking sequence is continued to be updated, and the overlapping area of ​​the last sequence is continued to be calculated until the ratio of the overlapping area to the area of ​​the target detection frame is greater than the preset ratio threshold. The timestamp of the image corresponding to the last sequence of the updated hand tracking sequence is used as the end time of the target action.

[0109] Based on the above embodiment, before obtaining the coordinate information of the hand detection frame corresponding to the last sequence in the current hand tracking sequence and the coordinate information of the drug detection frame located in the same frame image as the hand detection frame, the method further includes:

[0110] Step S103E: starting from the second sequence of the hand tracking sequence, determine in turn the size change of the hand detection frame corresponding to each sequence relative to the hand detection frame corresponding to the previous sequence; the size change includes three situations: change, decrease and remain unchanged.

[0111] Step S103F: When the size changes of a preset number of consecutive hand detection frames relative to the previous hand detection frame are all increased, and the increase values ​​are all greater than the preset change threshold, execute the subsequent steps.

[0112] In one example, the preset number is 3, and the preset change threshold is 0.02; that is, when three consecutive hand detection frames are all larger than the previous hand detection frame by 0.02, steps S103A-S103D can be executed.

[0113] The embodiment of the present invention determines whether the hand has the intention to grab the medicine by comparing the size changes of the detection frame in consecutive frames. If there is no intention to grab the medicine, the overlapping area is calculated. It is possible that the hand detection frame and the medicine detection frame do not overlap at this time, resulting in invalid calculation, wasting computing resources, and reducing recognition efficiency. When the intention to grab the medicine is recognized, the overlapping area is calculated again, which can help quickly and accurately identify the overlapping part of the hand detection frame and the medicine detection frame, thereby improving the efficiency of drug recognition.

[0114] Another embodiment of the present invention provides a drug verification method, such as Figure 2 As shown, the method comprises the following steps:

[0115] Step S201: Identify the drug name on the prescription based on the OCR (Optical Character Recognition) technology.

[0116] Recognizing the drug names on the prescription based on OCR technology usually includes the following steps:

[0117] The first step is image preprocessing: including image denoising, contrast enhancement, tilt correction and other operations.

[0118] This step is to improve the accuracy of OCR.

[0119] The second step is text detection: locating the text area in the image. Modern OCR systems usually use deep learning models to achieve this function, such as CTPN (Connectionist Text Proposal Network) or EAST (Efficient and Accurate Scene Text Detector).

[0120] The third step is text recognition: identifying the characters in the detected text area. This can be done through traditional machine learning methods such as SVM combined with HOG features, or through deep learning methods such as CRNN (Convolutional Recurrent Neural Network).

[0121] The fourth step is post-processing: correct the recognized text, such as correcting spelling errors, removing irrelevant characters, etc.

[0122] Step 5: Result output: Depending on the application requirements, the identified drug names may need to be converted into structured data, such as JSON format, for further processing or storage.

[0123] Step S202: Identify the captured drug category, which includes the drug name.

[0124] The drug category is identified using the method of the above embodiment, which will not be repeated here.

[0125] Step S203: Match the captured drug name with the drug name on the prescription.

[0126] In this step, regular expressions can be used to match the drug names on the prescription one by one.

[0127] Step S204: If the match is successful, the first instruction is returned, otherwise the second instruction is returned; the first instruction indicates that the medicine is dispensed successfully, and the second instruction indicates that an alarm is issued.

[0128] By identifying the drug category based on hand movement detection and then checking it with the drug name on the prescription, the accuracy of grabbing drugs is greatly improved, ensuring the safety of patients' medication.

[0129] The specific implementation process of the drug verification method of the present invention is described in detail below in conjunction with an embodiment of the drug identification method. Figure 3 As shown, the following steps are included:

[0130] Step S301: Acquire a sequence of image frames collected by a camera in real time;

[0131] Step S302: Perform OCR detection, drug detection and hand movement detection on each frame in the image frame sequence simultaneously to obtain a detection result; the detection result is one or more of a prescription detection result, a drug detection result and a hand movement detection result.

[0132] Step S303: Determine whether the test result includes the drug order test result, if yes, execute step S304; if no, execute step S305;

[0133] Step S304: Obtain and store prescription information;

[0134] Step S305: Determine whether the detection result includes the hand movement detection result and the drug detection result; if the hand movement detection result is included, execute step S306; if the drug detection action is included, execute step S307; if both are present, execute steps S306 and S307 simultaneously; if neither is present, return to step S301;

[0135] Step S306: judging whether there is a hand-holding medicine action according to the hand action detection result; if so, executing step S307; if not, returning to step S301;

[0136] Step S307: Obtain and store drug testing results;

[0137] Step S308: Determine whether the drug is on the prescription based on the prescription information and the drug test results. If yes, end; if not, execute step S309.

[0138] Step S309: The buzzer sounds an alarm.

[0139] Based on the same inventive concept, a drug identification device is provided, such as Figure 4 As shown, the device comprises:

[0140] An acquisition unit 401 is used to acquire an image sequence in the process of picking medicines through a wearable image acquisition device;

[0141] The recognition unit 402 is used to input each frame of the image sequence into the pre-trained action recognition model and the drug recognition model to obtain a hand detection frame, a drug detection frame and a drug category; the drug category includes a drug name category and a drug packaging box category;

[0142] A first determining unit 403 is used to determine a target action and a start time and an end time of the target action based on the hand detection frame and the drug detection frame, wherein the target action is an action of grabbing the drug by hand;

[0143] The second determining unit 404 is configured to determine a final drug category recognition result based on the drug category recognized by the image sequence between the start time and the end time of the target action.

[0144] Based on the same technical concept, an embodiment of the present invention further provides an electronic device, such as Figure 5 As shown, it includes a processor 501 , a communication interface 502 , a memory 503 and a communication bus 504 , wherein the processor 501 , the communication interface 502 , and the memory 503 communicate with each other via the communication bus 504 .

[0145] Memory 503, used for storing computer programs;

[0146] The processor 501 is used to implement the steps of the drug identification method when executing the program stored in the memory 503.

[0147] The communication bus mentioned in the above electronic device can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0148] The communication interface is used for communication between the above electronic device and other devices.

[0149] The memory may include a random access memory (RAM) or a non-volatile memory (NVM), such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

[0150] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0151] The computer program product for performing a drug identification method provided in an embodiment of the present invention includes a computer-readable storage medium storing program code. The instructions included in the program code can be used to execute the method described in the previous method embodiment. The specific implementation can be found in the method embodiment, which will not be repeated here.

[0152] The device for drug identification provided in the embodiment of the present invention can be specific hardware on the device or software or firmware installed on the device. The implementation principle and technical effects of the device provided in the embodiment of the present invention are the same as those of the aforementioned method embodiment. For the sake of brief description, for matters not mentioned in the device embodiment, reference can be made to the corresponding contents in the aforementioned method embodiment. Technical personnel in the relevant field can clearly understand that for the convenience and simplicity of description, the specific working processes of the systems, devices and units described above can all refer to the corresponding processes in the aforementioned method embodiment, and will not be repeated here.

[0153] In the embodiments provided by the present invention, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some communication interfaces, and the indirect coupling or communication connection of devices or units can be electrical, mechanical or other forms.

[0154] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0155] In addition, each functional unit in the embodiment provided by the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0156] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc., which can store program codes.

[0157] It should be noted that similar numbers and letters represent similar items in the following figures. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures. In addition, the terms "first", "second", "third", etc. are only used to distinguish the description and are not to be understood as indicating or implying relative importance.

[0158] Finally, it should be noted that the above-described embodiments are only specific implementations of the present invention, which are used to illustrate the technical solutions of the present invention, rather than to limit them. The protection scope of the present invention is not limited thereto. Although the present invention is described in detail with reference to the above-described embodiments, those skilled in the art should understand that any person skilled in the art can still modify the technical solutions described in the above-described embodiments within the technical scope disclosed by the present invention, or can easily think of changes, or perform equivalent replacements on some of the technical features thereof; and these modifications, changes or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present invention. They should all be covered within the protection scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the protection scope of the claims.

Claims

1. A drug identification method, characterized in that: The method comprises: The image sequence during the medicine picking process is acquired through a wearable image acquisition device; Input each frame of the image sequence into the pre-trained action recognition model and the drug recognition model to obtain a hand detection frame, a drug detection frame and a drug category; the drug category includes a drug name category and a drug packaging box category; Determining a target action and a start time and an end time of the target action based on the hand detection frame and the drug detection frame, wherein the target action is an action of grabbing medicine with a hand; The final drug category recognition result is determined based on the drug category recognized by the image sequence between the start time and the end time of the target action.

2. The method according to claim 1, characterized in that Before determining the target action and the start time and end time of the target action based on the hand detection frame and the drug detection frame, the method further includes: Inputting the hand detection frame into a target tracking model to obtain a hand tracking sequence; When the number of the hand tracking sequences is greater than a preset length threshold, subsequent steps are performed.

3. The method according to claim 2, characterized in that The determining of the target action and the start time and the end time of the target action based on the hand detection frame and the drug detection frame includes: Obtaining coordinate information of a hand detection frame corresponding to the last sequence in the hand tracking sequence and coordinate information of a drug detection frame located in the same frame image as the hand detection frame; Determine an overlapping area of ​​the hand detection frame and the drug detection frame based on the coordinate information of the hand detection frame and the drug detection frame corresponding to the last sequence; Determine the ratio of the overlapping area to the area of ​​the target detection frame, where the target detection frame is the smaller detection frame between the hand detection frame and the drug detection frame in the same frame image; When the ratio of the overlapping area to the area of ​​the target detection frame is greater than a preset ratio threshold, it is determined that the target action is detected; and the timestamp of the image corresponding to the first sequence in the hand tracking sequence is determined as the start time of the target action, and the timestamp of the image corresponding to the last sequence in the hand tracking sequence is determined as the end time of the target action.

4. The method according to claim 3, characterized in that Before obtaining the coordinate information of the hand detection frame corresponding to the last sequence in the current hand tracking sequence and the coordinate information of the drug detection frame located in the same frame image as the hand detection frame, the method further includes: Starting from the second sequence of the hand tracking sequence, determining in sequence the size change of the hand detection frame corresponding to each sequence relative to the hand detection frame corresponding to the previous sequence; the size change includes three situations: change, decrease and remain unchanged; When the size changes of a consecutive preset number of hand detection frames relative to the previous hand detection frame are all increased, and the increase values ​​are all greater than the preset change threshold, the subsequent steps are executed.

5. The method according to claim 1, characterized in that The drug recognition model includes a target recognition sub-model and a feature vector extraction sub-model; Inputting each frame of the image sequence into the pre-trained drug recognition model to obtain the drug detection frame and drug category includes: Inputting each frame of the image in the image sequence into a pre-trained target recognition sub-model to obtain a drug region image; Inputting the drug region image into a pre-trained feature vector extraction sub-model to obtain a drug feature vector; Based on the similarity matching algorithm, the drug category with the highest similarity to the drug feature vector is searched in the pre-built drug feature database.

6. The method according to claim 1, characterized in that The method of determining the final drug category recognition result based on the drug category recognized by the image sequence between the start time and the end time of the target action includes: Obtaining the drug category identified by the image sequence between the start time and the end time of the target action; Based on the voting mechanism, the drug category that appears most frequently between the start time and the end time of the target action is selected and used as the final drug category recognition result.

7. A drug verification method, characterized in that: The method comprises: Identify drug names on prescriptions based on OCR technology; Identifying the captured drug category based on the method described in any one of claims 1 to 6, wherein the drug category includes the drug name; Match the captured drug name with the drug name on the prescription; If the match is successful, the first instruction is returned, otherwise the second instruction is returned; the first instruction indicates that the medicine is dispensed successfully, and the second instruction indicates that an alarm is issued.

8. A drug identification device, characterized in that: The device comprises: An acquisition unit, used for acquiring an image sequence in the medicine picking process through a wearable image acquisition device; A recognition unit, used for inputting each frame of the image in the image sequence into a pre-trained action recognition model and a drug recognition model to obtain a hand detection frame, a drug detection frame and a drug category; the drug category includes a drug name category and a drug packaging box category; A first determining unit, configured to determine a target action and a start time and an end time of the target action based on the hand detection frame and the drug detection frame, wherein the target action is an action of grasping the drug by hand; The second determination unit is configured to determine a final drug category recognition result based on the drug category recognized by the image sequence between the start time and the end time of the target action.

9. An electronic device, characterized in that: It includes a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other through the communication bus; Memory, used to store computer programs; A processor, for implementing the method steps described in any one of claims 1 to 7 when executing a program stored in a memory.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method steps described in any one of claims 1 to 7 are implemented.

Images