Drug identification method, collation method, drug identification device, and electronic device

Abstract

The application provides a medicine identification method, a verification method, a medicine identification device and an electronic device. The method obtains an image sequence in a medicine taking process through a wearable image acquisition device; each frame of image in the image sequence is input into a pre-trained action recognition model and a medicine identification model to obtain a hand detection frame, a medicine detection frame and a medicine category; a target action, a start time and an end time of the target action are determined based on the hand detection frame and the medicine detection frame, the target action being a hand medicine taking action; and a final medicine category identification result is determined based on the medicine category identified from the image sequence between the start time and the end time of the target action. According to the application, the target action of hand medicine taking is identified according to action recognition, and the identification result of the medicine category is further determined according to the target action, so that automatic identification of the medicine category is realized, the efficiency of medicine identification is improved, and the accuracy is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, in particular to a medicine identification method, a medicine collation method, a medicine identification device and an electronic device. BACKGROUND

[0002] When dispensing medicines in a hospital pharmacy or a drugstore, the staff may make mistakes in medicine dispensing, which cannot ensure the safety of patient medication. In order to avoid such situations, the category of the dispensed medicines, such as the medicine name, needs to be identified and collated.

[0003] However, the existing medicine identification and collation are processed by manual work, which is not only inefficient but also prone to errors. SUMMARY

[0004] Therefore, the purpose of the present application is to provide a medicine identification method, a medicine collation method, a medicine identification device and an electronic device to improve the efficiency and accuracy of medicine identification.

[0005] In a first aspect, a medicine identification method is provided, the method comprising:

[0006] obtaining an image sequence in a medicine dispensing process by a wearable image acquisition device;

[0007] inputting each frame of image in the image sequence into a pre-trained action recognition model and a medicine identification model respectively to obtain a hand detection frame, a medicine detection frame and a medicine category; the medicine category includes a medicine name category and a medicine packaging box category;

[0008] determining a target action and a start time and an end time of the target action based on the hand detection frame and the medicine detection frame, the target action being a hand dispensing action;

[0009] determining a final medicine category identification result based on the image sequence between the start time and the end time of the target action.

[0010] Optionally, before determining the target action and the start time and the end time of the target action based on the hand detection frame and the medicine detection frame, the method further comprises:

[0011] inputting the hand detection frame into a target tracking model to obtain a hand tracking sequence;

[0012] when the number of the hand tracking sequence is greater than a preset length threshold, performing the subsequent steps.

[0013] Optionally, determining the target action and the start time and the end time of the target action based on the hand detection frame and the medicine detection frame comprises:

[0014] obtain coordinate information of a hand detection box corresponding to a last sequence in the hand tracking sequence and coordinate information of a medicine detection box in a same frame image as the hand detection box;

[0015] determine an overlapping area of the hand detection box and the medicine detection box based on the coordinate information of the hand detection box and the medicine detection box corresponding to the last sequence;

[0016] determine a proportion of the overlapping area to an area of a target detection box, the target detection box being a detection box with a smaller area between the hand detection box and the medicine detection box in the same frame image;

[0017] when the proportion of the overlapping area to the area of the target detection box is greater than a preset proportion threshold, determine that the target action is detected; and determine a timestamp of an image corresponding to a first sequence in the hand tracking sequence as a time when the target action starts, and determine a timestamp of an image corresponding to a last sequence in the hand tracking sequence as a time when the target action ends.

[0018] Optionally, before obtaining the coordinate information of the hand detection box corresponding to the last sequence in the current hand tracking sequence and the coordinate information of the medicine detection box in the same frame image as the hand detection box, the method further comprises:

[0019] starting from a second sequence of the hand tracking sequence, sequentially determine a size change of the hand detection box corresponding to each sequence relative to a hand detection box corresponding to a previous sequence; the size change includes three cases of change, decrease and no change;

[0020] when the size change of the hand detection box corresponding to each sequence relative to the hand detection box corresponding to the previous sequence is all increased, and the increased value is greater than a preset change threshold, execute the subsequent steps.

[0021] Optionally, the medicine recognition model includes a target recognition sub-model and a feature vector extraction sub-model; the method of inputting each frame image in the image sequence into the pre-trained medicine recognition model to obtain the medicine detection box and the medicine category includes:

[0022] inputting each frame image in the image sequence into the pre-trained target recognition sub-model to obtain a medicine region image;

[0023] inputting the medicine region image into the pre-trained feature vector extraction sub-model to obtain a medicine feature vector;

[0024] based on a similarity matching algorithm, searching for a medicine category with the highest similarity to the medicine feature vector in a pre-constructed medicine feature database.

[0025] Optionally, determining the final medicine category recognition result based on the medicine category identified by the image sequence between the start time and the end time of the target action includes:

[0026] obtaining a medicine category identified from the image sequence between the start time and the end time of the target action;

[0027] Based on the voting mechanism, the medicine category with the highest occurrence frequency between the start time and the end time of the target action is selected as the final medicine category recognition result.

[0028] In a second aspect, a medicine checking method is provided, and the method comprises:

[0029] obtaining a medicine name on the medicine list based on an OCR technology;

[0030] identifying the medicine category based on the method of any one of the first aspect, and the medicine category comprises the medicine name;

[0031] matching the medicine name obtained with the medicine name on the medicine list;

[0032] if the matching is successful, a first instruction is returned, otherwise a second instruction is returned; the first instruction indicates that the medicine fetching is successful, and the second instruction indicates that an alarm is issued.

[0033] In a third aspect, a medicine recognition device is provided, and the device comprises:

[0034] an obtaining unit configured to obtain an image sequence in a medicine fetching process by using a wearable image collection device;

[0035] an identifying unit configured to input each frame of image in the image sequence into a pre-trained action recognition model and a medicine recognition model respectively to obtain a hand detection frame, a medicine detection frame and a medicine category; the medicine category comprises a medicine name category and a medicine packaging box category;

[0036] a first determining unit configured to determine a target action and a start time and an end time of the target action based on the hand detection frame and the medicine detection frame; the target action is a hand medicine fetching action;

[0037] a second determining unit configured to determine a final medicine category recognition result based on a medicine category identified from the image sequence between the start time and the end time of the target action.

[0038] In a fourth aspect, an electronic device is provided, which comprises a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory complete mutual communication through the communication bus;

[0039] the memory is configured to store a computer program;

[0040] the processor is configured to execute the program stored on the memory to implement the method steps of any one of the first aspect or the second aspect.

[0041] Fifthly, a computer-readable storage medium is provided, wherein a computer program is stored therein, and when executed by a processor, the computer program implements the steps of the method described in either the first or second aspect.

[0042] This invention provides a drug identification method, verification method, drug identification device, and electronic device. The method acquires an image sequence during the drug-grabbing process using a wearable image acquisition device. Each frame of the image sequence is input into a pre-trained action recognition model and a drug identification model to obtain corresponding hand detection boxes, drug detection boxes, and drug categories. Based on the hand and drug detection boxes, the target action and its start and end times are determined. The final drug category identification result is determined based on the drug category identified from the image sequence between the start and end times of the target action. This invention identifies the target action of hand grasping drug based on action recognition, and further determines the drug category based on the target action, achieving automatic drug category identification. This not only improves the efficiency but also the accuracy of drug identification.

[0043] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings. Attached Figure Description

[0044] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the present invention and should not be regarded as a limitation on the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0045] Figure 1 A flowchart of a drug identification method provided by an embodiment of the present invention is shown;

[0046] Figure 2 A flowchart of a drug verification method provided by an embodiment of the present invention is shown;

[0047] Figure 3 This invention provides a specific implementation flowchart of a drug verification method according to an embodiment of the present invention.

[0048] Figure 4 This diagram illustrates the structure of a drug identification device provided in an embodiment of the present invention.

[0049] Figure 5 A schematic diagram of the structure of an electronic device provided in an embodiment of the present invention is shown. Detailed Implementation

[0050] In order to make the purposes, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, but not all the embodiments. The components of the embodiments of the present application described and shown in the drawings can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the claimed present application, but only represents selected embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without creative work fall within the scope of the present application.

[0051] Considering that the existing drug identification and checking are processed by manual operation, which is not only inefficient, but also prone to errors, based on this, the embodiments of the present application provide a drug identification method and device, which will be described below through embodiments.

[0052] The embodiments of the present application provide a drug identification method, as shown in the method, the method comprises the following steps: Figure 1

[0053] Step S101: acquiring an image sequence in a drug picking process through a wearable image acquisition device.

[0054] In the embodiments of the present application, the wearable image acquisition device is worn on the body of the drug picking personnel, and the wearing form is at least one of the following: head-mounted, neck-hung, binocular.

[0055] Step S102: inputting each frame of image in the image sequence into a pre-trained action recognition model and a drug identification model respectively to obtain a hand detection frame, a drug detection frame and a drug category.

[0056] The drug category includes a drug name category and a drug packaging box category.

[0057] In the embodiments of the present application, the action recognition model can adopt a model based on deep learning, such as YOLO, SSD, Faster R-CNN, etc., to recognize and locate the hand region in the image, and output the hand detection frame.

[0058] In a feasible implementation, the drug identification model can also adopt a detection model based on deep learning to identify the drug in the image, and output the drug detection frame and the specific name category and packaging box category of the drug. In an example, the packaging box category includes a box type, a transparent bottle type, an opaque bottle type, a tubular type and a plastic belt type, etc., which is used for auxiliary identification and plays a role of rapid screening and filtering.

[0059] ​The medicine recognition model adopts an end-to-end manner, directly outputs a medicine detection frame and a medicine category after inputting an image. Through the end-to-end manner, the speed of recognizing the medicine is fast and the efficiency is high.

[0060] In addition, in order to improve the recognition ability of the medicine recognition model, some easily confused medicines can be collected as training samples during training of the medicine recognition model.

[0061] The end-to-end manner has high recognition efficiency, but the total number of recognized medicine categories is fixed, and the medicine recognition model needs to be retrained if a new medicine category is added.

[0062] Therefore, in another possible implementation, the medicine recognition model includes a target recognition sub-model and a feature vector extraction sub-model; each frame of image in the image sequence is input into the pre-trained medicine recognition model to obtain a medicine detection frame and a medicine category, including:

[0063] Step S102A: each frame of image in the image sequence is input into the pre-trained target recognition sub-model to obtain a medicine region image.

[0064] In this step, the target recognition sub-model can also use the above-mentioned detection model based on deep learning to recognize the medicine, output the coordinate information of the medicine detection frame, and crop the medicine region image by using the coordinate information of the medicine detection frame.

[0065] Step S102B: input the medicine region image into the pre-trained feature vector extraction sub-model to obtain a medicine feature vector.

[0066] In this step, a pre-trained convolutional neural network can be used as the feature extraction sub-model. Common choices include VGG, ResNet50, Inception, etc. These models are pre-trained on large-scale data sets and can extract representative visual features.

[0067] The medicine feature vector includes a feature vector of a medicine name and a feature vector of a medicine appearance.

[0068] Step S102C: based on a similarity matching algorithm, find a medicine category with the highest similarity to the medicine feature vector in a pre-constructed medicine feature database.

[0069] In the embodiment of the application, common similarity matching algorithms include cosine similarity, Euclidean distance, Manhattan distance, etc. In a specific example, a cosine similarity algorithm can be used for calculation.

[0070] When constructing the medicine feature database, the feature extraction sub-model is used to pre-extract feature vectors of medicines of all categories, and the feature vectors are stored in the medicine feature database.

[0071] In actual application, the feature vectors of the actually collected medicines and the feature vectors in the database are matched one by one through traversal by using the cosine similarity algorithm, the similarity values of the two are calculated, the calculated similarity values are compared with the preset similarity threshold, and the result with the highest similarity is returned.

[0072] The medicine categories are identified through the multi-stage method, two sub-models need to be trained respectively, but when a new medicine category is added, the model does not need to be iteratively trained again, and only the feature vector of the new medicine needs to be stored in the medicine database.

[0073] In actual application, the user can select any mode according to the own demand to detect and identify medicines.

[0074] Step S103: determining a target action and start time and end time of the target action based on the hand detection frame and the medicine detection frame, the target action being a hand medicine grabbing action.

[0075] In this step, the overlapping area of the hand detection frame and the medicine detection frame can be determined based on the coordinate information of the two, the target action is identified by judging whether the two overlap and the size of the overlapping area, and the start time and end time of the corresponding target action are recorded according to the identified target action.

[0076] Step S104: determining a final medicine category recognition result based on the medicine category recognized by the image sequence between the start time and the end time of the target action.

[0077] In a feasible embodiment, first, the medicine category recognized by the image sequence between the start time and the end time of the target action is obtained; the medicine categories recognized by each frame of image in this interval are counted, and the category with the most occurrences is selected as the final recognition result based on a voting mechanism. The category with the most occurrences is the medicine category with the most quantity in the interval.

[0078] Since the medicine categories recognized by the image may not all be accurate, some frame of image may be recognized incorrectly, and then the category with the most occurrences is selected as the final recognition result through the voting mechanism, thereby greatly improving the accuracy of medicine recognition.

[0079] The present application identifies the target action of hand medicine grabbing through hand action recognition, further determines the recognition result of the medicine category according to the target action, and realizes automatic recognition of the medicine category, thereby improving the efficiency and accuracy of medicine recognition.

[0080] On the basis of the above embodiment, before determining the target action and the start time and end time of the target action based on the hand detection frame and the medicine detection frame, the method further comprises:

[0081] Step S105: input the hand detection frame into the target tracking model to obtain a hand tracking sequence.

[0082] In the embodiments of the present application, the commonly used tracking algorithm of the target tracking model includes KFC (Kernelized Correlation Filters), which is a correlation filter tracking algorithm, MOSSE (Minimum Output Sum of Squared Error), which is a very fast correlation filter tracking algorithm, and ByteTrack tracking algorithm.

[0083] Step S106: when the number of hand tracking sequences is greater than or equal to a preset length threshold, perform the subsequent step.

[0084] In this step, the preset length threshold is, for example, 8. When the number of hand tracking sequences is greater than or equal to 8, the subsequent step of calculating the overlapping area is performed, and if it is less than 8, the next frame of image is continuously acquired and the hand tracking sequence is updated until the preset length threshold is met.

[0085] This is because if the hand tracking sequence is too short, it may contain less information, and the data lacks stability and reliability, which is not conducive to the subsequent recognition of hand actions. By setting the preset length threshold as the starting condition of the subsequent step, the reliability of the data can be improved, and the accuracy of the subsequent target action recognition can be improved.

[0086] On the basis of the above embodiments, determining the target action and the start time and end time of the target action based on the hand detection frame and the medicine detection frame includes the following steps:

[0087] Step S103A: obtaining the coordinate information of the hand detection frame corresponding to the last sequence in the hand tracking sequence and the coordinate information of the medicine detection frame located in the same frame of image as the hand detection frame.

[0088] In the embodiments of the present application, the coordinates of the detection frame can be represented as (x, y, w, h), where x represents the X coordinate of the top-left corner of the detection frame, y represents the Y coordinate of the top-left corner of the detection frame, w represents the width of the detection frame, and h represents the height of the detection frame.

[0089] In one example, (x, y, w, h) = (100, 150, 200, 300) represents that the top-left corner of the detection frame is located at (100, 150), the width is 200 pixels, and the height is 300 pixels.

[0090] In another embodiment, the coordinates of the detection frame can also be represented by the coordinates of the bottom-left corner and the bottom-right corner.

[0091] Step S103B: determining the overlapping area of the hand bounding box and the medicine bounding box based on the coordinate information of the hand bounding box and the medicine bounding box corresponding to the last sequence.

[0092] In one example, it is assumed that the hand bounding box is B1 and the medicine bounding box is B2, and their coordinates are as follows: B1: (x1_1, y1_1, x2_1, y2_1); B2: (x1_2, y1_2, x2_2, y2_2).

[0093] Wherein (x1_1, y1_1) and (x1_2, y1_2) are the coordinates of the top-left corner of the bounding box, and (x2_1, y2_1) and (x2_2, y2_2) are the coordinates of the bottom-right corner of the bounding box.

[0094] The steps of calculating the overlapping area are as follows:

[0095] Find the overlapping part of the two bounding boxes in the X-axis:

[0096] The left edge of the overlapping part is the maximum value of the left edges of the two bounding boxes: left = max(x1_1, x1_2);

[0097] The right edge of the overlapping part is the minimum value of the right edges of the two bounding boxes: right = min(x2_1, x2_2);

[0098] If left >= right, it means that the two bounding boxes do not overlap in the X-axis, and the overall overlapping area is 0.

[0099] Similarly, find the overlapping part of the two bounding boxes in the Y-axis:

[0100] The top edge of the overlapping part is the maximum value of the top edges of the two bounding boxes: top = max(y1_1, y1_2);

[0101] The bottom edge of the overlapping part is the minimum value of the bottom edges of the two bounding boxes: bottom = min(y2_1, y2_2);

[0102] If top >= bottom, it means that the two bounding boxes do not overlap in the Y-axis, and the overall overlapping area is also 0.

[0103] If the two bounding boxes overlap in both the X-axis and the Y-axis, the width of the overlapping area is width = right - left, and the height is height = bottom - top, so the overlapping area is area = width * height.

[0104] Step S103C: determining the proportion of the overlapping area to the area of the target detection frame, the target detection frame being the detection frame with smaller area between the hand detection frame and the medicine detection frame in the same frame image.

[0105] In practice, it is possible that the area of the medicine is larger than that of the hand, or the area of the hand is larger than that of the medicine. In this case, the proportion of the overlapping area to the detection frame with the smallest area is calculated to determine whether the hand has grabbed the medicine, which is more accurate.

[0106] Step S103D: when the proportion of the overlapping area to the area of the target detection frame is greater than a preset proportion threshold, determining that the target action is detected; and determining the timestamp of the image corresponding to the first sequence in the hand tracking sequence as the time when the target action starts, and determining the timestamp of the image corresponding to the last sequence in the hand tracking sequence as the time when the target action ends.

[0107] In this step, the greater the proportion of the overlapping area is, the greater the probability of grabbing the medicine is. According to experience, the preset proportion threshold is set to 0.1.

[0108] When the proportion of the overlapping area to the area of the target detection frame is less than the preset proportion threshold, a new image is re-acquired, the detection frame in the new image is recognized, and the detection frame is input into the target tracking model to update the hand tracking sequence. The coordinates of the detection frame corresponding to the last sequence of the updated hand tracking sequence and the coordinates of the medicine detection frame are used to re-calculate the overlapping area and the proportion of the overlapping area to the area of the target detection frame. If the proportion is less than the preset proportion threshold, the hand tracking sequence is continuously updated, and the overlapping area of the last sequence is continuously calculated until the proportion of the overlapping area to the area of the target detection frame is greater than the preset proportion threshold. The timestamp of the image corresponding to the last sequence of the updated hand tracking sequence is used as the end time of the target action.

[0109] On the basis of the above embodiment, before acquiring the coordinate information of the hand detection frame corresponding to the last sequence in the current hand tracking sequence and the coordinate information of the medicine detection frame in the same frame image as the hand detection frame, the method further comprises:

[0110] Step S103E: starting from the second sequence of the hand tracking sequence, the size change of the hand detection frame corresponding to each sequence relative to the hand detection frame corresponding to the previous sequence is determined in sequence; the size change includes three cases of change, decrease, and no change.

[0111] Step S103F: when the size change of the continuously preset number of hand detection frames relative to the previous hand detection frame is all increased, and the increased value is all greater than a preset change threshold, the subsequent steps are executed.

[0112] In one example, the preset number is 3 and the preset change threshold is 0.02; that is, when the three consecutive hand detection boxes are all increased by 0.02 compared with the previous hand detection box, steps S103A-S103D can be executed.

[0113] The embodiment of the present application judges whether the hand has the intention of grabbing the medicine by comparing the size change of the detection box in the continuous frames, because if there is no intention of grabbing the medicine, the overlapping area is calculated, and at this time, the hand detection box and the medicine detection box may not be overlapped, resulting in invalid calculation, waste of calculation resources, and reduction of recognition efficiency. When the intention of grabbing the medicine is recognized, the overlapping area is calculated again, which can help to quickly and accurately identify the overlapping part of the hand detection box and the medicine detection box, and improve the efficiency of medicine recognition.

[0114] Another embodiment of the present application provides a medicine checking method, as shown in the figure, the method comprises the following steps: Figure 2

[0115] Step S201: identifying the medicine name on the medicine list based on the OCR (Optical Character Recognition) technology.

[0116] Identifying the medicine name on the medicine list based on the OCR technology generally includes the following steps:

[0117] First step, image preprocessing: including image denoising, contrast enhancement, tilt correction and other operations.

[0118] This step is to improve the accuracy of OCR.

[0119] Second step, text detection: locate the text area in the image. Modern OCR systems usually use deep learning models to achieve this function, such as CTPN (Connectionist Text Proposal Network) or EAST (Efficient and Accurate Scene Text Detector) and the like.

[0120] Third step, text recognition: recognize the characters in the detected text area. This can be done through traditional machine learning methods such as SVM combined with HOG features, or through deep learning methods such as CRNN (Convolutional Recurrent Neural Network).

[0121] Fourth step, post-processing: correct the recognized text, for example, correct spelling errors, remove irrelevant characters, etc.

[0122] ​Step 5: Result output: According to the application requirements, the identified drug name may need to be converted into structured data, such as JSON format, for further processing or storage.

[0123] Step S202: Identify the category of the captured drug, which includes the drug name.

[0124] The drug category is identified by the method described in the above embodiment, which is not repeated here.

[0125] Step S203: Match the captured drug name with the drug name on the drug list.

[0126] In this step, regular expressions can be used to match drug names one by one on the drug list.

[0127] Step S204: If the match is successful, return the first instruction, otherwise return the second instruction; the first instruction indicates that the drug picking is successful, and the second instruction indicates that an alarm is issued.

[0128] By recognizing the drug category according to hand action detection, and then checking with the drug name on the drug list, the accuracy of picking up drugs is greatly improved, ensuring the safety of patient medication.

[0129] The specific implementation process of the drug checking method of the present application will be described in detail below in combination with the embodiment of the drug recognition method, as shown in Figure 3 The method comprises the following steps:

[0130] Step S301: Obtain the image frame sequence captured by the camera in real time;

[0131] Step S302: Perform OCR detection, drug detection and hand action detection on each image in the image frame sequence respectively at the same time, and obtain the detection result; the detection result is one or more of the drug list detection result, the drug detection result and the hand action detection result.

[0132] Step S303: Determine whether the detection result contains the drug list detection result, if yes, execute step S304, if no, execute step S305;

[0133] Step S304: Obtain and store the drug list information;

[0134] Step S305: Determine whether the detection result contains the hand action detection result and the drug detection result; if the hand action detection result is contained, execute step S306; if the drug detection action is contained, execute step S307; if both exist, execute steps S306 and S307 at the same time; if none exists, return to step S301;

[0135] Step S306: determining whether there is a hand medicine taking action according to the hand action detection result; if yes, executing step S307; if no, returning to step S301;

[0136] Step S307: obtaining and storing the medicine detection result.

[0137] Step S308: determining whether the medicine is on the medicine list according to the medicine list information and the medicine detection result; if yes, ending; if no, executing step S309.

[0138] Step S309: buzzer alarm.

[0139] Based on the same inventive concept, a medicine identification device is provided, as shown in the accompanying drawings, which comprises: Figure 4 An acquisition unit 401 is configured to acquire an image sequence in a medicine taking process through a wearable image acquisition device.

[0140] An identification unit 402 is configured to input each frame of image in the image sequence into a pre-trained action recognition model and a medicine identification model respectively to obtain a hand detection frame, a medicine detection frame and a medicine category correspondingly; the medicine category comprises a medicine name category and a medicine packaging box category.

[0141] A first determination unit 403 is configured to determine a target action and a start time and an end time of the target action based on the hand detection frame and the medicine detection frame, the target action being a hand medicine taking action.

[0142] A second determination unit 404 is configured to determine a final medicine category identification result based on the medicine category identified from the image sequence between the start time and the end time of the target action.

[0143] Based on the same technical concept, the present application further provides an electronic device, as shown in the accompanying drawings, which comprises a processor 501, a communication interface 502, a memory 503 and a communication bus 504, wherein the processor 501, the communication interface 502 and the memory 503 complete mutual communication through the communication bus 504.

[0144] Figure 5 The memory 503 is configured to store a computer program.

[0145] The processor 501 is configured to execute the program stored in the memory 503 to realize the steps of the medicine identification method.

[0146] The processor 501 is configured to execute the program stored in the memory 503 to realize the steps of the medicine identification method.

[0147] ​The communication bus mentioned in the above electronic device can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For the convenience of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0148] The communication interface is used for communication between the above electronic device and other devices.

[0149] The memory can include a Random Access Memory (RAM) and can also include a Non-Volatile Memory (NVM), for example, at least one disk memory. Optionally, the memory can also be at least one storage device located away from the aforementioned processor.

[0150] The processor mentioned above can be a general-purpose processor, including a Central Processing Unit (CPU), a Network Processor (NP), etc.; can also be a Digital Signal Processing (DSP), an Application Specific Integrated Circuit (ASIC), a Field-Programmable Gate Array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0151] The computer program product for performing the medicine identification method provided by the embodiment of the application includes a computer readable storage medium storing program codes, the program codes include instructions for executing the method described in the foregoing method embodiments, and specific implementation can be referred to the method embodiments, which will not be described here.

[0152] The device for medicine identification provided by the embodiment of the application can be specific hardware on the equipment or software or firmware installed on the equipment, etc. The device provided by the embodiment of the application has the same implementation principle and generated technical effects as the foregoing method embodiments, and for the sake of brief description, the part not mentioned in the device embodiment can be referred to the corresponding content in the foregoing method embodiments. Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the system, device and unit described above can be referred to the corresponding process in the foregoing method embodiments, which will not be described here.

[0153] In the embodiments of the present application, it should be understood that the disclosed device and method can be implemented in other manners. The described embodiments of the device are merely exemplary, for example, the division of the units is only a logical function division, and there can be another division manner in actual implementation; for example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the displayed or discussed mutual couplings or direct couplings or communication connections can be indirect couplings or communication connections through some interfaces, devices or units, and can be electric, mechanical or in other forms.

[0154] The units described as separate components can or can not be physically separate, and the components displayed as units can or can not be physical units, i.e., can be located in one place, or can be distributed on a plurality of network units. In actual implementation, some or all of the units can be selected according to the actual needs to achieve the purposes of the embodiments of the present application.

[0155] In addition, each functional unit in the embodiments of the present application can be integrated in one processing unit, or each unit can exist physically as a separate unit, or two or more units can be integrated in one unit.

[0156] If the functions are implemented in the form of software function units and sold or used as independent products, they can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the present application essentially or the parts that make contributions to the prior art or parts of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium, and includes a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, and various media that can store program codes.

[0157] It should be noted that: similar reference numerals and letters in the following drawings represent similar items, and therefore, once an item is defined in one drawing, it does not need to be further defined and explained in subsequent drawings. In addition, the terms "first", "second", "third", etc. are only used to distinguish descriptions, and cannot be understood as indicating or implying relative importance.

[0158] Finally, it should be noted that the above-described embodiments are merely specific embodiments of the present application, which are used to illustrate the technical solutions of the present application, but not to limit the present application, and the protection scope of the present application is not limited thereto. Although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art can still modify or easily think of changes to the technical solutions recorded in the foregoing embodiments, or make equivalent replacements to some technical features therein, within the technical scope disclosed by the present application, and these modifications, changes or replacements do not cause the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present application. All should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A medicine identification method characterized by, The method comprises: acquiring an image sequence in a medicine taking process by a wearable image acquisition device; inputting each frame of image in the image sequence into a pre-trained action recognition model and a medicine recognition model respectively to obtain a hand detection box, a medicine detection box and a medicine category; the medicine category comprises a medicine name category and a medicine packaging box category; the medicine recognition model comprises a target recognition sub-model and a feature vector extraction sub-model; inputting each frame of image in the image sequence into the pre-trained medicine recognition model to obtain the medicine detection box and the medicine category comprises: inputting each frame of image in the image sequence into the pre-trained target recognition sub-model to obtain a medicine region image; inputting the medicine region image into the pre-trained feature vector extraction sub-model to obtain a medicine feature vector; the medicine feature vector comprises a medicine name feature vector and a medicine appearance feature vector; based on a similarity matching algorithm, searching for a medicine category with the highest similarity with the medicine feature vector in a pre-constructed medicine feature database; determining an overlapping area of the hand detection box and the medicine detection box based on coordinate information of the two, and determining a target action, a start time and an end time of the target action through the size of the overlapping area, the target action being a hand medicine taking action; determining a final medicine category recognition result based on the medicine category identified by the image sequence between the start time and the end time of the target action.

2. The method of claim 1, wherein, Before determining the target action, the start time and the end time of the target action based on the hand detection box and the medicine detection box, the method further comprises: inputting the hand detection box into a target tracking model to obtain a hand tracking sequence; when the number of the hand tracking sequence is greater than a preset length threshold, performing subsequent steps.

3. The method of claim 2, wherein, the determination of the target action, the start time and the end time of the target action based on the hand detection box and the medicine detection box comprises: acquiring coordinate information of a hand detection box corresponding to a last sequence in the hand tracking sequence and coordinate information of a medicine detection box in the same frame of image as the hand detection box; determining an overlapping area of the hand detection box and the medicine detection box based on the coordinate information of the hand detection box and the medicine detection box corresponding to the last sequence; determining a proportion of the overlapping area to an area of a target detection box, the target detection box being a detection box with a smaller area in the hand detection box and the medicine detection box in the same frame of image; when the proportion of the overlapping area to the area of the target detection box is greater than a preset proportion threshold, determining that the target action is detected; and determining a timestamp of an image corresponding to a first sequence in the hand tracking sequence as a time when the target action starts, and determining a timestamp of an image corresponding to the last sequence in the hand tracking sequence as a time when the target action ends.

4. The method of claim 3, wherein, before acquiring the coordinate information of the hand detection box corresponding to the last sequence in the current hand tracking sequence and the coordinate information of the medicine detection box in the same frame of image as the hand detection box, the method further comprises: From the second sequence of the hand tracking sequence, the size change of each sequence corresponding hand detection box relative to the previous sequence corresponding hand detection box is determined in turn; the size change includes three cases of change, decrease and keep unchanged; When the size change of the continuous preset number of hand detection boxes relative to the previous hand detection box is all increased, and the increased value is all greater than the preset change threshold, the subsequent step is executed.

5. The method of claim 1, wherein, The final medicine category recognition result determined based on the medicine category recognized from the image sequence between the start time and the end time of the target action includes: Obtaining the medicine category recognized from the image sequence between the start time and the end time of the target action; Based on the voting mechanism, the medicine category with the highest appearance frequency between the start time and the end time of the target action is selected as the final medicine category recognition result.

6. A medicine collation method characterized by comprising: The method comprises: Recognizing the medicine name on the medicine list based on the OCR technology; The method of any one of claims 1-5 is used to recognize the medicine category, which includes the medicine name; Matching the medicine name obtained with the medicine name on the medicine list; If the matching is successful, a first instruction is returned, otherwise a second instruction is returned; the first instruction indicates that the medicine fetching is successful, and the second instruction indicates that an alarm is sent.

7. A drug identification device, characterized by, The device comprises: An acquisition unit configured to acquire an image sequence in a medicine fetching process by using a wearable image acquisition device; An identification unit configured to input each frame of image in the image sequence into a pre-trained action recognition model and a medicine recognition model respectively to obtain a hand detection box, a medicine detection box and a medicine category; the medicine category includes a medicine name category and a medicine packaging box category; The medicine recognition model comprises a target recognition sub-model and a feature vector extraction sub-model; each frame of image in the image sequence is input into the pre-trained medicine recognition model to obtain the medicine detection box and the medicine category, which comprises: Each frame of image in the image sequence is input into the pre-trained target recognition sub-model to obtain a medicine region image; The medicine region image is input into the pre-trained feature vector extraction sub-model to obtain a medicine feature vector; the medicine feature vector includes a medicine name feature vector and a medicine appearance feature vector; Based on a similarity matching algorithm, the medicine category with the highest similarity with the medicine feature vector is searched in a pre-constructed medicine feature database; A first determination unit configured to determine an overlapping area of the hand detection box and the medicine detection box based on coordinate information of the two, and determine a target action and a start time and an end time of the target action by the size of the overlapping area, the target action being a hand medicine fetching action; A second determination unit configured to determine a final medicine category recognition result based on the medicine category recognized from the image sequence between the start time and the end time of the target action.

8. An electronic device, comprising: The device comprises a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory complete mutual communication through the communication bus; The memory is used to store a computer program. A processor for implementing the method steps of any one of claims 1-6 when the processor executes a program stored on a memory.

9. A computer-readable storage medium, characterized in that, A computer program stored in the computer readable storage medium, the computer program being executed by a processor to implement the method steps of any one of claims 1-6.

Images