Computer vision-based medicine verification method and system

Abstract

A computer vision-based medicine verification method and system, applied to the technical field of article sorting. A weight increment feature is generated by obtaining a real-time weight; when the weight increment feature is greater than or equal to a weight increment threshold, multi-dimensional photographing is performed to generate a three-dimensional model; and a size feature of an article to be sorted is generated. A reference model for medicine boxes to be sorted is obtained, reference size features of the medicine boxes are extracted, and a primary-sorting medicine box type is obtained; when the number of primary-sorting medicine box types is greater than 1, secondary sorting is performed by combining reference color features of the medicine boxes to obtain a secondary-sorting medicine box type; when the number of secondary sorting medicine box types is greater than 1, tertiary sorting is performed on the basis of a text feature of the article to be sorted to obtain a tertiary-sorting medicine box type; and when the number of tertiary sorting medicine box types is 1, a manipulator is activated on the basis of the tertiary-sorting medicine box type, to grab the article to be sorted for sorting. The technical problems in the prior art of high verification cost and low verification accuracy of medicine verification methods are solved.

Description

A drug verification method and system based on computer vision Technical Field

[0001] The present invention relates to the field of article sorting, and in particular to a medicine verification method and system based on computer vision. Background Art

[0002] Drug verification is the process of verifying the consistency of dispensed medications with prescribed medications. Existing drug verification solutions often rely on manual verification or a single image comparison algorithm. While manual verification offers high accuracy, it is also costly. Furthermore, image comparison algorithms are more likely to produce erroneous results when comparing completely different medications from the same manufacturer and packaging, compromising drug verification accuracy.

[0003] Therefore, the drug verification method in the prior art has technical problems of high verification cost and low verification accuracy.

[0004] Summary of the Invention

[0005] The present application solves the technical problems of high verification cost and low verification accuracy in the existing drug verification methods by providing a drug verification method and system based on computer vision.

[0006] The present application provides a drug verification method based on computer vision, which is applied to a computer vision drug system, wherein the system is communicatively connected to a drug box sorting device, wherein the drug box sorting device includes a manipulator, a pan-tilt camera and a drug box detection area, wherein the drug box detection area is equipped with a weight sensor and a size sensor, including: comparing the real-time weight of the detection area of ​​the interactive weight sensor with the blank detection weight to generate a weight increment feature; when the weight increment feature is greater than or equal to the weight increment threshold, activating the pan-tilt camera to move to the drug box detection area for multi-dimensional shooting to generate a three-dimensional model of the object to be sorted; activating the size sensor to calibrate the size of the three-dimensional model of the object to be sorted to generate the size feature of the object to be sorted; obtaining a reference model of the drug box to be sorted, extracting the drug The first-level sorting is performed based on the box reference size characteristics, and the size characteristics of the items to be sorted are combined to obtain the first-level sorting medicine box type; when the number of the first-level sorting medicine box types is greater than 1, the color distribution characteristics of the items to be sorted are extracted according to the three-dimensional model of the items to be sorted, and the first-level sorting medicine box type is performed based on the medicine box reference color characteristics to obtain the second-level sorting medicine box type; when the number of the second-level sorting medicine box types is greater than 1, the text characteristics of the items to be sorted are extracted according to the three-dimensional model of the items to be sorted, and the second-level sorting medicine box type is performed based on the medicine box reference text characteristics to obtain the third-level sorting medicine box type; when the number of the third-level sorting medicine box types is 1, the manipulator is activated according to the third-level sorting medicine box type to grab the items to be sorted for sorting.

[0007] The present application also provides a drug verification system based on computer vision, wherein the system is communicatively connected to a medicine box sorting device, wherein the medicine box sorting device includes a manipulator, a pan-tilt camera and a medicine box detection area, wherein the medicine box detection area is equipped with a weight sensor and a size sensor, including: a weight increment acquisition module for comparing the real-time weight of the detection area of ​​the interactive weight sensor with the blank detection weight to generate a weight increment feature; a three-dimensional model acquisition module for activating the pan-tilt camera to move to the medicine box detection area for multi-dimensional shooting when the weight increment feature is greater than or equal to the weight increment threshold, and generating a three-dimensional model of the object to be sorted; a size feature acquisition module for activating the size sensor to calibrate the size of the three-dimensional model of the object to be sorted, and generating a size feature of the object to be sorted; a primary sorting module for acquiring a reference model of the medicine box to be sorted, extracting The medicine box reference size feature is combined with the size feature of the object to be sorted to perform first-level sorting to obtain the first-level sorted medicine box type; the second-level sorting module is used to extract the color distribution features of the object to be sorted according to the three-dimensional model of the object to be sorted when the number of the first-level sorting medicine box types is greater than 1, and perform second-level sorting on the first-level sorting medicine box type in combination with the medicine box reference color feature to obtain the second-level sorting medicine box type; the third-level sorting module is used to extract the text features of the object to be sorted according to the three-dimensional model of the object to be sorted when the number of the second-level sorting medicine box types is greater than 1, and perform third-level sorting on the second-level sorting medicine box type in combination with the medicine box reference text feature to obtain the third-level sorting medicine box type; the sorting grasping module is used to activate the manipulator according to the third-level sorting medicine box type when the number of the third-level sorting medicine box types is 1, and grasp the object to be sorted for sorting.

[0008] The present application also provides an electronic device, comprising:

[0009] a memory for storing executable instructions;

[0010] The processor is used to implement a computer vision-based drug verification method provided by the present application when executing the executable instructions stored in the memory.

[0011] The present application provides a computer-readable storage medium storing a computer program. When the program is executed by a processor, a computer vision-based drug verification method provided by the present application is implemented.

[0012] The present application proposes a computer vision-based drug verification method and system. By acquiring real-time weight and generating weight increment features, the system performs multi-dimensional photography when the weight increment is greater than or equal to a weight increment threshold, generates a three-dimensional model, and generates size features of the items to be sorted. A baseline model of the medicine box to be sorted is obtained, the baseline size features of the medicine box are extracted, and the type of the first-level sorted medicine box is obtained. When the number of first-level sorted medicine box types is greater than one, a second-level sorting is performed based on the baseline color features of the medicine box to obtain the type of the second-level sorted medicine box. When the number of second-level sorted medicine box types is greater than one, a third-level sorting is performed based on the text features of the items to be sorted to obtain the type of the third-level sorted medicine box. When the number of third-level sorted medicine box types is one, a robotic arm is activated based on the third-level sorted medicine box type to grab the items to be sorted for sorting. This achieves multi-directional verification of medicines, greatly reducing the probability of false detection, avoiding a large amount of manual work, and reducing verification costs. This solves the technical problems of high verification costs and low verification accuracy in existing drug verification methods.

[0013] The above description is only an overview of the technical solution of the present application. In order to more clearly understand the technical means of the present application, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the present application more obvious and easy to understand, the specific implementation methods of the present application are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] In order to more clearly illustrate the technical solutions of the embodiments of the present disclosure, the following briefly introduces the drawings of the embodiments of the present disclosure. Obviously, the drawings described below only relate to some embodiments of the present disclosure and are not intended to limit the present disclosure.

[0015] FIG1 is a flow chart of a computer vision-based drug verification method provided in an embodiment of the present application;

[0016] FIG2 is a schematic diagram of a flow chart of a computer vision-based drug verification method for constructing weight intervals for drug boxes to be sorted, provided in an embodiment of the present application;

[0017] FIG3 is a schematic diagram of a process for primary sorting of a drug verification method based on computer vision provided in an embodiment of the present application;

[0018] FIG4 is a schematic diagram of the structure of a system for a computer vision-based drug verification method provided in an embodiment of the present application;

[0019] FIG5 is a schematic diagram of the structure of a system electronic device for a computer vision-based drug verification method provided by an embodiment of the present invention.

[0020] Explanation of the accompanying drawings: weight increment acquisition module 11, three-dimensional model acquisition module 12, size feature acquisition module 13, primary sorting module 14, secondary sorting module 15, tertiary sorting module 16, sorting and grasping module 17, processor 31, memory 32, input device 33, output device 34. DETAILED DESCRIPTION

[0021] Example 1

[0022] In order to make the purpose, technical solutions and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. The described embodiments should not be regarded as limiting this application. All other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0023] In the following description, reference is made to “some embodiments”, which describes a subset of all possible embodiments, but it will be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0024] In the following description, the terms "first\second\third" involved are merely used to distinguish similar objects and do not represent a specific ordering of the objects. It can be understood that "first\second\third" can be interchanged with a specific order or sequence where permitted, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein.

[0025] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application.

[0026] Although this application makes various references to certain modules in the systems according to embodiments of the application, any number of different modules may be used and run on the user terminal and / or server, the modules are illustrative only, and different aspects of the systems and methods may use different modules.

[0027] Flowcharts are used throughout this application to illustrate the operations performed by the systems of the embodiments of the present application. It should be understood that the preceding or following operations are not necessarily performed in exact order. Instead, the various steps may be processed in reverse order or simultaneously, as needed. Furthermore, other operations may be added to these processes, or one or more operations may be removed from these processes.

[0028] As shown in FIG1 , an embodiment of the present application provides a computer vision-based drug verification method, which is applied to a computer vision drug system. The system is communicatively connected to a drug box sorting device. The drug box sorting device includes a manipulator, a pan-tilt camera, and a drug box detection area. The drug box detection area is equipped with a weight sensor and a size sensor, including:

[0029] The real-time weight of the detection area of ​​the interactive weight sensor is compared with the blank detection weight to generate a weight increment feature;

[0030] When the weight increment feature is greater than or equal to the weight increment threshold, the pan-tilt camera is activated to move to the medicine box detection area for multi-dimensional shooting to generate a three-dimensional model of the object to be sorted;

[0031] activating the size sensor to calibrate the size of the three-dimensional model of the object to be sorted, and generating size features of the object to be sorted;

[0032] Drug verification is a method for verifying the consistency of delivered drugs with prescribed medications. Existing drug verification solutions often rely on manual verification or a single image comparison algorithm. While manual verification offers high accuracy, it is costly. However, image comparison algorithms are more likely to produce erroneous results when comparing completely different drugs from the same manufacturer and packaging, compromising drug verification accuracy. The real-time weight of the detection area of ​​the interactive weight sensor is compared with the blank detection weight, which is the weight of a blank without any drugs placed in it, to generate a weight increment signature. When this weight increment signature is greater than or equal to a weight increment threshold, it indicates a possible overdose of drugs, necessitating further verification. A pan-tilt camera is activated and moved to the drug box detection area for multi-dimensional imaging, generating a three-dimensional model of the items to be sorted. When the weight increment signature is less than the weight increment threshold, it indicates a drug underdose, and manual drug quantity verification is performed. The weight increment threshold is set based on the weight range of the drug boxes to be sorted, and this weight increment threshold is within a specific range. Furthermore, the size sensor is activated to calibrate the size of the three-dimensional model of the object to be sorted, and generate size features of the object to be sorted.

[0033] As shown in FIG2 , the method provided in the embodiment of the present application further includes:

[0034] Extract the weight test records of the medicine boxes according to the list of medicine box types to be sorted;

[0035] Traversing the list of medicine box types to be sorted, performing characteristic weight evaluation on the weight detection records of the medicine boxes, and generating a plurality of representative weights of the medicine boxes to be sorted;

[0036] Constructing weight intervals of the medicine boxes to be sorted according to the representative weights of the plurality of medicine boxes to be sorted;

[0037] When the weight increment characteristic does not meet the weight range of the medicine boxes to be sorted, the robot is activated to grab the items to be sorted and transport them to the abnormal area, and the user terminal is reminded to perform manual review.

[0038] When acquiring the weight increment feature, extract the medicine box weight detection record according to the list of medicine box types to be sorted. The medicine box type list corresponds to the medicine type, that is, multiple historical weight detection records of the corresponding medicine box type are obtained according to the medicine box type list. Further, traverse the list of medicine box types to be sorted, perform feature weight evaluation on the medicine box weight detection records, that is, perform data statistical screening on the medicine box weight detection records, and generate multiple representative weights of medicine boxes to be sorted. Further, based on the multiple representative weights of medicine boxes to be sorted, construct a weight range of medicine boxes to be sorted. When the weight increment feature meets the weight range of medicine boxes to be sorted, it means that the weight of the medicine meets the requirements. When the weight increment feature does not meet the weight range of medicine boxes to be sorted, activate the robot to grab the item to be sorted and transport it to the abnormal area, and remind the user end to perform manual review.

[0039] The method provided in the embodiment of the present application also includes:

[0040] Extracting a first type of medicine box to be sorted and a first group of medicine box weight detection records according to the list of medicine box types to be sorted;

[0041] Clustering and grouping the first group of medicine box weight detection records according to a preset weight deviation to generate a weight detection record grouping result;

[0042] Clean the weight detection records in which the number of detection records in the group of the weight detection record grouping result is less than the preset number of records, obtain the weight detection record cleaning result, perform mean calculation on the weight detection record cleaning result, generate the representative weight of the first type of medicine boxes to be sorted, and add it to the representative weights of the multiple medicine boxes to be sorted.

[0043] According to the list of medicine box types to be sorted, the first medicine box type to be sorted and the first group of medicine box weight detection records are extracted, wherein the first medicine box type to be sorted is a medicine box type in the list of medicine box types to be sorted, and the first group of medicine box weight detection records are the historical medicine box weight detection records corresponding to the first medicine box type to be sorted. Subsequently, the first group of medicine box weight detection records are clustered and grouped according to the preset weight deviation to generate the weight detection record grouping result. The preset weight deviation is a pre-set weight deviation corresponding to the standard weight of the medicine box type, such as a deviation of 1g to 5g, -5g to -1g, etc. from the standard weight. The specific grouping category can be set based on the actual situation. The weight deviation interval and quantity of the heavier medicine box type are set higher, otherwise the deviation interval and quantity are set lower. Each preset weight deviation interval corresponds to a group, and the first group of medicine box weight detection records are clustered and grouped according to the preset weight deviation to generate the weight detection record grouping result. Furthermore, the weight detection records whose number of detection records in the group of the weight detection record grouping result is less than the preset number of records are cleaned, that is, the weight detection records whose number of detection records in the group of the weight detection record grouping result is less than the preset number of records are filtered out, wherein the preset number of records can be set based on the data volume of the overall weight detection record. The higher the data volume, the more preset records correspond to it, and vice versa. The weight detection record grouping result after the weight detection record cleaning is obtained, and the weight detection record cleaning result is obtained. The mean calculation is performed on the weight detection record cleaning result, that is, the mean of the data of each group in the weight detection record cleaning result is respectively performed, and the data mean of all weight detection record cleaning result groups is obtained to generate the first representative weight of the type of medicine box to be sorted, and the representative weights of the multiple medicine boxes to be sorted are added.

[0044] Obtain a reference model of the medicine box to be sorted, extract the reference size features of the medicine box, perform primary sorting based on the size features of the objects to be sorted, and obtain the type of the first-level sorted medicine box;

[0045] When the number of the first-level sorting medicine box types is greater than 1, extracting the color distribution characteristics of the objects to be sorted based on the three-dimensional model of the objects to be sorted, and performing second-level sorting on the first-level sorting medicine box types in combination with the medicine box reference color characteristics to obtain second-level sorting medicine box types;

[0046] When the number of the secondary sorting medicine box types is greater than 1, extracting text features of the objects to be sorted based on the three-dimensional model of the objects to be sorted, and performing a three-level sorting on the secondary sorting medicine box types in combination with the medicine box reference text features to obtain three-level sorting medicine box types;

[0047] When the number of the three-level sorting medicine box types is 1, the manipulator is activated according to the three-level sorting medicine box types to grab the items to be sorted for sorting.

[0048] A reference model of the medicine box to be sorted is obtained, and reference size features of the medicine box are extracted. Primary sorting is performed based on the size features of the items to be sorted, thereby obtaining a primary sorted medicine box type. Specifically, primary sorting is performed by aligning the sizes of the medicine box to be sorted with the items to be sorted. Within the primary sorted medicine box type, the alignment deviation between the sizes of the medicine box to be sorted and the items to be sorted is small. If the number of primary sorted medicine box types is greater than one, indicating that there are items to be sorted with similar sizes to the medicine box to be sorted, the color distribution features of the items to be sorted are extracted based on the three-dimensional models of the items to be sorted. Secondary sorting is performed on the primary sorted medicine box types based on the reference color features of the medicine box, thereby obtaining a secondary sorted medicine box type. If the number of secondary sorted medicine box types is greater than one, indicating that there are items to be sorted with similar RGB colors to the medicine box to be sorted, the text features of the items to be sorted are extracted based on the three-dimensional models of the items to be sorted. Thirdary sorting is performed on the secondary sorted medicine box types based on the reference text features of the medicine box, thereby obtaining a tertiary sorted medicine box type. When the number of three-level sorting box types is 1, and a drug that meets the sorting requirements exists among the items to be sorted, the robot arm is activated based on the three-level sorting box type to grab the item to be sorted and sort it. This enables multi-directional verification of drugs, greatly reducing the probability of false positives, avoiding extensive manual work, and lowering verification costs.

[0049] As shown in FIG3 , the method provided in the embodiment of the present application further includes:

[0050] Aligning the intersecting edges of the reference model of the medicine box to be sorted and the three-dimensional model of the object to be sorted to generate multiple alignment schemes;

[0051] Traversing the multiple alignment schemes, and counting the distances between multiple groups of nearest corner points of the reference model of the medicine box to be sorted and the three-dimensional model of the object to be sorted;

[0052] Counting the average values ​​of multiple corner point distances of the multiple groups of nearest corner point distances, and extracting the maximum value of the nearest corner point distance of the alignment solution with the minimum value of the multiple corner point distance averages;

[0053] When the maximum value of the nearest corner distance is less than or equal to the first corner distance threshold, and the mean corner distance is less than or equal to the second corner distance threshold, the to-be-sorted medicine box type corresponding to the to-be-sorted medicine box benchmark model is added to the first-level sorting medicine box type.

[0054] The intersecting edges of the reference model of the medicine box to be sorted and the three-dimensional model of the object to be sorted are aligned, that is, the intersecting corners of the two edges to be aligned are aligned first, and then the two edges are aligned so that the two models overlap, and the distances between the other corner points are calculated. Different corner points are used for alignment to generate multiple alignment schemes. Among them, the medicine box to be sorted is the medicine originally required, and the object to be sorted is the medicine actually taken at present. Further, the multiple alignment schemes are traversed, and the distances between multiple groups of nearest corner points of the reference model of the medicine box to be sorted and the three-dimensional model of the object to be sorted are counted. That is, after the two models are aligned, each corner point of the reference model of the medicine box to be sorted can find a nearest corner point in the three-dimensional model of the object to be sorted, and the distances between multiple groups of nearest corner points of the three-dimensional model are counted. Subsequently, the averages of the multiple corner distances of the multiple groups of nearest corner distances are calculated, and the maximum nearest corner distance of the alignment solution with the minimum of the multiple corner distance averages is extracted. This is to obtain the alignment solution with the minimum of the multiple corner distance averages. This solution achieves the best alignment due to its minimum average. The maximum nearest corner distance in this solution is then obtained, which is the maximum distance deviation of the misaligned corners. Finally, when the maximum nearest corner distance is less than or equal to a first corner distance threshold, and the average corner distance is less than or equal to a second corner distance threshold, the to-be-sorted medicine box type corresponding to the to-be-sorted medicine box reference model is added to the primary sorting medicine box types. The first and second corner distance thresholds are both pre-set distance thresholds; smaller thresholds correspond to higher alignment coincidence accuracy requirements. When the value is greater than the first and second corner distance thresholds, the alignment effect is poor, possibly indicating that the to-be-sorted medicine box reference model and the to-be-sorted item are of different categories. In this case, the number of primary sorting medicine box types is zero.

[0055] The method provided in the embodiment of the present application also includes:

[0056] Extracting a first pixel point distribution position set and a first pixel RGB value set of the three-dimensional model of the object to be sorted according to the alignment scheme of the first-level sorting medicine box type;

[0057] Extracting a second pixel point distribution position set and a second pixel RGB value set of the reference model of the medicine box to be sorted according to the alignment scheme of the first-level sorting medicine box type;

[0058] performing color Euclidean distance calculation based on the first pixel point distribution position set, the first pixel RGB value set, the second pixel point distribution position set, and the second pixel RGB value set to generate a plurality of color distribution distances;

[0059] Counting a mean value of the color distribution distances according to the multiple color distribution distances, and counting a ratio of the number of abnormal pixels whose color distribution distance is greater than or equal to the color distribution distance threshold;

[0060] When the color distribution distance mean is less than or equal to the color distribution distance threshold, and the abnormal pixel number ratio is less than or equal to the abnormal pixel number ratio threshold, the to-be-sorted medicine box type corresponding to the to-be-sorted medicine box benchmark model is added to the secondary sorting medicine box type.

[0061] Based on the alignment scheme for the primary sorting medicine box type, a first pixel distribution position set and a first pixel RGB value set are extracted from the three-dimensional model of the object to be sorted. Based on the alignment scheme for the primary sorting medicine box type, a second pixel distribution position set and a second pixel RGB value set are extracted from the reference model of the medicine box to be sorted. Color Euclidean distance calculation is performed based on the first pixel distribution position set, the first pixel RGB value set, the second pixel distribution position set, and the second pixel RGB value set to generate multiple color distribution distances. A mean color distribution distance is calculated based on the multiple color distribution distances. The ratio of abnormal pixels with a color distribution distance greater than or equal to a color distribution distance threshold is calculated. Pixels with a color distribution distance greater than or equal to the color distribution distance threshold are considered abnormal pixels. The ratio of abnormal pixels to all pixels is calculated as the abnormal pixel ratio. The color distribution distance threshold is a parameter preset by a technician based on actual conditions. When the mean color distribution distance is less than or equal to the color distribution distance threshold, the color distribution of the medicine box to be sorted and the object to be sorted are highly consistent. If the abnormal pixel ratio is less than or equal to the abnormal pixel ratio threshold, the number of abnormal points in the color distribution of the objects to be sorted meets a preset ratio requirement. The abnormal pixel ratio threshold is a percentage threshold pre-set by a technician; a smaller percentage threshold corresponds to a stricter judgment. The medicine box types corresponding to the baseline models of medicine boxes to be sorted that meet the requirements are added to the secondary sorting medicine box types.

[0062] The method provided in the embodiment of the present application also includes:

[0063] Extracting a first text code set and a first text distribution position set of the three-dimensional model of the object to be sorted according to the alignment scheme of the first-level sorting medicine box type, wherein the text code sequence represents text content;

[0064] Extracting a second character code set and a second character distribution position set of the benchmark model of the medicine box to be sorted according to the alignment scheme of the first-level sorting medicine box type;

[0065] Calculate Hamming distances based on the first character encoding set and the first character distribution position set, and the second character encoding set and the second character distribution position set to obtain multiple Hamming distances;

[0066] According to the multiple Hamming distances, calculating the mean of the Hamming distances, and calculating the ratio of the number of characters whose Hamming distances are greater than or equal to the Hamming distance threshold;

[0067] When the mean Hamming distance is less than or equal to the Hamming distance threshold, and the character quantity ratio is less than or equal to the character quantity ratio threshold, the to-be-sorted medicine box type corresponding to the to-be-sorted medicine box benchmark model is added to the three-level sorting medicine box type.

[0068] Based on the alignment scheme for the first-level sorting medicine box types, a first character code set and a first character distribution position set are extracted from the three-dimensional model of the object to be sorted, where the character code sequence represents the text content. Based on the alignment scheme for the first-level sorting medicine box types, a second character code set and a second character distribution position set are extracted from the reference model of the medicine box to be sorted. Furthermore, Hamming distances are calculated based on the first character code set and the first character distribution position set, as well as the second character code set and the second character distribution position set, to obtain multiple Hamming distances. Subsequently, based on these multiple Hamming distances, a mean Hamming distance is calculated, and the ratio of the number of characters whose Hamming distance is greater than or equal to a Hamming distance threshold is calculated, i.e., the ratio of the number of characters whose Hamming distance is greater than or equal to the Hamming distance threshold to the total number of characters is calculated. When the mean Hamming distance is less than or equal to the Hamming distance threshold, and the character number ratio is less than or equal to the character number ratio threshold, the medicine box type to be sorted corresponding to the reference model of the medicine box to be sorted is added to the third-level sorting medicine box types.

[0069] The method provided in the embodiment of the present application also includes:

[0070] When the number of the three-level sorting medicine box types is 0, the robot is activated to grab the items to be sorted and transport them to the abnormal area, and the user terminal is reminded to perform manual review;

[0071] When the number of the three-level sorting medicine box types is greater than 1, the three-dimensional model of the object to be sorted and the three-level sorting medicine box type are sent to the user terminal for sorting to obtain a four-level sorting medicine box type, wherein the number of the four-level sorting medicine box types is 1;

[0072] The robot is activated according to the type of the four-level sorting medicine box to grab the items to be sorted for sorting.

[0073] When the number of the three-level sorting box types is 0, at this time, the items to be sorted and the medicine box to be sorted do not meet the sorting requirements, and the consistency between the items to be sorted and the medicine box to be sorted is low, then the robot arm is activated, the items to be sorted are grabbed and transported to the abnormal area, and the user terminal is reminded to perform manual review. When the number of the three-level sorting box types is greater than 1, at this time, there are medicines that meet the sorting requirements among the items to be sorted, then the three-dimensional model of the items to be sorted and the three-level sorting box type are sent to the user terminal for sorting, and the corresponding sorted medicines are confirmed by the user terminal to obtain the four-level sorting box type, wherein the number of the four-level sorting box types is 1, and the robot arm is activated according to the four-level sorting box type, and the items to be sorted are grabbed for sorting, thereby achieving accurate extraction of medicines.

[0074] The technical solution provided by the embodiments of the present invention uses the real-time weight of the detection area of ​​the interactive weight sensor to generate a weight increment feature. When the weight increment feature is greater than or equal to a weight increment threshold, the pan-tilt camera is activated to move to the medicine box detection area for multi-dimensional imaging, generating a three-dimensional model of the object to be sorted. The size sensor is activated to calibrate the size of the three-dimensional model of the object to be sorted, generating a size feature of the object to be sorted. A reference model of the medicine box to be sorted is obtained, and the reference size feature of the medicine box is extracted to obtain the first-level sorting medicine box type. When the number of the first-level sorting medicine box types is greater than one, the color distribution feature of the object to be sorted is extracted based on the three-dimensional model of the object to be sorted. Combined with the reference color feature of the medicine box, a second-level sorting is performed to obtain the second-level sorting medicine box type. When the number of the second-level sorting medicine box types is greater than one, the text feature of the object to be sorted is extracted based on the three-dimensional model of the object to be sorted, and a third-level sorting is performed to obtain the third-level sorting medicine box type. When the number of the third-level sorting medicine box types is one, the robot arm is activated based on the third-level sorting medicine box type to grab the object to be sorted and sort it. This method realizes multi-directional verification of drugs, greatly reduces the probability of false detection, avoids a lot of manual work and reduces verification costs. It solves the technical problems of high verification costs and low verification accuracy in the existing drug verification methods.

[0075] Example 2

[0076] Based on the same inventive concept as the computer vision-based drug verification method in the aforementioned embodiment, the present invention also provides a system for a computer vision-based drug verification method. The system can be implemented in hardware and / or software and can generally be integrated into an electronic device to perform the method provided by any embodiment of the present invention. As shown in Figure 4, the system is in communication with a medicine box sorting device, which includes a manipulator, a pan-tilt camera, and a medicine box detection area. The medicine box detection area is equipped with a weight sensor and a size sensor, including:

[0077] The weight increment acquisition module 11 is used to compare the real-time weight of the detection area of ​​the interactive weight sensor with the blank detection weight to generate a weight increment feature;

[0078] A three-dimensional model acquisition module 12 is configured to activate a pan-tilt camera to move to a medicine box detection area for multi-dimensional photography when the weight increment feature is greater than or equal to a weight increment threshold, thereby generating a three-dimensional model of the object to be sorted;

[0079] A size feature acquisition module 13 is used to activate the size sensor to calibrate the size of the three-dimensional model of the object to be sorted, and generate the size feature of the object to be sorted;

[0080] The primary sorting module 14 is used to obtain a reference model of the medicine box to be sorted, extract the reference size characteristics of the medicine box, perform primary sorting based on the size characteristics of the objects to be sorted, and obtain the type of the medicine box to be sorted;

[0081] The secondary sorting module 15 is configured to, when the number of the primary sorted medicine box types is greater than one, extract the color distribution characteristics of the objects to be sorted based on the three-dimensional model of the objects to be sorted, perform secondary sorting on the primary sorted medicine box types in combination with the medicine box reference color characteristics, and obtain the secondary sorted medicine box types;

[0082] The three-level sorting module 16 is configured to, when the number of the second-level sorted medicine box types is greater than one, extract text features of the objects to be sorted based on the three-dimensional models of the objects to be sorted, and perform three-level sorting on the second-level sorted medicine box types in combination with the medicine box reference text features to obtain three-level sorted medicine box types;

[0083] The sorting and grabbing module 17 is used to activate the manipulator according to the three-level sorting medicine box type when the number of the three-level sorting medicine box types is 1, and grab the items to be sorted for sorting.

[0084] Furthermore, the weight increment acquisition module 11 is further configured to:

[0085] Extract the weight test records of the medicine boxes according to the list of medicine box types to be sorted;

[0086] Traversing the list of medicine box types to be sorted, performing characteristic weight evaluation on the weight detection records of the medicine boxes, and generating a plurality of representative weights of the medicine boxes to be sorted;

[0087] Constructing weight intervals of the medicine boxes to be sorted according to the representative weights of the plurality of medicine boxes to be sorted;

[0088] When the weight increment characteristic does not meet the weight range of the medicine boxes to be sorted, the robot is activated to grab the items to be sorted and transport them to the abnormal area, and the user terminal is reminded to perform manual review.

[0089] Furthermore, the weight increment acquisition module 11 is further configured to:

[0090] Extracting a first type of medicine box to be sorted and a first group of medicine box weight detection records according to the list of medicine box types to be sorted;

[0091] Clustering and grouping the first group of medicine box weight detection records according to a preset weight deviation to generate a weight detection record grouping result;

[0092] Clean the weight detection records in which the number of detection records in the group of the weight detection record grouping result is less than the preset number of records, obtain the weight detection record cleaning result, perform mean calculation on the weight detection record cleaning result, generate the representative weight of the first type of medicine boxes to be sorted, and add it to the representative weights of the multiple medicine boxes to be sorted.

[0093] Furthermore, the primary sorting module 14 is also used for:

[0094] Aligning the intersecting edges of the reference model of the medicine box to be sorted and the three-dimensional model of the object to be sorted to generate multiple alignment schemes;

[0095] Traversing the multiple alignment schemes, and counting the distances between multiple groups of nearest corner points of the reference model of the medicine box to be sorted and the three-dimensional model of the object to be sorted;

[0096] Counting the average values ​​of multiple corner point distances of the multiple groups of nearest corner point distances, and extracting the maximum value of the nearest corner point distance of the alignment solution with the minimum value of the multiple corner point distance averages;

[0097] When the maximum value of the nearest corner distance is less than or equal to the first corner distance threshold, and the mean corner distance is less than or equal to the second corner distance threshold, the to-be-sorted medicine box type corresponding to the to-be-sorted medicine box benchmark model is added to the first-level sorting medicine box type.

[0098] Furthermore, the secondary sorting module 15 is also used for:

[0099] Extracting a first pixel point distribution position set and a first pixel RGB value set of the three-dimensional model of the object to be sorted according to the alignment scheme of the first-level sorting medicine box type;

[0100] Extracting a second pixel point distribution position set and a second pixel RGB value set of the reference model of the medicine box to be sorted according to the alignment scheme of the first-level sorting medicine box type;

[0101] performing color Euclidean distance calculation based on the first pixel point distribution position set, the first pixel RGB value set, the second pixel point distribution position set, and the second pixel RGB value set to generate a plurality of color distribution distances;

[0102] Counting a mean value of the color distribution distances according to the multiple color distribution distances, and counting a ratio of the number of abnormal pixels whose color distribution distance is greater than or equal to the color distribution distance threshold;

[0103] When the color distribution distance mean is less than or equal to the color distribution distance threshold, and the abnormal pixel number ratio is less than or equal to the abnormal pixel number ratio threshold, the to-be-sorted medicine box type corresponding to the to-be-sorted medicine box benchmark model is added to the secondary sorting medicine box type.

[0104] Furthermore, the three-stage sorting module 16 is also used for:

[0105] Extracting a first text code set and a first text distribution position set of the three-dimensional model of the object to be sorted according to the alignment scheme of the first-level sorting medicine box type, wherein the text code sequence represents text content;

[0106] Extracting a second character code set and a second character distribution position set of the benchmark model of the medicine box to be sorted according to the alignment scheme of the first-level sorting medicine box type;

[0107] Calculate Hamming distances based on the first character encoding set and the first character distribution position set, and the second character encoding set and the second character distribution position set to obtain multiple Hamming distances;

[0108] According to the multiple Hamming distances, calculating the mean of the Hamming distances, and calculating the ratio of the number of characters whose Hamming distances are greater than or equal to the Hamming distance threshold;

[0109] When the mean Hamming distance is less than or equal to the Hamming distance threshold, and the character quantity ratio is less than or equal to the character quantity ratio threshold, the to-be-sorted medicine box type corresponding to the to-be-sorted medicine box benchmark model is added to the three-level sorting medicine box type.

[0110] Furthermore, the sorting and grabbing module 17 is also used for:

[0111] When the number of the three-level sorting medicine box types is 0, the robot is activated to grab the items to be sorted and transport them to the abnormal area, and the user terminal is reminded to perform manual review;

[0112] When the number of the three-level sorting medicine box types is greater than 1, the three-dimensional model of the object to be sorted and the three-level sorting medicine box type are sent to the user terminal for sorting to obtain a four-level sorting medicine box type, wherein the number of the four-level sorting medicine box types is 1;

[0113] The robot is activated according to the type of the four-level sorting medicine box to grab the items to be sorted for sorting.

[0114] The various units and modules included are divided only according to functional logic, but are not limited to the above division, as long as the corresponding functions can be achieved; in addition, the specific names of the functional units are only for the convenience of distinguishing each other and are not used to limit the scope of protection of the present invention.

[0115] Example 3

[0116] FIG5 is a schematic diagram of the structure of the electronic device provided in Example 3 of the present invention, showing a block diagram of an exemplary electronic device suitable for implementing the embodiments of the present invention. The electronic device shown in FIG5 is merely an example and should not impose any restrictions on the functions and scope of use of the embodiments of the present invention. As shown in FIG5 , the electronic device includes a processor 31, a memory 32, an input device 33, and an output device 34; the number of processors 31 in the electronic device can be one or more, and FIG5 takes one processor 31 as an example. The processor 31, memory 32, input device 33, and output device 34 in the electronic device can be connected via a bus or other means, and FIG5 takes connection via a bus as an example.

[0117] Memory 32, as a computer-readable storage medium, can be used to store software programs, computer-executable programs, and modules, such as the program instructions / modules corresponding to the computer vision-based drug verification method in the embodiments of the present invention. Processor 31 executes the software programs, instructions, and modules stored in memory 32 to perform various computer functions and data processing, thereby implementing the aforementioned computer vision-based drug verification method.

[0118] Note that the above are only preferred embodiments of the present invention and the technical principles employed. Those skilled in the art will understand that the present invention is not limited to the specific embodiments described herein, and that various obvious changes, readjustments, and substitutions can be made by those skilled in the art without departing from the scope of protection of the present invention. Therefore, although the present invention has been described in detail through the above embodiments, the present invention is not limited to the above embodiments and may include many other equivalent embodiments without departing from the concept of the present invention. The scope of the present invention is determined by the scope of the appended claims.

Claims

1. A drug verification method based on computer vision, characterized in that: A pharmaceutical system applied to computer vision, wherein the system is in communication with a medicine box sorting device, wherein the medicine box sorting device comprises a manipulator, a pan / tilt camera, and a medicine box detection area, wherein a weight sensor and a size sensor are deployed in the medicine box detection area, including: The real-time weight of the detection area of ​​the interactive weight sensor is compared with the blank detection weight to generate a weight increment feature; When the weight increment feature is greater than or equal to the weight increment threshold, the pan / tilt camera is activated to move to the medicine box detection area for multi-dimensional shooting to generate a three-dimensional model of the object to be sorted; Activating the size sensor to calibrate the size of the three-dimensional model of the object to be sorted, and generating the size characteristics of the object to be sorted; Obtain a reference model of the medicine box to be sorted, extract reference size features of the medicine box, perform primary sorting based on the size features of the objects to be sorted, and obtain the type of the primary sorted medicine box; When the number of the first-level sorting medicine box types is greater than 1, extracting the color distribution characteristics of the objects to be sorted according to the three-dimensional model of the objects to be sorted, performing second-level sorting on the first-level sorting medicine box types in combination with the medicine box reference color characteristics, and obtaining second-level sorting medicine box types; When the number of the secondary sorting medicine box types is greater than 1, extracting text features of the objects to be sorted according to the three-dimensional model of the objects to be sorted, and performing three-level sorting on the secondary sorting medicine box types in combination with the medicine box reference text features to obtain three-level sorting medicine box types; When the number of the three-level sorting medicine box types is 1, the manipulator is activated according to the three-level sorting medicine box types to grab the items to be sorted for sorting.

2. The method according to claim 1, characterized in that The real-time weight of the detection area of ​​the interactive weight sensor is compared with the blank detection weight to generate a weight increment feature, which also includes: Extract the weight test records of medicine boxes according to the list of medicine box types to be sorted; Traversing the list of types of medicine boxes to be sorted, performing characteristic weight evaluation on the weight detection records of the medicine boxes, and generating a plurality of representative weights of the medicine boxes to be sorted; Constructing weight intervals of medicine boxes to be sorted according to the representative weights of the plurality of medicine boxes to be sorted; When the weight increment characteristic does not meet the weight range of the medicine boxes to be sorted, the robot is activated to grab the items to be sorted and transport them to the abnormal area, and the user end is reminded to perform manual review.

3. The method according to claim 2, characterized in that Traversing the list of types of medicine boxes to be sorted, performing characteristic weight evaluation on the weight detection records of the medicine boxes, and generating multiple representative weights of the medicine boxes to be sorted, including: Extracting the first type of medicine box to be sorted and the first group of medicine box weight detection records according to the list of medicine box types to be sorted; Clustering and grouping the first group of medicine box weight detection records according to a preset weight deviation to generate a weight detection record grouping result; The weight detection records in which the number of detection records in the group of the weight detection record grouping result is less than the preset number of records are cleaned, and the weight detection record cleaning result is obtained, and the mean calculation is performed on the weight detection record cleaning result to generate a representative weight of the first type of medicine boxes to be sorted, and add it to the representative weights of the multiple medicine boxes to be sorted.

4. The method according to claim 1, characterized in that Obtain a reference model of the medicine box to be sorted, extract the reference size features of the medicine box, perform primary sorting based on the size features of the items to be sorted, and obtain the type of the primary sorted medicine box, including: Aligning the intersecting edges of the reference model of the medicine box to be sorted and the three-dimensional model of the object to be sorted to generate multiple alignment schemes; Traversing the multiple alignment schemes, and counting the distances between multiple groups of nearest corner points of the reference model of the medicine box to be sorted and the three-dimensional model of the object to be sorted; Counting the average values ​​of multiple corner point distances of the multiple groups of nearest corner point distances, and extracting the maximum value of the nearest corner point distance of the alignment solution with the minimum value of the average values ​​of the multiple corner point distances; When the maximum value of the nearest corner distance is less than or equal to the first corner distance threshold, and the mean corner distance is less than or equal to the second corner distance threshold, the to-be-sorted medicine box type corresponding to the to-be-sorted medicine box benchmark model is added to the primary sorted medicine box type.

5. The method according to claim 4, characterized in that when the number of the first-level sorting medicine box types is greater than 1, the color distribution characteristics of the objects to be sorted are extracted according to the three-dimensional model of the objects to be sorted, and the first-level sorting medicine box types are subjected to second-level sorting in combination with the medicine box reference color characteristics to obtain the second-level sorting medicine box types, comprising: Extracting a first pixel point distribution position set and a first pixel RGB value set of the three-dimensional model of the object to be sorted according to the alignment scheme of the first-level sorting medicine box type; Extracting a second pixel point distribution position set and a second pixel RGB value set of the benchmark model of the medicine box to be sorted according to the alignment scheme of the first-level sorted medicine box type; Performing color Euclidean distance calculation according to the first pixel point distribution position set, the first pixel RGB value set, the second pixel point distribution position set, and the second pixel RGB value set to generate a plurality of color distribution distances; According to the plurality of color distribution distances, a color distribution distance mean is counted, and a ratio of the number of abnormal pixels whose color distribution distance is greater than or equal to the color distribution distance threshold is counted; When the color distribution distance mean is less than or equal to the color distribution distance threshold, and the abnormal pixel number ratio is less than or equal to the abnormal pixel number ratio threshold, the to-be-sorted medicine box type corresponding to the to-be-sorted medicine box benchmark model is added to the secondary sorting medicine box type.

6. The method according to claim 4, characterized in that When the number of the secondary sorting medicine box types is greater than 1, extracting text features of the objects to be sorted according to the three-dimensional model of the objects to be sorted, performing three-level sorting on the secondary sorting medicine box types in combination with the medicine box reference text features, and obtaining three-level sorting medicine box types, including: According to the alignment scheme of the first-level sorting medicine box type, extracting the first text code set and the first text distribution position set of the three-dimensional model of the object to be sorted, wherein the text code sequence represents the text content; Extracting a second character code set and a second character distribution position set of the benchmark model of the medicine box to be sorted according to the alignment scheme of the first-level sorting medicine box type; Perform Hamming distance calculation based on the first character encoding set and the first character distribution position set, and the second character encoding set and the second character distribution position set to obtain multiple Hamming distances; According to the multiple Hamming distances, calculating the mean value of the Hamming distances, and calculating the ratio of the number of characters whose Hamming distances are greater than or equal to the Hamming distance threshold; When the mean Hamming distance is less than or equal to the Hamming distance threshold, and the character quantity ratio is less than or equal to the character quantity ratio threshold, the to-be-sorted medicine box type corresponding to the to-be-sorted medicine box benchmark model is added to the three-level sorted medicine box type.

7. The method according to claim 1, characterized in that Also includes: When the number of the three-level sorting medicine box types is 0, the robot is activated to grab the items to be sorted and transport them to the abnormal area, and the user terminal is reminded to perform manual review; When the number of the three-level sorting medicine box types is greater than 1, the three-dimensional model of the object to be sorted and the three-level sorting medicine box type are sent to the user end for sorting, and a four-level sorting medicine box type is obtained, wherein the number of the four-level sorting medicine box types is 1; The robot arm is activated according to the type of the four-stage sorting medicine box to grab the objects to be sorted for sorting.

8. A drug verification system based on computer vision, characterized in that: The system is in communication connection with a medicine box sorting device, the medicine box sorting device includes a manipulator, a pan / tilt camera and a medicine box detection area, the medicine box detection area is equipped with a weight sensor and a size sensor, including: The weight increment acquisition module is used to compare the real-time weight of the detection area of ​​the interactive weight sensor with the blank detection weight to generate the weight increment feature; A three-dimensional model acquisition module is used to activate the pan-tilt camera to move to the medicine box detection area for multi-dimensional shooting to generate a three-dimensional model of the object to be sorted when the weight increment feature is greater than or equal to the weight increment threshold; A size feature acquisition module is used to activate the size sensor to calibrate the size of the three-dimensional model of the object to be sorted, and generate the size feature of the object to be sorted; A primary sorting module is used to obtain a reference model of a medicine box to be sorted, extract reference size features of the medicine box, perform primary sorting based on the size features of the object to be sorted, and obtain the type of the primary sorted medicine box; The secondary sorting module is used to sort the medicine boxes according to the first-level sorting method when the number of the first-level sorting medicine boxes is greater than 1. Extracting color distribution characteristics of the objects to be sorted according to the three-dimensional model of the objects to be sorted, performing secondary sorting on the primary sorted medicine box types in combination with the reference color characteristics of the medicine boxes, and obtaining secondary sorted medicine box types; A three-level sorting module is used for, when the number of the two-level sorting medicine box types is greater than 1, extracting text features of the objects to be sorted according to the three-dimensional model of the objects to be sorted, and performing three-level sorting on the two-level sorting medicine box types in combination with the medicine box reference text features to obtain three-level sorting medicine box types; The sorting and grabbing module is used to activate the manipulator according to the three-level sorting medicine box type when the number of the three-level sorting medicine box types is 1, and grab the objects to be sorted for sorting.

9. An electronic device, characterized in that: The electronic device comprises: A memory for storing executable instructions; A processor, used to implement a computer vision-based drug verification method as described in any one of claims 1 to 7 when executing executable instructions stored in the memory.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, a computer vision-based drug verification method as described in any one of claims 1 to 7 is implemented.

Images