Medicine checking and automatic distribution method based on machine vision

Abstract

The invention discloses a medicine checking and automatic distributing method based on machine vision. The method comprises the steps that a medicine rechecking area is determined, an image of a medicine sorted by a pharmacist is obtained through collection equipment, the feature vector of the medicine is extracted, a medicine recognition model is constructed, the feature vector is input into the model for recognition, medicine recognition information is obtained, and the medicine rechecking area is used for checking the medicine. The method comprises the following steps: acquiring the position of an acquisition device, comparing the information with data in a medicine database, and determining the automatic distribution of medicines or feeding back abnormal information according to the comparison result, so that the position of the acquisition device is reasonably set, the medicine feature vector processing flow is optimized, and an automatic distribution and abnormal feedback mechanism is introduced; the accuracy of medicine identification and the robustness of the system are remarkably improved, errors in manual operation are effectively reduced, and the efficiency and safety of medicine checking and distribution are improved.

Description

Technical Field

[0001] The present invention relates to the technical field of drug image processing, and in particular to a drug checking and automatic dispensing method based on machine vision. Background Art

[0002] As the digitalization and intelligentization of the medical field continue to advance, the demand for automation of drug management systems is increasing. The traditional drug verification and distribution process mainly relies on manual operation. Although this method can meet the needs in small-scale scenarios, with the expansion of medical institutions and the increase in the types and quantities of drugs, the limitations of manual operation are gradually emerging. In addition, visual fatigue and human negligence during the manual verification process can easily lead to misidentification and misdelivery of drugs, which in turn affects the safety of patients' medication. In addition, the packaging forms of drugs vary, and some drug packaging is highly similar, which increases the difficulty of manual identification and the probability of error. In order to reduce human errors and improve the efficiency of drug verification and distribution, machine vision recognition technology has gradually become a research hotspot. Machine vision technology can automatically identify drugs through image acquisition and intelligent algorithms, without the need for manual operation, reducing the error rate and improving the overall efficiency of drug verification.

[0003] However, there are still some technical problems in the existing machine vision recognition technology for the verification and distribution of drugs. For example, before being sent to the model for processing, the drug image acquisition area is not fully considered, resulting in low image recognition in the drug image acquisition area, which limits the robustness of the model. Especially when the drug packaging is similar, the image quality is greatly affected by ambient light, and the size of the acquisition area is uncertain, the recognition accuracy and reliability of the model cannot achieve the ideal goal. In addition, if the solution adopts a fully automatic method, for new drugs that appear, it is necessary to enter the drug information immediately, otherwise the model will be difficult to recognize and affect subsequent recognition operations. Model recognition requires a large amount of labeled data to be sent to the model for training. On the one hand, it is easy to cause the cost of the recognition model to be too high. On the other hand, there is a lack of a large number of labeled data sets in China. At the same time, the model must be trained and convergence judged to ensure that the model can achieve accurate recognition in large-scale data sets, which invisibly increases the high requirements for the model, making it difficult to apply in some hospitals. Summary of the invention

[0004] The purpose of this section is to summarize some aspects of embodiments of the present invention and briefly introduce some preferred embodiments. Some simplifications or omissions may be made in this section and the specification abstract and the invention title of this application to avoid blurring the purpose of this section, the specification abstract and the invention title, and such simplifications or omissions cannot be used to limit the scope of the present invention.

[0005] In view of the above existing problems, the present invention is proposed. Therefore, the present invention provides a drug checking and automatic dispensing method based on machine vision to solve the problems mentioned in the background technology.

[0006] In order to solve the above technical problems, the present invention provides the following technical solutions: a drug verification and automatic distribution method based on machine vision, comprising:

[0007] Determine a drug review area, collect drug images after being picked by a pharmacist in the drug review area through a collection device, obtain drug feature vectors based on the drug images, and store drug information after being picked by the pharmacist in a drug database;

[0008] Constructing a drug recognition model, inputting the drug feature vector into the drug recognition model, performing drug recognition, obtaining drug recognition information, and storing the drug recognition information in a drug recognition database;

[0009] The drug identification database is compared with the drug information database, and according to the comparison result, a choice is made to distribute the drug or to provide abnormal feedback on the drug information.

[0010] As a preferred solution of the machine vision-based drug checking and automatic dispensing method of the present invention, the step of determining the drug review area includes:

[0011] Consider the total length L of the drug conveyor belt t With conveyor belt speed v t and the drug’s residence time in the review area t r The minimum value of the drug review area determines the length L r ;

[0012] Consider the maximum height H of the drug max The height difference h between the maximum height of the collection equipment and the drug c , determine the height H of the review area r .

[0013] As a preferred solution of the machine vision-based drug checking and automatic dispensing method of the present invention, it also includes:

[0014] A positioning mark is set at the center of the drug review area, and the positions of the entrance and exit sensors in the review area are set by the positioning mark.

[0015] As a preferred solution of the machine vision-based drug checking and automatic dispensing method of the present invention, the drug image after being picked by the pharmacist is collected in the drug review area by a collection device, including:

[0016] According to the height H of the review arear and the maximum height H of the drug max Determine the height H of the acquisition device at the horizontal viewing angle c , and according to the horizontal viewing angle of the acquisition device and the width of the drug W p , get the distance D between the acquisition device and the conveyor belt from the side view c ;

[0017] According to the conveyor belt speed v t and the length of time L that the drug stays in the field of view of the collection device v , determine the interval time for each camera to collect drug images and the shooting interval Δt of the drug images.

[0018] As a preferred solution of the machine vision-based drug checking and automatic dispensing method of the present invention, the drug feature vector is obtained according to the drug image, including:

[0019] For the obtained drug feature vector V f ,The normalization method is used to standardize each component in the drug feature vector into interval form.

[0020] As a preferred solution of the drug verification and automatic distribution method based on machine vision of the present invention, wherein: constructing a drug recognition model, inputting the drug feature vector into the drug recognition model, performing drug recognition, and obtaining drug recognition information, including:

[0021] The components in the standardized drug feature vector are passed to the feature fusion layer, which performs weighted summation on each component in the feature vectors of different drugs, and passes the result of the weighted summation to the convolution layer to form a low-dimensional feature map;

[0022] The low-dimensional feature map is linearly combined through the feature expansion layer to generate a new drug feature vector, and the low-dimensional feature map and the new drug feature vector are fused by two fully connected layers. The output of the drug recognition model is mapped to a probability distribution using the Softmax activation function, and the probability value of each drug category is output. The category with the highest probability value is selected as the drug recognition result to obtain drug recognition information.

[0023] As a preferred solution of the machine vision-based drug checking and automatic dispensing method of the present invention, it also includes:

[0024] The drug identification information includes drug category ID, drug identification confidence and drug difference analysis.

[0025] As a preferred solution of the machine vision-based drug checking and automatic dispensing method of the present invention, it also includes:

[0026] The two fully connected layers contain n and m neurons respectively, and a Dropout layer is introduced after the fully connected layer. For each fusion operation of the low-dimensional feature map and the new drug feature vector, half of the neurons are randomly discarded.

[0027] As a preferred solution of the machine vision-based drug checking and automatic dispensing method of the present invention, the drug difference analysis includes:

[0028] By calculating the highest probability value and the second highest probability value of each drug category, a difference analysis result is generated, and the drug identification model is evaluated based on the difference analysis result.

[0029] As a preferred solution of the machine vision-based drug verification and automatic distribution method of the present invention, the drug identification database is compared with the drug information database, and according to the comparison result, drug distribution or abnormal feedback of drug information is selected, including:

[0030] According to the drug category ID, each record in the drug identification database is compared with the corresponding record in the drug information database. If the current drug identification confidence is equal to or lower than the average drug identification confidence, the currently compared drug information record is marked as "drug information abnormality", the drug distribution operation is stopped, and the pharmacist is notified to recheck the drug information; otherwise, the drug distribution operation is executed, the currently compared drug information record is marked as "drug information normal", and the current timestamp of the successfully distributed drug, the distributed patient information, and the current picking pharmacist information are stored in the drug management database.

[0031] Compared with the prior art, the invention has the following beneficial effects:

[0032] 1. The present invention reasonably sets the position and angle of the acquisition device in the drug review area, ensuring high-quality acquisition of drug images, so that the drug recognition model can more accurately obtain drug feature vectors, thereby improving the accuracy of drug recognition;

[0033] 2. The drug feature vectors are normalized and analyzed for differences, which improves the model's adaptability to different drug packaging and complex environments. Regardless of the diversity and similarity of drug packaging or changes in image acquisition conditions, the model can maintain a high recognition accuracy without frequent iterations, thereby ensuring the stability of drug verification;

[0034] 3. Through the semi-automated drug verification and distribution process, the records of the drug identification database and the drug information database can be automatically compared according to the drug identification information, and when the drug identification confidence is lower than the preset threshold, abnormal feedback will be issued in time to notify the pharmacist to recheck, thereby significantly reducing the drug picking errors caused by manual operations. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for describing the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work. Among them:

[0036] Figure 1 This is an overall flow chart of a method for checking and automatically dispensing medicines based on machine vision according to an embodiment of the present invention;

[0037] Figure 2 This is a comparison chart of system performance in complex drug recognition, exception handling and high-load operation of a drug verification and automatic distribution method based on machine vision according to an embodiment of the present invention;

[0038] Figure 3 A model analysis comparison diagram of drug recognition accuracy of a drug verification and automatic distribution method based on machine vision according to an embodiment of the present invention;

[0039] Figure 4 This is a comparison chart of system fault tolerance of a machine vision-based drug verification and automatic distribution method according to an embodiment of the present invention. DETAILED DESCRIPTION

[0040] In order to make the above-mentioned purposes, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are described in detail below in conjunction with the drawings of the specification. Obviously, the described embodiments are part of the embodiments of the present invention, but not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary persons in the art without creative work should fall within the scope of protection of the present invention.

[0041] In the following description, many specific details are set forth to facilitate a full understanding of the present invention, but the present invention may also be implemented in other ways different from those described herein, and those skilled in the art may make similar generalizations without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0042] Secondly, the term "one embodiment" or "embodiment" as used herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The term "in one embodiment" that appears in different places in this specification does not necessarily refer to the same embodiment, nor does it refer to a separate or selective embodiment that is mutually exclusive with other embodiments.

[0043] The present invention is described in detail with reference to schematic diagrams. When describing the embodiments of the present invention, for the sake of convenience, the cross-sectional diagrams showing the device structure will not be partially enlarged according to the general scale, and the schematic diagrams are only examples, which should not limit the scope of protection of the present invention. In addition, in actual production, the three-dimensional dimensions of length, width and depth should be included.

[0044] At the same time, in the description of the present invention, it should be noted that the directions or positional relationships indicated by the terms "upper, lower, inner and outer" are based on the directions or positional relationships shown in the drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific direction, be constructed and operated in a specific direction, and therefore cannot be understood as limiting the present invention. In addition, the terms "first, second or third" are only used for descriptive purposes and cannot be understood as indicating or implying relative importance.

[0045] In the present invention, unless otherwise clearly specified and limited, the terms "install, connect, connect" should be understood in a broad sense, for example: it can be a fixed connection, a detachable connection or an integral connection; it can also be a mechanical connection, an electrical connection or a direct connection, or it can be indirectly connected through an intermediate medium, or it can be the internal communication of two components. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0046] Example 1

[0047] Reference Figure 1 , which is the first embodiment of the present invention, provides a drug verification and automatic distribution method based on machine vision, comprising:

[0048] S1. Determine a drug review area, collect drug images after being picked by a pharmacist in the drug review area through a collection device, obtain drug feature vectors based on the drug images, and store drug information after being picked by the pharmacist in a drug database;

[0049] Furthermore, considering the total length L of the drug conveyor belt t With conveyor belt speed v t and the drug’s residence time in the review area t r The minimum value of the drug review area determines the length L r ;

[0050] It should be noted that the longer the drug stays in the review area (the longer the stay time, the slower the conveyor belt speed), the higher the quality of the image captured by the acquisition device, and the more accurate the drug feature vector will be. On the contrary, if the drug stays in the review area for a shorter time (the shorter the stay time, the faster the conveyor belt speed), the lower the quality of the image captured by the acquisition device, and the deviation of the drug feature vector will be too large. Since the traditional conveyor belt moves horizontally, v t It also affects the length of the review region; if v t If the speed is too fast, the length of the review area will be insufficient, and the drug may leave the review area before the image acquisition is completed, resulting in image acquisition errors or failures.

[0051] Specifically, the above description relationship is expressed as:

[0052] L r =min(L t ,v t ×t r )

[0053] It should be noted that this formula ensures that the length of the review area does not exceed the total length of the conveyor belt, and adapts to different conveying speeds and dwell times;

[0054] Furthermore, considering the maximum height H of the drug max The height difference h between the maximum height of the collection equipment and the drug c , determine the height H of the review area r ;

[0055] Specifically, the height H of the review area r It is expressed as:

[0056] H r =H max +2×h c

[0057] It should be noted that the total size of the review area can be determined by the height and length of the review area;

[0058] Furthermore, a positioning mark is set at the center of the drug review area, and the position of the entrance and exit sensors in the review area is set by the positioning mark;

[0059] Specifically, a positioning mark is set at the center of the drug review area to indicate:

[0060]

[0061] Among them, x c It is represented as the center position of the drug review area, and x0 is the review area L r×H r The starting position on the conveyor belt;

[0062] Specifically, the positions of the inlet and outlet sensors are expressed as:

[0063] x in =x c -δ

[0064] x out =x c +δ

[0065] Among them, δ is a small distance used to ensure the sensing accuracy of the sensor;

[0066] It should be noted that the entrance and exit sensors can accurately detect when the drugs enter or leave the review area to reduce manual misoperation. If there is no sensor, the collection device will miss the time when the drugs enter or leave the area, resulting in the collection device being mistakenly activated when the drugs have not reached the review area. It can also prevent multiple drugs from entering the review area at the same time, ensuring that the drugs processed each time are single;

[0067] Furthermore, according to the height H of the review area r and the maximum height H of the drug max Determine the height H of the acquisition device at the horizontal viewing angle c , and according to the horizontal viewing angle of the acquisition device and the width of the drug W p , get the distance D between the acquisition device and the conveyor belt from the side view c ;

[0068] Specifically, the height H of the acquisition device at the horizontal viewing angle c It is expressed as:

[0069]

[0070] Specifically, the distance D between the collection device and the conveyor belt from the side view c It is expressed as:

[0071]

[0072] Among them, α c Indicates the horizontal viewing angle;

[0073] Furthermore, according to the conveyor belt speed v t and the length of time L that the drug stays in the field of view of the collection device v , determine the interval time for each camera to collect drug images and the shooting interval Δt of the drug image;

[0074] Specifically, the interval time for collecting drug images It is expressed as:

[0075]

[0076] Specifically, the shooting interval Δt of the drug image is expressed as:

[0077]

[0078] Among them, N c The number of images captured by the camera;

[0079] Furthermore, the obtained drug feature vector V f ,Using the normalization method, each component in the drug feature vector is standardized into interval form;

[0080] Specifically, the drug feature vector V f It is expressed as:

[0081] V f ={S f ,C f ,T f}

[0082] Among them, S f represents the shape feature vector, C f represents the feature vector of the QR code, T f represents a text feature vector;

[0083] It should be explained that the components in each drug feature vector are represented by the corresponding drug feature vector, for example, S f ={v1,v2,v3,…,v n}, C f ={w1,w2,w3,…,w n}, T f ={z1,z2,z3,…,z n}; Each v n 、w n 、z n The range of the normalized interval form of the components is [0,1];

[0084] Furthermore, the drug information in the drug database is converted into a standard feature vector V s and the obtained drug feature vector V f Compare and determine whether the drug image is the same as the actual drug information by calculating the similarity between feature vectors;

[0085] Specifically, the similarity formula is expressed as:

[0086]

[0087] Among them, Sim(Vf ,V s ) is the similarity, ‖·‖ is expressed in Euclidean form, i.e., the size or length of the vector;

[0088] It should be noted that by considering the similarity, checking whether the drug in the drug image is a drug in the drug database prevents information errors when the drug information is entered into the drug database;

[0089] S2, constructing a drug recognition model, inputting the drug feature vector into the drug recognition model, performing drug recognition, obtaining drug recognition information, and storing the drug recognition information in a drug recognition database;

[0090] Furthermore, the components in the standardized drug feature vector are passed to the feature fusion layer, which performs weighted summation on each component in the feature vectors of different drugs, and passes the weighted summation result to the convolution layer to form a low-dimensional feature map;

[0091] It should be noted that the low-dimensional feature map is a representation of the drug feature vector after it has been compressed in the spatial dimension. Even if the size is compressed, the information remains unchanged. The advantage of this is that redundancy is reduced.

[0092] Exemplarily, weighted summation is performed on each component in the feature vectors of different drugs, for example: v1+w1+z1 or v1+w2..., and so on, but it should be noted that each component must be weighted summed until all components in the feature vector are summed, and the weighted summation must be at least two components, but they cannot be repeated, for example: v1+w1, where v1 and w1 have been weighted summed, they cannot continue to appear in the weighted summation of the remaining components;

[0093] It should be noted that by introducing the feature fusion layer, the model's perception of key drug features can be enhanced to reduce the interference of drug image noise, improve the robustness of the model, and enable it to better adapt to different application scenarios;

[0094] Specifically, three convolutional layers are used, each of which consists of multiple convolutional kernels with a size of 3×3. After each convolutional layer, a maximum pooling layer is added with a pooling window size of 2×2.

[0095] Furthermore, the low-dimensional feature map is linearly combined through the feature expansion layer to generate a new drug feature vector, and the low-dimensional feature map and the new drug feature vector are fused by two fully connected layers. The output of the drug recognition model is mapped to a probability distribution using the Softmax activation function, and the probability value of each drug category is output. The category with the highest probability value is selected as the drug recognition result to obtain drug recognition information;

[0096] Specifically, the generated new drug feature vector V ext for:

[0097] V ext =V conv +V fusion

[0098] Among them, V conv is the output result of the convolution layer, V fusion is the weighted sum result;

[0099] Specifically, the drug identification information includes drug category ID, drug identification confidence, and drug difference analysis;

[0100] Furthermore, by calculating the highest probability value and the second highest probability value of each drug category, a difference analysis result is generated, and the drug recognition model is evaluated through the difference analysis result;

[0101] Specifically, the difference analysis result formula is expressed as:

[0102] ΔP=P max -P second

[0103] It should be noted that ΔP can be used to determine the confidence value of the drug recognition model for a certain drug category; if ΔP is large, it means that the model is very confident in the recognition result, that is, the drug recognition is correct; if ΔP is small, it means that there may be confusion, and the model is not sure about the recognition result, that is, there is an anomaly in the drug recognition;

[0104] Specifically, the two fully connected layers contain n and m neurons respectively, and a Dropout layer is introduced after the fully connected layer. For each fusion operation of the low-dimensional feature map and the new drug feature vector, half of the neurons are randomly discarded;

[0105] It should be noted that randomly dropping neurons can prevent the drug recognition model from over-relying on certain specific neurons (drug feature vectors with extremely high feature similarity), thereby reducing overfitting. Through this randomness, the robustness of the model is further improved, and the model will not perform poorly due to image or data noise. The Dropout layer only uses half of the neurons for feature fusion each time, so that the model can still make accurate predictions when facing new drug information.

[0106] S3, comparing the drug identification database with the drug information database, and selecting to distribute the drug or to provide abnormal feedback on the drug information according to the comparison result;

[0107] Furthermore, according to the drug category ID, each record in the drug identification database is compared with the corresponding record in the drug information database. If the current drug identification confidence is equal to or lower than the average drug identification confidence, the currently compared drug information record is marked as "drug information abnormality", the drug distribution operation is stopped, and the pharmacist is notified to recheck the drug information; otherwise, the drug distribution operation is executed, the currently compared drug information record is marked as "drug information normal", and the current timestamp of the successfully distributed drug, the distributed patient information, and the current picking pharmacist information are stored in the drug management database;

[0108] It should be noted that by creating a drug management database, the current timestamp of successful drug distribution, patient information, and information of the current picking pharmacist can be recorded, which facilitates subsequent traceability of the drugs.

[0109] Example 2

[0110] Reference Figures 2 to 4 , which is the second embodiment of the present invention, and this embodiment provides a method for drug verification and automatic distribution based on machine vision, including: This embodiment adopts a simulation experiment method to compare the performance of the fully automatic method, the traditional manual operation method and the semi-automatic method proposed in the present invention in drug verification and distribution, especially in terms of processing speed, recognition accuracy, error rate, system fault tolerance, and exception handling capability; the simulation experiment environment is carried out in a hospital pharmacy, involving multiple drug categories, including common drugs, similarly packaged drugs, and special drugs (such as cold chain drugs); wherein, the experiment simulates the system performance under high-load operation state, so as to evaluate its effectiveness and reliability in practical applications;

[0111] System configuration and parameter setting:

[0112] Fully automatic system, the system configuration includes high-definition camera, robotic arm, automatic distribution module and real-time database; the camera is installed above the conveyor belt, and the system recognizes and distributes drugs by capturing drug images; manual operation system, two experienced pharmacists manually check drug information and distribute drugs through the robotic arm; semi-automatic system (the present invention), the semi-automatic system of the present invention combines machine vision, manual operation and difference analysis, the system automatically processes high-confidence drugs, and low-confidence drugs are sent to the manual review station;

[0113] Data collection and processing:

[0114] Processing speed, recording the processing speed of each system under different load conditions, including normal load (50 units / minute), high load (100 units / minute) and extreme load (150 units / minute); where unit refers to a single drug or drug package, and each "unit" represents an independent drug item processed by the system; minute refers to the time unit, that is, the number of drugs processed by the system in one minute; recognition accuracy and error rate, recording the recognition accuracy and error rate of the system for different types of drugs, and comparing the performance of each system; system fault tolerance and exception handling, testing the system's processing capabilities under abnormal conditions (such as damaged drug labels, missing information, etc.), recording the exception handling time and system recovery time;

[0115] Experimental data processing:

[0116] The simulation experiment collects data based on 8 hours of continuous operation. The data will be divided into three types: common drugs (60%), similar packaged drugs (30%) and special drugs (10%) for statistics. The processing data of each drug includes recognition time, recognition accuracy, error rate, abnormal processing time, etc. The simulation results refer to Table 1;

[0117] Table 1

[0118]

[0119] From Table 1, we can see that, firstly, in terms of processing speed, although the fully automatic system performs best under normal load, under high load and extreme load, due to processing too many drug units, the accuracy of identification and processing begins to decline, and the error rate of the system increases. The manual operation system shows that the processing capacity is seriously insufficient under extreme load, and the processing speed drops from 30 units / minute under normal load to 20 units / minute under extreme load. The semi-automatic system of the present invention shows a stable processing speed under various load conditions, especially under extreme load, the processing speed reaches 100 units / minute, which is far superior to the performance of the manual operation system. Secondly, in terms of recognition accuracy, the recognition accuracy of the fully automatic system for common drugs reaches 98.5%, but when processing similar packaged drugs and special drugs, the accuracy drops to 95.0% and 94.0% respectively. The recognition accuracy of the manual operation system is relatively low, especially when processing similar packaged drugs, it is only 92.0%. The semi-automatic system of the present invention combines machine recognition with manual review, and the recognition accuracy of common drugs reaches 99.0%, and the recognition accuracy of similar packaged drugs and special drugs reaches 99.0%. The accuracy rates reached 97.5% and 96.5% respectively, which were significantly better than the fully automatic system and the manual operation system. This shows that by manually reviewing low-confidence drugs, the system of the present invention effectively improves the recognition accuracy, especially in the identification of complex drugs. In addition, in terms of error rate, the semi-automatic system of the present invention also shows advantages, especially when dealing with similar packaged drugs and special drugs, the error rates are 2.5% and 3.5% respectively, which are significantly lower than the error rates of the fully automatic system and the manual operation system. This low error rate is due to the manual intervention measures of the system when the recognition confidence is low, which reduces the misjudgment caused by the characteristics of complex drugs. Finally, in terms of exception handling and system recovery capabilities, the semi-automatic system of the present invention shows excellent performance. When abnormal situations such as drug label damage or missing database information occur, the average exception handling time of the system is 2 seconds, which is significantly better than the 3.5 seconds of the fully automatic system and the 10 seconds of the manual operation system. At the same time, the average recovery time of the system is 2 minutes per hour, which is also better than the manual operation system, indicating that the system of the present invention is not only more efficient in dealing with sudden exceptions, but also can resume normal operation faster.

[0120] In summary, the semi-automatic drug checking and dispensing system of the present invention combines machine vision with manual review, and has both the high efficiency of the fully automatic system and the accuracy of the manual operation system. It has obvious advantages in complex drug identification, abnormal processing and high-load operation. Figure 2 ; It can be seen that the system of the present invention not only improves the recognition accuracy, but also Figure 3 , which reduces the error rate and significantly enhances the system's fault tolerance, such as Figure 4 As shown, it has extremely high reliability and practical value in practical applications.

[0121] Those skilled in the art will appreciate that the embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the application can adopt the form of complete hardware embodiments, complete software embodiments, or embodiments in combination with software and hardware. Moreover, the application can adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program codes. The scheme in the embodiments of the present application can be implemented in various computer languages, for example, object-oriented programming language Java and literal scripting language JavaScript, etc.

[0122] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0123] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0124] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0125] Although the preferred embodiments of the present application have been described, those skilled in the art may make other changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications falling within the scope of the present application.

[0126] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. A drug verification and automatic distribution method based on machine vision, characterized in that: include: Determine a drug review area, collect drug images after being picked by a pharmacist in the drug review area through a collection device, obtain drug feature vectors based on the drug images, and store drug information after being picked by the pharmacist in a drug database; Constructing a drug recognition model, inputting the drug feature vector into the drug recognition model, performing drug recognition, obtaining drug recognition information, and storing the drug recognition information in a drug recognition database; The drug identification database is compared with the drug information database, and according to the comparison result, a choice is made to distribute the drug or to provide abnormal feedback on the drug information.

2. The machine vision-based drug verification and automatic distribution method according to claim 1, characterized in that: Identify drug review areas, including: Consider the total length L of the drug conveyor belt t With conveyor belt speed v t and the drug’s residence time in the review area t r The minimum value of the drug review area determines the length L r ; Consider the maximum height H of the drug max The height difference h between the maximum height of the collection equipment and the drug c , determine the height H of the review area r .

3. The machine vision-based drug verification and automatic distribution method according to claim 2, characterized in that: Also includes: A positioning mark is set at the center of the drug review area, and the positions of the entrance and exit sensors in the review area are set by the positioning mark.

4. The machine vision-based drug verification and automatic distribution method according to claim 2 or 3, characterized in that: The collection device collects images of the medicines picked by the pharmacist in the medicine review area, including: According to the height H of the review area r and the maximum height H of the drug max Determine the height H of the acquisition device at the horizontal viewing angle c , and according to the horizontal viewing angle of the acquisition device and the width of the drug W p , get the distance D between the acquisition device and the conveyor belt from the side view c ; According to the conveyor belt speed v t and the length of time L that the drug stays in the field of view of the collection device v , determine the interval time for each camera to collect drug images and the shooting interval Δt of the drug images.

5. The machine vision-based drug checking and automatic distribution method according to claim 4, characterized in that: Acquiring a drug feature vector according to the drug image includes: For the obtained drug feature vector V f ,The normalization method is used to standardize each component in the drug feature vector into interval form.

6. The machine vision-based drug checking and automatic distribution method according to claim 5, characterized in that: Construct a drug recognition model, input the drug feature vector into the drug recognition model, perform drug recognition, and obtain drug recognition information, including: The components in the standardized drug feature vector are passed to the feature fusion layer, which performs weighted summation on each component in the feature vectors of different drugs, and passes the result of the weighted summation to the convolution layer to form a low-dimensional feature map; The low-dimensional feature map is linearly combined through the feature expansion layer to generate a new drug feature vector, and the low-dimensional feature map and the new drug feature vector are fused by two fully connected layers. The output of the drug recognition model is mapped to a probability distribution using the Softmax activation function, and the probability value of each drug category is output. The category with the highest probability value is selected as the drug recognition result to obtain drug recognition information.

7. The machine vision-based drug checking and automatic distribution method according to claim 6, characterized in that: Also includes: The drug identification information includes drug category ID, drug identification confidence and drug difference analysis.

8. The machine vision-based drug checking and automatic distribution method according to claim 6, characterized in that: Also includes: The two fully connected layers contain n and m neurons respectively, and a Dropout layer is introduced after the fully connected layer. For each fusion operation of the low-dimensional feature map and the new drug feature vector, half of the neurons are randomly discarded.

9. The machine vision-based drug checking and automatic distribution method according to claim 6, characterized in that: The drug difference analysis includes: By calculating the highest probability value and the second highest probability value of each drug category, a difference analysis result is generated, and the drug identification model is evaluated based on the difference analysis result.

10. The machine vision-based drug checking and automatic distribution method according to claim 1 or 6, characterized in that: Comparing the drug identification database with the drug information database, and selecting to distribute the drug or to provide abnormal feedback on the drug information according to the comparison result, including: According to the drug category ID, each record in the drug identification database is compared with the corresponding record in the drug information database. If the current drug identification confidence is equal to or lower than the average drug identification confidence, the currently compared drug information record is marked as "drug information abnormality", the drug distribution operation is stopped, and the pharmacist is notified to recheck the drug information; otherwise, the drug distribution operation is executed, the currently compared drug information record is marked as "drug information normal", and the current timestamp of the successfully distributed drug, the distributed patient information, and the current picking pharmacist information are stored in the drug management database.

Images