Multi-object drug traceability code identification and checking method and system based on visual identification

Abstract

The invention provides a multi-object drug traceability code identification and checking method and system based on visual identification. The method comprises the following specific steps: obtaining a multi-angle drug package image of a multi-object drug based on visual identification; extracting a traceability code area on the multi-angle medicine packaging image; and decoding the traceability code in the traceability code area to obtain traceability code information, and checking the traceability code information with the drug database. Through a multi-angle image acquisition technology and an intelligent algorithm and in combination with bar code automatic identification and drug information matching functions, drug checking and traceability code identification can be rapidly and accurately completed, and the manual operation intensity and the error rate are significantly reduced.

Description

Technical Field

[0001] The present invention belongs to the technical field of drug management and artificial intelligence automation, and particularly relates to a method and system for identifying and verifying multi-object drug traceability codes based on visual recognition. Background Art

[0002] With the rapid development of the medical industry, the need for informatization and intelligentization of drug management is becoming increasingly urgent. The construction of a drug verification and traceability system has become an important task to ensure the safety of patients' medication. The drug traceability system is required to cover the entire life cycle of drug production, circulation, and use. In particular, through the "one drug, one code" drug traceability code, the transparent management of the entire drug chain is realized. This measure not only helps to strengthen drug quality supervision and prevent the circulation of counterfeit drugs, but also is a basic technical means to improve the drug management level of medical institutions.

[0003] In medical institutions, the outpatient drug dispensing and distribution link is an important gateway to ensure the safety of patients' medication. In actual work, pharmacists need to simultaneously complete the verification of drugs and the scanning of traceability codes, including confirming the types, dosages, quantities of drugs, and whether they are consistent with the prescription information. This link is directly related to whether patients can take drugs safely and correctly. However, with the continuous increase in the number of outpatients, the complexity of drug types, and the high-intensity dispensing tasks, the traditional manual verification and traceability code scanning process has gradually revealed significant problems.

[0004] First of all, the operations of manual verification and scanning traceability codes one by one are time-consuming and inefficient. Especially during peak hours, pharmacists need to frequently turn over medicine boxes to find the traceability code facing the barcode scanner and scan and verify one by one, which greatly increases the work burden. Secondly, there are relatively high error risks in manual operations, including errors in drug types, dosage verification errors, quantity omissions, or traceability code recognition failures. These problems may lead to patients taking incorrect drugs, missing necessary medications, or even triggering serious medication safety incidents. In addition, barcode recognition is often affected by external factors such as various medicine box specifications, non-fixed barcode positions, and reflections and stains, further reducing the verification efficiency and accuracy.

[0005] To solve the above problems, there is an urgent need for a set of efficient and intelligent drug verification and traceability system to meet the actual operation requirements. Summary of the Invention

[0006] Aiming at the deficiencies of traditional manual operations in the prior art, the present invention proposes a method and system for identifying and verifying multi-object drug traceability codes based on visual recognition. Through multi-angle image acquisition technology and intelligent algorithms, combined with the function of automatic barcode recognition and drug information matching, it can quickly and accurately complete the verification of drugs and the recognition of traceability codes, significantly reducing the intensity of manual operations and the error rate.

[0007] To achieve the above object, the present invention provides the following technical solution: A method for identifying and verifying multi-object drug traceability codes based on visual recognition, the specific steps are as follows: Obtain multi-angle drug packaging images of multi-object drugs based on visual recognition; Extract the traceability code area on the multi-angle drug packaging image; Decode the traceability code in the traceability code area to obtain traceability code information, and verify the traceability code information with the drug database.

[0008] Further, in the step of extracting the traceability code area on the multi-angle drug packaging image, the specific steps are as follows: 1) Place the medicine boxes of multi-object drugs flat, and obtain multi-angle drug packaging images of multi-object drugs based on visual recognition; 2) Use the region growing segmentation method to segment the side areas of each stacked medicine box in the obtained multi-angle drug packaging image to obtain multiple medicine box side image areas, then use median filtering to filter out small-size noise, perform a connection operation to extract each connected domain of the segmented image, and use size threshold parameters to screen out the actual medicine box area image; 3) Transmit the actual medicine box area image to the SSD model to extract the traceability code area image; 4) Perform image affine transformation and binarization processing on the extracted traceability code area image to obtain a corrected traceability code area image.

[0009] Further, in the step of decoding the traceability code in the traceability code area to obtain traceability code information and verifying the traceability code information with the drug database: Use the Zebra crossing decoding algorithm to decode the traceability code in the corrected traceability code area image to obtain traceability code information, and the traceability code is a one-dimensional barcode or a two-dimensional code.

[0010] Further, an optical imaging module and a motion control module are used to implement the step of obtaining multi-angle drug packaging images of multi-object drugs based on visual recognition. Among them, the motion control module and the optical imaging module are both arranged on the operating table, and the multi-object drugs are placed on the motion control module. The multi-angle drug packaging images of multi-object drugs are obtained through the optical imaging module. Specifically: The optical imaging module includes an industrial camera, a fixed-focus lens with a large depth of field, a ring shadowless light source, and a light source controller; the motion control module includes a motion controller and a precision turntable; the industrial camera uses a fixed-focus lens with a large depth of field and is arranged on both sides of the precision turntable through a two-dimensional adjustment bracket, and the light source controller is connected to the ring shadowless light source to control the illumination intensity of the industrial camera's photographing environment; the precision turntable is used to place multi-object drugs, and the motion controller is connected to the precision turntable to control the rotation of the precision turntable to obtain multi-angle drug packaging images of multi-object drugs.

[0011] Furthermore, the industrial camera is connected to a computer to obtain multi-angle drug packaging images of multi-object drugs; the motion controller is connected to the computer to control the adjustment direction of the precision turntable; the light source controller is connected to the computer to control the illumination intensity of the industrial camera's photographing environment; the industrial camera is connected to a user-side triggering device to control the industrial camera to take pictures.

[0012] The present invention also provides a multi-object drug traceability code recognition and verification system based on visual recognition, including: A drug packaging visual inspection system for obtaining multi-angle drug packaging images of multi-object drugs based on visual recognition; A drug traceability code recognition system for extracting the traceability code area on the multi-angle drug packaging image; A drug traceability code verification system for decoding the traceability code in the traceability code area to obtain traceability code information and verifying the traceability code information with the drug database.

[0013] Furthermore, the drug packaging visual inspection system includes an optical imaging module and a motion control module. The motion control module and the optical imaging module are both arranged on the operating table. The multi-object drugs are placed on the motion control module, and multi-angle drug packaging images of the multi-object drugs are obtained through the optical imaging module. Specifically: The optical imaging module includes an industrial camera, a fixed-focus lens with a large depth of field, a ring shadowless light source, and a light source controller; the motion control module includes a motion controller and a precision turntable; the industrial camera uses a fixed-focus lens with a large depth of field and is arranged on both sides of the precision turntable through a two-dimensional adjustment bracket, and the light source controller is connected to the ring shadowless light source to control the illumination intensity of the industrial camera's photographing environment; the precision turntable is used to place multi-object drugs, and the motion controller is connected to the precision turntable to control the rotation of the precision turntable to obtain multi-angle drug packaging images of multi-object drugs; the industrial camera is connected to a computer to obtain multi-angle drug packaging images of multi-object drugs; the motion controller is connected to the computer to control the adjustment direction of the precision turntable; the light source controller is connected to the computer to control the illumination intensity of the industrial camera's photographing environment; the industrial camera is connected to a user-side triggering device to control the industrial camera to take pictures.

[0014] The present invention also provides a terminal device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps of the above-mentioned method for identifying and verifying multi-object drug traceability codes based on visual recognition are implemented.

[0015] The present invention also provides a computer-readable storage medium storing a computer program, and when the computer program is executed by a processor, the steps of the above-mentioned method for identifying and verifying multi-object drug traceability codes based on visual recognition are implemented.

[0016] The present invention also provides a computer program product including a computer program, and when the computer program is executed by a processor, the steps of the above-mentioned method for identifying and verifying multi-object drug traceability codes based on visual recognition are implemented.

[0017] Compared with the prior art, the present invention has at least the following beneficial effects: A method for identifying and verifying multi-object drug traceability codes based on visual recognition proposed by the present invention can comprehensively capture the traceability code information on the drug package through multi-angle image acquisition technology, avoiding the problems of traceability code occlusion or unclear recognition caused by single-angle shooting. Combined with intelligent algorithms, the method can accurately extract the traceability code area on the drug package image and quickly and accurately analyze the traceability code information through advanced decoding technology. Comparing the analyzed traceability code information with the drug database ensures the accuracy and consistency of drug information and effectively prevents the occurrence of drug dispensing errors.

[0018] Traditional drug verification and traceability work often rely on manual operations, which not only have a large labor intensity but also are prone to errors. The method of the present invention reduces the cumbersome steps of manually turning the medicine box and manually scanning the code through automated processing, and can simultaneously scan the traceability codes of multiple drugs, significantly reducing the manual operation intensity and realizing the rapid processing of drug dispensing and distribution. The automated verification and recognition technology effectively reduces human operation errors, reduces the dispensing error rate, and improves the safety guarantee level of patients' medication.

[0019] Manual code scanning is prone to missed scanning or incorrect recognition of the traceability code due to insufficient light, improper angle, or incorrect operation. The method of the present invention is based on machine vision technology, which can achieve full-coverage scanning and accurate recognition of drug traceability codes, ensuring the integrity and accuracy of traceability information. By using image processing technologies such as region growing segmentation method, median filtering, connection operation, and size threshold parameter screening, the actual medicine box area image can be effectively extracted, further improving the accuracy of traceability code recognition. By adopting the SSD model and Zebra crossing decoding algorithm, one-dimensional barcodes or two-dimensional codes can be decoded efficiently and accurately, ensuring the integrity and accuracy of traceability information. Complete traceability information helps to monitor and trace the entire process of drug circulation, promptly discover and handle drug quality problems, and ensure the safety of patients' medication.

[0020] The system of the present invention integrates a drug packaging visual inspection system, a drug traceability code recognition system, and a drug traceability code verification system, realizing fully automated processing from image acquisition to traceability code verification. It can be deeply integrated with hospital information management systems (such as HIS, LIS, etc.), achieving the sharing and interconnection of drug information, and providing technical support for the intelligent upgrading of the medical industry. The popularization and application of this method contribute to improving the drug management level and service quality of the entire medical industry, and promoting the process of medical intelligence development. At the same time, it also has important significance for quality supervision and data tracking in the drug circulation process, and helps to build a safer and more efficient drug management system. Description of the Drawings

[0021] Figure 1 is the drug traceability code recognition system; Figure 2 is the schematic diagram of the drug traceability code visual inspection system; Figure 3 is the model diagram of the drug traceability code visual inspection system; Figure 4 is the information processing flow; Figure 5 is the SSD network model structure; Figure 6 is the implementation process of the invention; Figure 7 are the detection and processing results of multi-object drug traceability codes, (a) initial image acquisition; (b) image preprocessing; (c) traceability code detection and positioning; (d) traceability code correction and decoding verification; In the drawings: 1, precision turntable; 2, optical imaging system; 3, light source controller; 4, user-side trigger device; 5, desktop computer; 6, operation table; 7, two-dimensional adjustment frame. Detailed Embodiment

[0022] The present invention will be further described below in conjunction with the accompanying drawings and specific embodiments.

[0023] The present invention provides a method and system for identifying and verifying multi-object drug traceability codes based on visual recognition. The composition and operation logic of the system are as Figure 1 shown. The outpatient pharmacist, as the main interaction object of the system, plays an important role and mainly performs tasks such as drug verification, loading drugs into the system, and operating the traceability code scanning software. The multi-specification drugs are loaded by the pharmacist into the intelligent visual detection system for image acquisition and then transmitted to the desktop computer. Subsequently, the traceability code scanning software performs image preprocessing, barcode detection, barcode positioning, and traceability code decoding on the collected multi-perspective drug images. Finally, the identified multi-object and multi-specification drug traceability code information is compared with the drug database to achieve intelligent verification, and the results are stored on the local disk for archiving and uploaded to the hospital information system for statistics at the same time.

[0024] Based on the intelligent machine vision detection technology, the key components and system working principle of a multi-object drug intelligent verification and traceability code identification system of the present invention are as Figure 2 shown: 1) The desktop computer serves as the control module, the core of data processing and data interaction, and undertakes the global control operation tasks.

[0025] 2) The optical imaging module consists of an industrial camera, a large-depth-of-field fixed-focus lens, a ring shadowless light source, and a light source controller. The lens is installed on the industrial area array camera to image the multi-object and multi-specification drug packages from multiple angles, obtain the appearance image containing the traceability code, and send it to the computer for processing and decoding through the Gigabit Ethernet Gige interface. The light source controller is controlled by the computer to drive the ring shadowless light source to provide illumination for the imaging system, and obtain a clear traceability code image at a free angle.

[0026] 3) The motion control module consists of a motion controller and a precision turntable. The system drives various types of multi-object drugs to rotate in space to assist the optical imaging system in collecting the traceability codes on the outer packages of drugs placed at any angle. Among them, the motion controller communicates with the computer through the USB serial port 485 protocol to drive the precision turntable to perform rotational motion.

[0027] 4) The user-side triggering device can be used as an outpatient pharmacist controller to trigger the system to automatically take pictures and scan codes through physical buttons, improving the processing efficiency. It is also possible to achieve convenient system control and processing without professional computer operation knowledge.

[0028] The fully automatic visual inspection system for drug traceability codes of the present invention can achieve efficient and accurate identification of traceability codes in scenarios with different sizes and specifications, different printing positions, and free placement postures. At the same time, it automatically stores on the local disk and synchronizes with the hospital information system without manual intervention. The advantages of this system are its compact layout structure and large imaging depth of field, which are suitable for batch scanning application scenarios at outpatient pharmacy windows and large pharmacy windows.

[0029] As Figure 3 shown, a desktop computer 5, a user-side trigger device 7, an optical imaging module, and a precision turntable 1 form a visual inspection system for drug packaging. A precision turntable 1 is installed in the center of the operation table 6 and is driven by a driver to rotate the freely placed multi-specification drugs, cooperating with the optical imaging system 2 for image acquisition. Then, two sets of optical imaging systems 2 composed of industrial area array cameras, large-depth-of-field lenses, and ring shadowless light sources are installed on both sides of the precision turntable 1 to obtain drug appearance images from multiple angles. Each imaging field of view angle reaches 120°, so the turntable only needs to rotate at a single fixed angle to achieve barcode recognition in any posture of the drug. The light source controller 3 is set on the left side of the operation table 6 and is used to manually fine-tune the illumination intensity and adjust it to the best imaging effect during the debugging process. The physical trigger 4 is set on the right side of the operation table 6 and is used for the system startup and shutdown functions. The desktop computer 5 is placed inside the operation table 6, and its display specifically shows the scanning information of the traceability code and the statistical information of the drugs, facilitating human-computer interaction. The advantages of the traceability code detection system designed by the present invention are its compact space, accurate recognition, high detection efficiency, and at the same time, it can be applied to the detection of freely placed drugs with multiple specifications and multiple objects.

[0030] Preferably, the industrial area array camera is installed on a two-dimensional adjustment frame 8 and can be finely adjusted in the X and Y directions.

[0031] The multi-angle drug packaging images collected by the visual inspection system for drug traceability codes are input into the image processing system for preprocessing to obtain the barcode areas of each drug packaging box, eliminate redundant image information, enhance the image quality, and improve the scanning efficiency and accuracy. As Figure 4 shown, the specific steps are as follows: 1) In the image preprocessing link, mainly perform image segmentation, image filtering, and enhancement processing; First, perform image segmentation. The present invention uses the region growing segmentation method to segment the side regions of each stacked drug packaging box in the original image I to obtain multiple side image regions (ROIs) of the drug boxes, I m1 , I m2, … I mnThis method can adapt to different lighting conditions. Since the free-placement drug traceability code may be printed on one of the four sides of the drug packaging box, through rotation and multi-view imaging, the system can collect the regional images of each side of each drug packaging box. Therefore, m ∈[1,2,3,4] is the side of the medicine box, and n∈[1,2,3… N is the number of drugs to be tested, and it is necessary to satisfy Σ L n < H , H is the field of view height of the imaging system, L n is the height of each medicine box.

[0032] Subsequently, image filtering is performed to eliminate the noise introduced by image segmentation. Here, median filtering median_image is used to filter out small-size noise, and the filtering window is set to 3×3 pixels.

[0033] Then, the connection operation is performed to extract each connected region of the segmented image, and the actual medicine box region is screened out using the size threshold parameter. Finally, histogram equalization is used to correct the sub-region I m1 , I m2, … I mn gray-scale distribution to improve the contrast between the traceability code and the background texture.

[0034] 2) Use the Single Shot MultiBox Detector (SSD) model to implement the detection and positioning of the target traceability code, and extract the traceability code region; Single Shot MultiBox Detector (SSD) is an object detection method based on convolutional neural network (CNN). SSD mainly realizes the detection of objects by performing convolutional operations on feature maps of different scales. It generates candidate regions through the Region Proposal Network (RPN) and further classifies and regresses. The advantage of SSD lies in its single-stage (SingleShot) structure, which can complete the detection tasks of all objects in one forward propagation, greatly improving the detection speed. SSD utilizes the multi-scale characteristics of convolutional neural networks to detect objects of different scales through multiple feature maps, and performs bounding box regression and classification. SSD does not rely on a complex region proposal generation process, but directly predicts candidate boxes at different positions, and can achieve high-quality object localization and classification in a short time. The architecture of SSD can be summarized into three key components, as Figure 5 shown: 1) Feature Extraction Network: The feature extraction network of SSD uses VGG16 as its backbone network. Its structure is simple and the number of parameters is moderate. Through multiple convolutional layers, it extracts multi-level feature maps of the image, providing rich semantic information.

[0035] 2) Multi-scale Feature Maps: SSD not only relies on the high-level features of the image. It enhances the detection ability for objects of different sizes by extracting feature maps at different scales (different resolutions). SSD generates prediction results on convolutional layers at multiple levels. The shallower layers are responsible for detecting smaller objects, while the deeper layers are responsible for detecting larger objects.

[0036] 3) Detection Head: On each layer of the feature map, SSD uses convolutional layers to predict the class and bounding box of the object. The outputs of these convolutional layers are used to regress the coordinates of each candidate box and predict the class to which the object belongs through a softmax layer. In this way, SSD can perform position regression and class classification simultaneously at each scale, thus achieving efficient object detection and finally completing the barcode detection and localization of multi-object drug images.

[0037] Localization Method: SSD introduces default boxes, which are predefined and cover different regions of the image. Each default box has different aspect ratios and scales for matching the shapes and sizes of various objects. For each default box, SSD predicts the exact position of the object through a regression task, and the regression content includes the four coordinates of the barcode information bounding box. The non-maximum suppression (NMS) algorithm is used to calculate the overlapping regions between the barcode target search boxes, removing those boxes with high intersection-over-union and low confidence, and outputting the optimal detection results. Class prediction is achieved through a softmax classifier, and the model outputs the probability distribution of each class for each default box. A multi-task loss function is used to optimize the model, including regression loss (position regression) and classification loss (class prediction). The regression loss uses the smooth L1 loss function, while the classification loss uses the cross-entropy loss function.

[0038] (1) The loss function is as shown in Equation 1, where y , y ’ are the regression results and the ground truth, SmoothL 1 is the regression loss, Cross_ entropy is the classification loss function, α , β are the weighting coefficients respectively.

[0039] Model Training and Optimization: The model training process combines the multi-task loss of location regression and classification, and its training objective is to minimize the weighted sum of the regression loss and the classification loss. During the training process, data augmentation techniques (such as flipping, rotation, scaling, etc.) are used to increase the robustness of the model to adapt to different object scales, poses, and background noises. SSD also uses the positive and negative sample assignment strategy during training. To ensure that the model can fully learn the classification and regression information of the target, the positive and negative samples are usually balanced during training to avoid excessive background boxes affecting the model performance.

[0040] 3) Perform image affine transformation and binarization on the extracted trace code area to correct the effects of inclination, scaling, flipping, etc., and provide effective information for barcode decoding; After completing the detection and positioning of the drug trace code, the system can accurately locate the barcode area (coordinate information and size information). At this time, since the drug is in a freely placed state, the trace code image has situations such as inclination, scaling, and flipping. Therefore, in this link, image affine transformation is used to correct the trace code.

[0041] Let the image of the trace code area before correction I c The coordinate system distribution is x , y , and the image after correction I c The coordinate system is u , v , then the image correction link can be represented by Equation (2).

[0042] (2) In the formula, t x , t y is the image offset, s x , s y is the image scaling factor, θ is the image rotation amount, N is the flipping amount, d x , d y is the image shear amount. Through the above correction, the drug package trace code is corrected to a consistent spatial distribution. After completion of the correction, binarization processing of the trace code pattern and the background is performed to improve the barcode contrast and thus improve the decoding rate.

[0043] 4) Decode the trace code based on the Zebra crossing decoding algorithm to obtain the final result and upload it to the information system.

[0044] The Zebra Crossing method is a barcode decoding method based on traditional image processing and pattern recognition technologies. Its core principle relies on the structural features (black and white stripes or matrices) and encoding specifications of barcodes. The principle of this method is as follows: The Zebra Crossing method uses multiple strategies to detect the barcode area: 1) Project the horizontal or vertical pixel values of the grayscale image and count the changes in brightness values for each row or column. The barcode area shows periodic peak-valley fluctuations. By analyzing the frequency and amplitude of the fluctuations, the boundaries of the barcode can be located. 2) Pattern matching and feature extraction: Use stripe patterns (such as specific stripe sequences for start and stop characters) to match the boundaries of one-dimensional barcodes. Detect the locators in two-dimensional barcodes and confirm the boundaries and orientation of the two-dimensional barcode through geometric relationships. One-dimensional barcodes are composed of black and white stripes, and the width and spacing of each stripe are encoded as numbers or characters. Two-dimensional barcodes consist of a matrix of multiple black and white small squares, and the state of each module represents binary data, usually embedded with redundant information for error correction. The decoding process of the Zebra Crossing method is as follows: 1) One-dimensional barcode decoding: In the stripe parsing step, scan the black and white stripes in the barcode area, calculate the width of each stripe, and map it to a standardized ratio sequence. Pattern matching: According to the encoding rules of barcode types (such as UPC, EAN, etc.), map the stripe sequence to numbers or characters. In the check digit verification step, the check digit of the one-dimensional barcode is calculated to verify the integrity of the decoding result.

[0045] 2) Two-dimensional barcode decoding: The algorithm grids the two-dimensional barcode area, converts each small module into binary data, and restores the binary data to text or a URL according to the encoding standard of the two-dimensional barcode (such as the version and format information of the QR code). Finally, use the embedded error correction code (such as the Reed-Solomon algorithm) to repair the possibly damaged data area.

[0046] In summary, the multi-object drug traceability code detection system of the present invention has the following advantages: 1) Through the intelligent verification and recognition technology based on machine vision, it can effectively reduce human operation errors, reduce the dispensing error rate, and significantly improve the safety guarantee level of patients' medication. 2) Through multi-angle image acquisition and automatic recognition technology, the traceability codes of multiple drugs can be scanned simultaneously, eliminating the cumbersome steps of flipping the medicine box and manual operation, and greatly improving the overall efficiency of drug dispensing and distribution. 3) By reducing repetitive and high-intensity manual operations, the automated system reduces the work fatigue of pharmacists, optimizes the work process, enabling pharmacists to devote more energy to high-value-added services such as drug consultation. 4) Manual barcode scanning is prone to missed scanning or incorrect recognition of traceability codes due to insufficient light or misoperation. The system based on machine vision can achieve full-coverage scanning and accurate recognition of drug traceability codes, ensuring the integrity and accuracy of traceability information.

[0047] Example 1: A method for identifying and verifying multi-object drug traceability codes based on visual recognition according to the present invention, as Figure 6 shown, includes the following steps: Step 1, collect multi-angle images of multi-object and multi-specification drugs; Step 101, the outpatient pharmacist stacks the drugs to be scanned on the turntable of the workbench. The drug boxes are placed flat, and the placement poses of the drugs can be free without specific arrangement; since the traceability code will only be printed on the side of the packaging box, having a contact surface does not affect shooting and scanning; in addition, since the turntable can rotate multiple angles, which is equivalent to the camera shooting from multiple angles, the traceability code can also be captured even with free placement poses. Step 102, start the system through a button, and collect partial angle images of the loaded drugs. The collection angle range is 0° - 120°; Step 103, determine whether the images of each drug in each direction have been collected. If not, the system controls the turntable to rotate 180° ± 10° to continue collecting until the images of each drug in each direction are collected, ensuring that all four sides of each drug packaging box are collected.

[0048] Step 2, preprocess the initial images of the drug traceability codes collected, including image segmentation, image filtering, image enhancement, etc., to obtain clear and noise-free segmented images of each drug; Adopt the region growing segmentation method to segment the side regions of each stacked drug packaging box in the original image to obtain multiple side image regions of the drug boxes, then perform median filtering with a 3×3 pixel window to eliminate noise, and finally perform histogram equalization to correct the gray distribution of the sub-regions, improving the contrast between the traceability code and the background texture.

[0049] Step 3, detect and locate the traceability codes in the preprocessed images based on the SSD model; Use VGG16 as the backbone network of SSD to extract multi-level feature maps of the images, generate prediction results on multiple convolutional layers respectively. The shallower layers are responsible for detecting smaller objects, and the deeper layers are responsible for detecting larger objects. Calculate the overlapping regions between the barcode target search boxes through the non-maximum suppression algorithm, remove those boxes with high confidence in the intersection over union ratio, and output the optimal detection results.

[0050] Step 4, correct the detected sub-images of the traceability codes, including image affine transformation and binarization; Step 401, perform image affine transformation on the detected sub-images of the traceability codes, including the rotation amount θ between 20° - 40°, the scaling amount s x , s y between 1.1 - 1.3, and the translation amount tx , t y Between -50 and 50 pixels, the shearing amount d x , d y Between -0.2 and 0.2, correct the influence of image inclination, scaling, flipping, etc., so that the traceability code images maintain a consistent spatial distribution; Step 402: Perform binarization processing on the corrected traceability code sub-images, and set the threshold to a global adaptive threshold or a fixed threshold between 120 - 180 to improve the contrast between the traceability code pattern and the background.

[0051] Step 5: Perform traceability code decoding based on the Zebracrossing method, and at the same time use the barcode's own check code to verify the scanning result; Step 501: Detect the barcode area of the binarized traceability code image, including methods such as horizontal or vertical pixel value projection, pattern matching, and feature extraction; Step 502: Analyze the stripes of the one-dimensional barcode, calculate the width of each stripe and map it to a standardized ratio sequence; Step 503: According to the encoding rules of barcode types such as UPC, EAN, etc., map the stripe sequence to numbers or characters, and verify the integrity of the decoding result through the check digit; Step 504: Perform grid processing on the two-dimensional barcode, and convert each small module into binary data; Step 505: According to the encoding standard of two-dimensional barcodes such as QR codes, restore the binary data to text or URL, and use the embedded Reed - Solomon error correction code to repair the possibly damaged data area.

[0052] Step 6: Compare the recognized multi - object and multi - specification drug traceability code information with the drug database to achieve intelligent verification, store the result on the local disk for archiving, and at the same time upload it to the hospital information system for statistics.

[0053] Example 2: A method for identifying and verifying multi - object drug traceability codes based on visual recognition according to the present invention includes the following steps: Step 1: Collect multi - angle images of multi - object and multi - specification drugs; Step 101: The outpatient pharmacist randomly stacks the drugs to be scanned on the turntable of the workbench without a specific arrangement; Step 102: Trigger the system to start through a button, and collect partial - angle images of the loaded drugs. The acquisition field - of - view angle range is 60° - 180°; Step 103: Determine whether the images of each drug in all directions have been collected. If not, the system controls the turntable to rotate 170° ± 20° and continue collecting until the images of each drug in all directions are all collected, ensuring that the four side images of each drug packaging box are obtained.

[0054] Step 2: Preprocess the initial images of the drug traceability codes collected, including image segmentation, image filtering, image enhancement, etc., to obtain clear and noise-free segmented images of each drug; Use the region growing segmentation algorithm to separate the side regions of each stacked drug packaging box in the original image, obtain multiple side image regions of the medicine boxes, then perform median filtering with a 5×5 pixel window to remove noise, and finally use adaptive histogram equalization to enhance the contrast of the sub-region images.

[0055] Step 3: Detect and locate the traceability codes in the preprocessed images based on the SSD model; Use the VGG16 convolutional neural network as the backbone network of SSD to extract multi-level feature maps of the image, generate prediction results on convolutional layers at multiple levels, with the shallow layer for detecting small targets and the deep layer for detecting large targets, and use the non-maximum suppression algorithm to merge overlapping candidate boxes and output the final detection results.

[0056] Step 4: Correct the detected sub-images of the traceability codes, including image affine transformation and binarization; Step 401: Perform image affine transformation on the detected sub-images of the traceability codes, including the rotation amount θ between 15° and 45°, the scaling amount s x , s y between 1.0 and 1.4, the translation amount t x , t y between -100 and 100 pixels, the shearing amount d x , d y between -0.3 and 0.3, to correct the effects of image tilt, scaling, flipping, etc., and keep the traceability code images in a unified direction and size; Step 402: Perform binarization processing on the corrected sub-images of the traceability codes, and set the threshold to an adaptive threshold or a fixed threshold between 100 and 200 to improve the contrast between the traceability code pattern and the background.

[0057] Step 5: Decode the traceability codes based on the Zebracrossing method, and at the same time use the barcode's own check code to verify the scanning result; Step 501: Detect the barcode area in the binarized traceability code image, including methods such as horizontal pixel value projection, pattern matching locator, and feature extraction; Step 502: Analyze the stripes of the one-dimensional barcode, calculate the width of each stripe, and standardize it into a proportion sequence; Step 503: According to the encoding rules of barcode types such as Code128, ITF, etc., map the stripe sequence to numbers or characters, and verify the integrity of the decoding result through the check code; Step 504: Perform grid processing on the two-dimensional barcode, and binarize each small module into binary data; Step 505: According to the encoding standard of two-dimensional barcodes such as PDF417, decode the binary data into text or URL, and use the embedded Reed-Solomon error correction code to repair the possibly damaged data area.

[0058] Step 6: Compare the identified multi-object and multi-specification drug traceability code information with the drug database to achieve intelligent verification, store the result on the local disk for archiving, and upload it to the hospital information system to complete statistical analysis.

[0059] Using the multi-object drug traceability code detection system designed by the present invention, multi-angle imaging of hospital drugs was carried out comprehensively and accurately. The initially collected images are as Figure 7 shown in a. These images contain multiple angles and details of the drug packaging box, providing a rich information basis for subsequent processing.

[0060] In the image preprocessing stage, the system performed operations such as segmentation, filtering, and enhancement on the initially collected images, and obtained the preprocessed images as Figure 7 shown in b. The segmentation operation effectively separated the drug packaging box from the background. The filtering process removed the noise and interference in the image, and the enhancement operation further improved the clarity and contrast of the image, making the traceability code area more prominent and easy to identify.

[0061] Next, the system performed traceability code detection and positioning on the preprocessed images. Through advanced algorithms and models, the system accurately identified the traceability code area on the drug packaging box, as Figure 7 shown in c. This step provided a key target area for subsequent image correction and traceability code decoding.

[0062] Finally, the system performed image correction and traceability code decoding on the located traceability code area. The image correction operation eliminated the image distortion and distortion caused by factors such as shooting angle and lighting conditions, making the traceability code presented in the clearest and most accurate form. Subsequently, the system used the decoding algorithm to decode the corrected traceability code and successfully obtained the final traceability code information, as Figure 7As shown in d. This information is of great significance for the traceability, management, and quality control of drugs.

[0063] The following are the device embodiments of the present invention, which can be used to implement the method embodiments of the present invention. For the details not disclosed in the device embodiments, please refer to the method embodiments of the present invention.

[0064] In one embodiment of the present invention, a multi-object drug traceability code recognition and verification system based on visual recognition is further provided, including: A drug packaging visual inspection system for obtaining multi-angle drug packaging images of multi-object drugs based on visual recognition; A drug traceability code recognition system for extracting the traceability code area on the multi-angle drug packaging images; A drug traceability code verification system for decoding the traceability code in the traceability code area to obtain traceability code information and comparing the traceability code information with the drug database.

[0065] The drug packaging visual inspection system includes an optical imaging module and a motion control module. The motion control module and the optical imaging module are both arranged on the operation table. The multi-object drugs are placed on the motion control module, and multi-angle drug packaging images of the multi-object drugs are obtained through the optical imaging module. Specifically: The optical imaging module includes an industrial camera, a large-depth-of-field fixed-focus lens, a ring shadowless light source, and a light source controller; the motion control module includes a motion controller and a precision turntable; the industrial camera uses a large-depth-of-field fixed-focus lens and is arranged on both sides of the precision turntable through a two-dimensional adjustment bracket. The light source controller is connected to the ring shadowless light source to control the illumination intensity of the industrial camera's shooting environment; the precision turntable is used to place the multi-object drugs, and the motion controller is connected to the precision turntable to control the rotation of the precision turntable to obtain multi-angle drug packaging images of the multi-object drugs; the industrial camera is connected to the computer to obtain multi-angle drug packaging images of the multi-object drugs; the motion controller is connected to the computer to control the adjustment direction of the precision turntable; the light source controller is connected to the computer to control the illumination intensity of the industrial camera's shooting environment; the industrial camera is connected to the user-side trigger device to control the shooting of the industrial camera.

[0066] Through steps such as multi-angle imaging, image preprocessing, traceability code detection and positioning, and image correction and traceability code decoding, the system of the present invention realizes the accurate identification and efficient decoding of hospital drug traceability codes, providing strong technical support for drug traceability and management. Although machine vision technology has been widely used in fields such as logistics and retail to quickly identify barcode information through industrial cameras and light sources. However, in the medical field, the application of machine vision for drug management scenarios is still in its infancy. This system combines barcode automatic positioning and multi-object recognition technology to meet the high-precision requirements of hospital drug management; this system adopts multi-angle image acquisition technology, by configuring multiple industrial cameras and light sources to collect the packaging information of medicine boxes from different angles, overcoming the difficulty of barcode recognition caused by reflection or occlusion. Combined with the SSD (Single Shot MultiBox Detector) algorithm, multiple barcodes can be accurately positioned in complex scenarios. Further, the barcodes on drug packages may affect the recognition accuracy due to reflection, dirt, or insufficient light. This system integrates image filtering, denoising, and gray-level equalization technologies to effectively optimize the image quality and provide high-quality data for subsequent barcode detection and decoding; further, the system of the present invention is seamlessly docked with the hospital information management platform, and can upload recognition data in real time, match with drug inventory, prescription management, and patient information, and build a complete closed-loop management of drug traceability.

[0067] In another embodiment of the present invention, a terminal device is further provided. The terminal device includes a processor and a memory. The memory is used to store a computer program. The computer program includes program instructions. The processor is used to execute the program instructions stored in the computer storage medium. The processor may be a central processing unit (CPU), or may also be other general-purpose processors, digital signal processors (DSPs), application specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. It is the computing core and control core of the terminal, and is suitable for implementing one or more instructions, specifically suitable for loading and executing one or more instructions to implement the corresponding method process or corresponding function; the processor described in the embodiment of the present invention can operate a multi-object drug traceability code recognition and verification method based on visual recognition.

[0068] In another embodiment of the present invention, the present invention further provides a storage medium, specifically a computer-readable storage medium (Memory). The computer-readable storage medium is a memory device in a terminal device and is used to store programs and data. It can be understood that the computer-readable storage medium here can include both the built-in storage medium in the terminal device and, of course, the extended storage medium supported by the terminal device. The computer-readable storage medium provides a storage space, and the operating system of the terminal is stored in this storage space. Moreover, one or more instructions suitable for being loaded and executed by the processor are stored in this storage space, and these instructions can be one or more computer programs (including program codes). It should be noted that the computer-readable storage medium here can be a high-speed RAM memory or a non-volatile memory, such as at least one disk memory. One or more instructions stored in the computer-readable storage medium can be loaded and executed by the processor to implement the corresponding steps of the method for identifying and verifying a multi-object drug traceability code based on visual recognition in the above embodiments.

[0069] Those skilled in the art should understand that the embodiments of the present application can be provided as a method, a system, or a computer program product. Therefore, the present application can adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present application can adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk memories, CD-ROMs, optical memories, etc.) containing computer-usable program codes.

[0070] The present application is described with reference to the flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or block in the flowchart and / or block diagram can be implemented by computer program instructions, and the combination of the processes and / or blocks in the flowchart and / or block diagram can also be implemented by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing devices to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing devices generate a device for implementing the functions specified in Figure 1 one process or multiple processes and / or blocks Figure 1 one block or multiple blocks.

[0071] These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer-readable memory generate a manufactured article including an instruction device, and the instruction device implements the functions in Figure 1 one process or multiple processes and / or blocksFigure 1 The functions specified in one or more boxes.

[0072] These computer program instructions can also be loaded onto a computer or other programmable data processing device, so that a series of operation steps are executed on the computer or other programmable device to generate a computer-implemented process. Thus, the instructions executed on the computer or other programmable device provide for implementing the steps of the functions specified in one Figure 1 process or multiple processes and / or boxes Figure 1 or more boxes.

[0073] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit them. Although the present invention has been described in detail with reference to the above embodiments, those of ordinary skill in the art should understand that: modifications or equivalent replacements can still be made to the specific implementation manners of the present invention. Any modification or equivalent replacement that does not depart from the spirit and scope of the present invention shall be covered by the protection scope of the claims of the present invention.

Claims

1. A method for identifying and verifying multi-object drug traceability codes based on visual recognition, characterized in that, The specific steps are as follows: Obtain multi-object drug multi-angle drug packaging images based on visual recognition; Extract the traceability code area on the multi-angle drug packaging images; Decode the traceability code in the traceability code area to obtain traceability code information, and check the traceability code information against the drug database.

2. The multi-object drug traceability code recognition and verification method based on visual recognition according to claim 1, wherein, In the step of extracting the traceability code area on the multi-angle drug packaging images, the specific steps are as follows: 1) Place the packaging boxes of multi-object drugs flat, and obtain multi-angle drug packaging images of multi-object drugs based on visual recognition; 2) Use the region growing segmentation method to segment the side regions of each stacked drug packaging box in the obtained multi-angle drug packaging images to obtain multiple medicine box side image regions, then use median filtering to filter out small-size noises, perform the connection operation to extract each connected domain of the segmented image, and use the size threshold parameter to screen out the actual medicine box region image; 3) Transmit the actual medicine box region image to the SSD model to extract the traceability code region image; 4) Perform image affine transformation and binarization processing on the extracted traceability code region image to obtain a corrected traceability code region image.

3. A method for identifying and verifying multi-object drug traceability codes based on visual recognition according to claim 1, characterized in that, In the step of decoding the traceability code in the traceability code area to obtain traceability code information and checking the traceability code information against the drug database: Use the Zebra crossing decoding algorithm to decode the traceability code in the corrected traceability code region image to obtain traceability code information, and the traceability code is a one-dimensional bar code or a two-dimensional code.

4. A method for identifying and verifying multi-object drug traceability codes based on visual recognition according to claim 1, characterized in that, The step of obtaining multi-angle drug packaging images of multi-object drugs based on visual recognition is implemented by using an optical imaging module and a motion control module. Among them, the motion control module and the optical imaging module are both set on the operation table, and multi-object drugs are placed on the motion control module. The multi-angle drug packaging images of multi-object drugs are obtained through the optical imaging module. Specifically: The optical imaging module includes an industrial camera, a large-depth-of-field fixed-focus lens, a ring shadowless light source, and a light source controller; the motion control module includes a motion controller and a precision turntable; the industrial camera uses a large-depth-of-field fixed-focus lens and is set on both sides of the precision turntable through a two-dimensional adjustment bracket. The light source controller is connected to the ring shadowless light source to control the illumination light intensity of the industrial camera shooting environment; the precision turntable is used to place multi-object drugs, and the motion controller is connected to the precision turntable to control the rotation of the precision turntable to obtain multi-angle drug packaging images of multi-object drugs.

5. A method for identifying and verifying multi-object drug traceability codes based on visual recognition according to claim 4, characterized in that, The industrial camera is connected to a computer to obtain multi-angle drug packaging images of multi-object drugs; the motion controller is connected to the computer to control the direction adjustment of the precision turntable; the light source controller is connected to the computer to control the illumination light intensity of the industrial camera shooting environment; the industrial camera is connected to the user-side trigger device to control the shooting of the industrial camera.

6. A multi-object drug traceability code recognition and verification system based on visual recognition, characterized in that, Including: A drug packaging visual detection system for obtaining multi-angle drug packaging images of multi-object drugs based on visual recognition; A drug traceability code recognition system for extracting the traceability code area on the multi-angle drug packaging images; A drug traceability code verification system for decoding the traceability code in the traceability code area to obtain traceability code information and checking the traceability code information against the drug database.

7. The multi-object drug traceability code recognition and verification system based on visual recognition according to claim 6, characterized in that, The drug packaging visual inspection system includes an optical imaging module and a motion control module. The motion control module and the optical imaging module are both arranged on the operating table. The multi-object drugs are placed on the motion control module, and multi-angle drug packaging images of the multi-object drugs are obtained through the optical imaging module. Specifically: The optical imaging module includes an industrial camera, a large-depth-of-field fixed-focus lens, a ring shadowless light source, and a light source controller; the motion control module includes a motion controller and a precision turntable. The industrial camera uses a large-depth-of-field fixed-focus lens and is arranged on both sides of the precision turntable through a two-dimensional adjustment bracket. The light source controller is connected to the ring shadowless light source to control the illumination light intensity of the industrial camera's photographing environment; the precision turntable is used to place the multi-object drugs, and the motion controller is connected to the precision turntable to control the rotation of the precision turntable to obtain multi-angle drug packaging images of the multi-object drugs; the industrial camera is connected to the computer to obtain multi-angle drug packaging images of the multi-object drugs; the motion controller is connected to the computer to control the adjustment direction of the precision turntable; the light source controller is connected to the computer to control the illumination light intensity of the industrial camera's photographing environment; the industrial camera is connected to the user-side trigger device to control the industrial camera to take pictures.

8. A terminal device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of a method for identifying and verifying multi-object drug traceability codes based on visual recognition as described in any one of claims 1 to 5.

9. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the steps of a method for identifying and verifying multi-object drug traceability codes based on visual recognition as described in any one of claims 1 to 5.

10. A computer program product comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the steps of a method for identifying and verifying multi-object drug traceability codes based on visual recognition as described in any one of claims 1 to 5.

Images