Intelligent medicine identification method and system based on image visual identification

Abstract

The invention discloses an intelligent medicine identification method and system based on image visual identification, and relates to the technical field of image visual identification, and the method comprises the steps: shooting a liquid bag and a medicine bottle, carrying out the model training, and obtaining information through scanning a two-dimensional code of a doctor's advice; identifying and comparing the liquid bag and the medicine bottle through a multi-model fusion algorithm; and uploading the identified and compared data to a server, and carrying out docking and tracing management on the data. According to the method, the adaptability of a medicine identification system is improved, manual input errors are reduced, the overall efficiency and safety of medicine management are improved, through efficient algorithm processing, the identification time is shortened, the medicine distribution and management work flow is optimized, complete data tracing historical records and statistical analysis are provided, and the medicine management efficiency is improved. Therefore, quality control and management decision making of the medical institution are facilitated, the relevance between the medicine information and the medical advice information is improved, the risk of medicine use errors is effectively reduced, and the medical service quality is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of image visual recognition, and in particular to an intelligent drug recognition method and system based on image visual recognition. Background Art

[0002] In today's wave of development of medical informatization and intelligence, image visual recognition technology has become one of the key technologies to improve the efficiency and quality of drug management. Through advanced image processing and deep learning algorithms, image visual recognition technology can quickly identify text, patterns and shapes on drug packaging, thus playing an important role in pharmacy automation, drug supply chain management and other fields. With the continuous advancement of technology, the scope and depth of application of image visual recognition in drug identification are constantly expanding, and medical institutions are increasingly in need of improving the accuracy and efficiency of drug identification.

[0003] However, the existing drug recognition technology still has many limitations when facing actual application scenarios. The diversity of drug packaging has brought great challenges to image recognition. The materials, colors, shapes and label designs of drug packaging such as liquid bags and medicine bottles are different. Traditional image recognition systems are often difficult to adapt to this complexity. When identifying drugs, the existing technology rarely considers the combination with the doctor's order information. The doctor's order contains important medication information. Without this information support, the accuracy of drug recognition is difficult to guarantee. The existing technology generally adopts a single model for recognition and lacks an effective multi-model fusion strategy, which affects the recognition speed and limits the improvement of recognition accuracy. In terms of data uploading and traceability management, the existing technology has also failed to form a complete solution, resulting in loopholes in the traceability and management of drug information. In summary, the existing drug recognition technology has deficiencies in recognition rate, information fusion, algorithm efficiency and traceability management. The existence of these problems limits the further application of drug recognition technology in the medical field. The present invention aims to overcome these technical difficulties and achieve accurate, efficient and comprehensive traceability of drug recognition through innovative image visual recognition methods to meet the standards and requirements of drug management in the modern medical environment. Summary of the invention

[0004] In view of the above-mentioned problems, the present invention is proposed.

[0005] Therefore, the technical problem solved by the present invention is: the existing image visual recognition technology has poor recognition accuracy, low system response speed, insufficient information integration capability, and the problem of how to achieve accurate matching of drug identification and efficient traceability management of data.

[0006] In order to solve the above technical problems, the present invention provides the following technical solutions: an intelligent drug identification method based on image visual recognition, comprising photographing liquid bags and medicine bottles and performing model training, and obtaining information by scanning the QR code of the doctor's order; identifying and comparing the liquid bags and medicine bottles through a multi-model fusion algorithm; uploading the data after identification and comparison to a server, and docking and traceability management of the data.

[0007] As a preferred solution of the intelligent drug identification method based on image visual recognition described in the present invention, the model training includes sample training, using dual cameras to take 20 to 30 clear pictures from multiple angles for each medicine bottle, and all the pictures of the medicine bottles are uniformly trained on the algorithm model after the shooting is completed.

[0008] As a preferred solution of the intelligent drug identification method based on image visual recognition described in the present invention, the multi-model fusion algorithm includes high-performance target detection, surface OCR technology, and image stitching.

[0009] As a preferred solution of the intelligent drug identification method based on image visual recognition described in the present invention, the high-performance target detection includes transferring the drug data through a high-performance target detection model, parsing the information that can be understood by the computer from the image, and identifying the medicine bottles and bags, and identifying the location of the object in the image.

[0010] As a preferred solution of the intelligent drug identification method based on image visual recognition described in the present invention, the surface OCR technology includes an optical character recognition model trained by surface image data, image preprocessing is completed by graying, binarization and denoising, surface detection and surface flattening are used to achieve surface correction and flattening, and a deep learning algorithm is used to detect text areas in the flattened image and recognize the segmented characters, and convert them into computer-readable character data.

[0011] As a preferred solution of the intelligent drug recognition method based on image visual recognition described in the present invention, the image stitching includes extracting and matching feature points of local feature descriptors through an image stitching model, eliminating erroneous matches and estimating the geometric transformation relationship between images using the RANSAC algorithm, aligning different images to the same coordinate system according to the estimated geometric transformation relationship, performing image fusion through a weighted average fusion method to generate a seamlessly stitched panoramic image, identifying text on the medicine bottle image according to a multi-model fusion algorithm, and reporting the detection results through comparison.

[0012] As a preferred solution of the intelligent drug identification method based on image visual recognition described in the present invention, the data docking and traceability management includes calling the WebAPI interface of the Kanglianda system through the medical order number identified on the liquid bag, obtaining the liquid bag information and the medicine bottle information associated with the liquid bag, and providing complete traceability history records and statistical analysis for all identification and detection reported data.

[0013] Another object of the present invention is to provide an intelligent drug recognition system based on image visual recognition, which can identify and compare liquid bags and medicine bottles through a multi-model fusion algorithm, thereby solving the problems of low recognition accuracy and insufficient model generalization ability of current image visual recognition technology.

[0014] As a preferred solution of the intelligent drug identification system based on image visual recognition described in the present invention, it includes: an acquisition training module, an intelligent recognition and comparison module, and a data synchronization management module; the acquisition training module is used to photograph the liquid bag and the medicine bottle and perform model training, and obtain information by scanning the QR code of the doctor's order; the intelligent recognition and comparison module is used to identify and compare the liquid bag and the medicine bottle through a multi-model fusion algorithm; the data synchronization management module is used to upload the data after the identification and comparison is completed to the server, and to connect and trace the data.

[0015] A computer device includes a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement a step of an intelligent drug recognition method based on image visual recognition.

[0016] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of an intelligent drug identification method based on image visual recognition.

[0017] Beneficial effects of the present invention: The intelligent drug recognition method based on image visual recognition provided by the present invention takes pictures of liquid bags and medicine bottles and performs model training, obtains information by scanning the two-dimensional code of the doctor's order form, realizes the preliminary collection and preprocessing of drug information, provides accurate training data and real-time doctor's order information for the subsequent recognition process, improves the adaptability of the drug recognition system, reduces human input errors, ensures the accuracy and real-time nature of the recognition process, thereby improving the overall efficiency and safety of drug management, and recognizes and compares liquid bags and medicine bottles through a multi-model fusion algorithm, improves the accuracy and robustness of recognition, reduces the situation of misidentification and missed recognition, and improves the reliability of the system, and shortens the recognition time through efficient algorithm processing, thereby optimizing the distribution and management workflow of drugs, and uploads the data after recognition and comparison to the server, and performs docking and traceability management on the data, realizing the whole process monitoring of drugs from production to use, which is convenient for medical institutions to carry out quality control and management decisions, and through data docking, enhances the correlation between drug information and doctor's order information, effectively reduces the risk of medication errors, and improves the quality of medical services. The present invention achieves better results in recognition accuracy, processing speed and system generalization ability. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.

[0019] Figure 1 An overall flow chart of an intelligent drug identification method based on image visual recognition provided by the first embodiment of the present invention.

[0020] Figure 2 An overall flow chart of an intelligent drug identification system based on image visual recognition provided in the third embodiment of the present invention. DETAILED DESCRIPTION

[0021] In order to make the above-mentioned purposes, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are described in detail below in conjunction with the drawings of the specification. Obviously, the described embodiments are part of the embodiments of the present invention, but not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary persons in the art without creative work should fall within the scope of protection of the present invention.

[0022] Example 1, reference Figure 1 , as an embodiment of the present invention, provides an intelligent drug recognition method based on image visual recognition, comprising:

[0023] S1: Take photos of the liquid bag and medicine bottle and train the model, and obtain information by scanning the QR code on the doctor's order.

[0024] Furthermore, model training includes sample training, using dual cameras to take 20 to 30 clear pictures from multiple angles for each medicine bottle. After all the pictures of the medicine bottles are taken, the algorithm model training is carried out uniformly.

[0025] It should be noted that the specific process of completing the unified algorithm model training is as follows: First, the system collects a large number of medicine bottle image samples, and each medicine bottle takes 20 to 30 clear images from different angles. These samples will be uploaded to the server and preprocessed, including image blur detection and denoising. Next, the system will annotate the data of all samples, identify the medicine bottles, labels and character areas in the image, and use the annotated data to train the multi-model fusion algorithm. The algorithm includes the target detection model, the surface OCR model and the image stitching model. The training process usually uses transfer learning technology: secondary training is performed based on the pre-trained model, and the knowledge of the existing model is used to accelerate the training process; during the training process, the system will continuously adjust the model parameters through optimization algorithms (such as Adam optimizer) to minimize the loss function to improve the accuracy of the model. After the training is completed, the model will evaluate its performance through cross-validation to ensure that the model has good generalization ability and is ultimately used for the recognition of actual medicine bottles and liquid bags.

[0026] It should also be noted that by photographing the liquid bags and medicine bottles and conducting model training, and obtaining information by scanning the QR code on the doctor's order form, the preliminary collection and preprocessing of drug information is achieved, providing accurate training data and real-time doctor's order information for the subsequent recognition process, improving the adaptability of the drug recognition system, reducing human input errors, and ensuring the accuracy and real-time nature of the recognition process, thereby improving the overall efficiency and safety of drug management.

[0027] S2: Identify and compare liquid bags and medicine bottles through multi-model fusion algorithm.

[0028] Furthermore, the multi-model fusion algorithm includes high-performance target detection, surface OCR technology, and image stitching.

[0029] It should be noted that high-performance target detection includes transfer learning of pharmaceutical data through high-performance target detection models, parsing information from images that can be understood by computers, and used to identify medicine bottles and bags, and identify the location of objects in the image.

[0030] It should be noted that surface OCR technology includes an optical character recognition model trained with surface image data, image preprocessing using grayscale, binarization, and denoising, surface correction and flattening using surface detection and flattening, and deep learning algorithms to detect text areas in the flattened image and recognize the segmented characters, converting them into computer-readable character data.

[0031] It should also be noted that image stitching includes extracting and matching feature points of local feature descriptors through an image stitching model, eliminating false matches and estimating the geometric transformation relationship between images using the RANSAC algorithm, aligning different images to the same coordinate system based on the estimated geometric transformation relationship, and performing image fusion through a weighted average fusion method to generate a seamless panoramic image. The text on the medicine bottle image is identified based on a multi-model fusion algorithm, and the detection results are reported through comparison.

[0032] It should also be noted that surface detection and surface flattening are used to achieve surface correction and flattening, and a deep learning algorithm is used to detect text areas in the flattened image and recognize the segmented characters to convert them into computer-readable character data. The specific process is as follows: First, the system obtains an image of a medicine bottle or other curved object through a camera. Since the label of the medicine bottle is usually affixed to a curved surface, this may cause the text in the image to bend and distorted. Therefore, the system first performs surface detection, identifies the curved surface area in the image, and determines the specific way in which the text is bent. Next, the system uses a surface flattening algorithm, such as a flattening technology based on geometric projection or convolutional neural network (CNN), to flatten and correct the text on the curved surface into a flat style. The corrected image is then processed by a deep learning algorithm, such as an OCR (optical character recognition) model based on a long short-term memory (LSTM) network or a convolutional network (CNN). The algorithm will first detect the text area in the flattened image and separate the text from the background. Finally, the system recognizes each segmented character one by one and converts the recognition result into computer-readable character data for subsequent processing.

[0033] It should also be noted that the specific process of identifying and comparing liquid bags and medicine bottles is as follows: First, the dispensing staff scans the QR code on the doctor's order form through the system. The system obtains the doctor's order information corresponding to the QR code, including the type and quantity of drugs in the liquid bag, by calling the API interface of the hospital management system or the drug management system. Then, the liquid bag is placed in a designated area for photography. The system detects the shape, label and content of the liquid bag through an image recognition algorithm. The recognition result is compared with the doctor's order information obtained through the QR code to ensure that the drugs and information in the liquid bag are completely consistent. If the comparison result is accurate, the system will generate a confirmation signal and report the test result; if an inconsistency is found, the system will issue an alarm to prompt possible errors. The entire process ensures the accurate matching of the liquid bag drugs with the doctor's order form, thereby avoiding dispensing errors.

[0034] It should also be noted that the multi-model fusion algorithm is used to identify and compare liquid bags and medicine bottles, which improves the accuracy and robustness of recognition, reduces misidentification and missed identification, improves the reliability of the system, and shortens the recognition time through efficient algorithm processing, thereby optimizing the drug distribution and management workflow.

[0035] S3: Upload the identified and compared data to the server, and perform data docking and traceability management.

[0036] Furthermore, data docking and traceability management includes calling the WebAPI interface of the Kanglianda system through the medical order number identified on the liquid bag, obtaining the liquid bag information and the medicine bottle information associated with the liquid bag, and providing complete traceability history records and statistical analysis for all identification and detection reported data.

[0037] It should be noted that the specific steps for providing complete traceability history records and statistical analysis are as follows: The management process of traceability history records starts with each identification and comparison operation. When the system completes the identification of the medicine bottle or liquid bag, it will automatically generate a unique tracking number. This number is stored in the system's database together with the identification data, comparison results, timestamp and other information. The specific details of each identification and comparison, including the identified image, identified drug information, and operator ID, will be saved as historical records. The system also provides statistical analysis tools that can analyze and visualize all traceability data. For example, administrators can generate reports on the dispensing success rate, identification error rate, and work efficiency of each operator over a period of time. These data are displayed through visual charts or dashboards so that managers can monitor and evaluate the operation of the system to ensure the safety and efficiency of the dispensing process.

[0038] It should also be noted that the data after identification and comparison is uploaded to the server, and the data is connected and traced, which realizes the whole process monitoring of drugs from production to use, facilitates medical institutions to carry out quality control and management decisions, and enhances the correlation between drug information and medical order information through data connection, effectively reducing the risk of medication errors and improving the quality of medical services.

[0039] Example 2 is an embodiment of the present invention, which provides an intelligent drug identification method based on image visual recognition. In order to verify the beneficial effects of the present invention, scientific demonstration is carried out through economic benefit calculation and simulation experiments.

[0040] First, the experiment selected three medicine bottles of different shapes and sizes (marked as medicine bottles A, B, and C) and three different types of liquid bags (marked as liquid bags D, E, and F) as experimental objects. The experimental process was strictly carried out according to the following steps: First, a dual-camera system was used to photograph each medicine bottle from six different angles, and 5 clear pictures were taken at each angle, for a total of 30 pictures. Similarly, the liquid bags were also photographed in the same way. All pictures were uploaded to the server after blur detection and denoising preprocessing; the uploaded pictures were annotated to clearly identify the outlines, labels, and character areas of the medicine bottles and liquid bags. Subsequently, the annotated data was used to train a multi-model fusion algorithm, including It includes target detection model, surface OCR model and image stitching model. Transfer learning technology is used in the training process, and secondary training is performed based on the pre-trained model to accelerate the training process and improve the model accuracy. In the actual application, the dispensing staff scans the QR code of the doctor's order form, and the system calls the API interface to obtain the doctor's order information, and places the liquid bag in the designated area for shooting. The system uses the image recognition algorithm to detect the shape, label and content of the liquid bag and medicine bottle, and compares the recognition result with the doctor's order information. After the comparison is completed, the recognition and comparison data are uploaded to the server, and the system WebAPI interface is called through the doctor's order number on the liquid bag to realize data docking and traceability management.

[0041] Example 3, reference Figure 2 , which is an embodiment of the present invention, provides an intelligent drug identification system based on image visual recognition, including an acquisition training module, an intelligent identification and comparison module, and a data synchronization management module.

[0042] The collection and training module is used to photograph liquid bags and medicine bottles and perform model training, and obtain information by scanning the QR code on the medical order form; the intelligent recognition and comparison module is used to identify and compare liquid bags and medicine bottles through a multi-model fusion algorithm; the data synchronization management module is used to upload the data after identification and comparison to the server, and to connect and trace the data.

[0043] If the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the methods of each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0044] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or in conjunction with such instruction execution systems, devices or apparatuses. For the purposes of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in conjunction with such instruction execution systems, devices or apparatuses.

[0045] More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or more wires (electronic device), a portable computer disk case (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be a paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, deciphering, or processing in another suitable manner as necessary, and then stored in a computer memory.

[0046] It should be understood that the various parts of the present invention can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc. It should be noted that the above embodiments are only used to illustrate the technical solution of the present invention and are not limited. Although the present invention is described in detail with reference to the preferred embodiments, it should be understood by those skilled in the art that the technical solution of the present invention can be modified or replaced by equivalents without departing from the spirit and scope of the technical solution of the present invention, which should be included in the scope of the claims of the present invention.

[0047] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention can be modified or replaced by equivalents without departing from the spirit and scope of the technical solutions of the present invention, which should be included in the scope of the claims of the present invention.

Claims

1. An intelligent drug recognition method based on image visual recognition, characterized in that: include: Take photos of fluid bags and medicine bottles and train the model, and obtain information by scanning the QR code on the doctor's order form; Through multi-model fusion algorithm, liquid bags and medicine bottles are identified and compared; The identified and matched data is uploaded to the server, and the data is connected and traced.

2. The intelligent drug identification method based on image visual recognition according to claim 1, characterized in that: The model training includes sample training, using dual cameras to take 20 to 30 clear pictures from multiple angles for each medicine bottle, and all the pictures of the medicine bottles are uniformly trained on the algorithm model after the pictures are taken.

3. The intelligent drug identification method based on image visual recognition as claimed in claim 2, characterized in that: The multi-model fusion algorithm includes high-performance target detection, surface OCR technology, and image stitching.

4. The intelligent drug identification method based on image visual recognition as claimed in claim 3, characterized in that: The high-performance target detection includes performing transfer learning on drug data through a high-performance target detection model, parsing information that can be understood by a computer from the image, and identifying medicine bottles and bags, and identifying the location of objects in the image.

5. The intelligent drug identification method based on image visual recognition as claimed in claim 3, characterized in that: The surface OCR technology includes an optical character recognition model trained with surface image data, image preprocessing using grayscale, binarization, and denoising, surface correction and flattening using surface detection and surface flattening, and detection of text areas in the flattened image using a deep learning algorithm and recognition of segmented characters, which are converted into computer-readable character data.

6. The intelligent drug identification method based on image visual recognition as claimed in claim 3, characterized in that: The image stitching includes extracting and matching feature points of local feature descriptors through an image stitching model, eliminating erroneous matches and estimating the geometric transformation relationship between images using the RANSAC algorithm, aligning different images to the same coordinate system based on the estimated geometric transformation relationship, performing image fusion through a weighted average fusion method to generate a seamlessly stitched panoramic image, identifying text on a medicine bottle image based on a multi-model fusion algorithm, and reporting the detection result through comparison.

7. The intelligent drug identification method based on image visual recognition as claimed in claim 6, characterized in that: The data docking and traceability management includes calling the WebAPI interface of the Kanglianda system through the medical order number identified on the liquid bag, obtaining the liquid bag information and the medicine bottle information associated with the liquid bag, and providing complete traceability history records and statistical analysis for all identification and detection reported data.

8. A system using the intelligent drug identification method based on image visual recognition as claimed in any one of claims 1 to 7, characterized in that: Including collection training module, intelligent recognition and comparison module, and data synchronization management module; The acquisition training module is used to photograph the liquid bag and medicine bottle and perform model training, and obtain information by scanning the QR code of the doctor's order; The intelligent recognition and comparison module is used to identify and compare the liquid bag and the medicine bottle through a multi-model fusion algorithm; The data synchronization management module is used to upload the data after identification and comparison to the server, and to perform docking and traceability management on the data.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the intelligent drug identification method based on image visual recognition according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the intelligent drug identification method based on image visual recognition according to any one of claims 1 to 7 are implemented.

Images