A real-time medicine box recognition method and system based on deep learning

Abstract

The present invention discloses a real-time medicine box recognition method and system based on deep learning, belonging to the field of medicine management technology. The method comprises the following steps: setting up a camera above the medicine box to collect image information of the medicine box's appearance; using a pre-trained medicine box recognition model based on Yolov5 to obtain medicine box edge and appearance-based medicine box recognition results; performing text detection using a pre-trained DBNet-based text detection model to obtain the area of the medicine box text; performing text recognition using a pre-trained DenseNet and CTC-based text recognition model to identify the text information of the medicine box; and performing a low-threshold fuzzy search and matching on each word or phrase in the text recognition result in turn in a medicine database. The present invention can effectively solve the problem of manual verification required when dispensing medicines in pharmacies. Moreover, the present method greatly ensures the accuracy of recognition and reduces the workload of pharmacy workers.

Description

Technical Field

[0001] The present invention relates to the technical field of drug management, and in particular to a method and system for real-time identification of medicine boxes based on deep learning. Background Art

[0002] In recent years, with rapid economic growth, public demand for healthcare and services has rapidly increased, leading to the rapid development of various automated and intelligent medical devices. Along with significant improvements in medical equipment and physician quality, the variety and quantity of medications has also rapidly increased. Traditional pharmacy service models rely primarily on manual labor to store and retrieve medications. In this model, the preparation and distribution of medications are entirely performed by professional pharmacists. Patients submit prescriptions at the counter, while pharmacists are busy dispensing medications, passively dispensing medications. This model has two major drawbacks. First, for pharmacists, the large variety of medications and the rigorous verification required to dispense medications create a heavy workload without improving their technical skills, diminishing their work skills and wasting valuable human and medical resources. Furthermore, the long, repetitive tasks create safety risks, leading to the mis-dispensing of medications. Furthermore, this service model hinders communication between doctors and patients. Because pharmacists are busy dispensing medications, they are unable to provide patients with more comprehensive medication consultation services. Furthermore, due to the inefficiency of manual medication dispensing, patients often have to wait in line to pick up their medications, impacting their healthcare experience. Therefore, realizing pharmacy automation and informatization is the reform trend of outpatient pharmacies and also the guarantee for further improving medical quality.

[0003] With the development of computer technology, deep learning techniques have also been widely applied. Image recognition technology can accurately detect and identify medicine boxes, enabling automated medication management. However, despite the advancements in current recognition technology, the performance of current medicine box image recognition algorithms is poor due to the limited differences in shape and color and the wide variety of medicine boxes. The accuracy is far from sufficient for practical pharmacy use. Therefore, improving the performance of medicine box recognition algorithms is crucial to the entire smart pharmacy verification process, as the quality of recognition directly impacts patient medication safety. Summary of the Invention

[0004] The technical task of the present invention is to address the above shortcomings and provide a real-time medicine box recognition method and system based on deep learning, which can accurately identify the type and quantity of medicines, improve the speed of medicine distribution in pharmacies, and reduce labor costs.

[0005] The technical solution adopted by the present invention to solve its technical problem is:

[0006] A real-time medicine box recognition method based on deep learning includes the following steps:

[0007] A camera is installed above the medicine box to collect image information of the appearance of the medicine box;

[0008] Use the pre-trained yolov5-based medicine box recognition model to detect and identify medicine boxes, and obtain medicine box edge and appearance-based medicine box recognition results;

[0009] The original image is cropped according to the obtained medicine box edges to obtain multiple images, each containing only a single medicine box. For each cropped medicine box image, text detection is performed using a pre-trained DBNet-based text detection model to obtain the area containing the medicine box text.

[0010] For the identified text area, use the pre-trained DenseNet and CTC-based text recognition model to perform text recognition and identify the text information on the medicine box;

[0011] Each word or phrase in the text recognition result is sequentially subjected to a low-threshold fuzzy search and matching in the drug database. The fuzzy search and matching content is the drug name, brand, manufacturer, and dosage attributes, and one or more drug numbers are obtained;

[0012] Search the unique drug number identified by appearance among the drug numbers found by text recognition; if the search is successful, output the recognition result;

[0013] If the search fails, each word or phrase in the text recognition result will be subjected to intelligent fuzzy search and matching in the drug database in turn, and finally the medicine box recognition result will be obtained.

[0014] Furthermore, before the step of setting up a camera above the medicine box to collect image information of the appearance of the medicine box, the following steps are included:

[0015] Before using the neural network model for medicine box appearance recognition and text recognition, all models need to be trained, including the medicine box appearance image recognition model based on yolov5, the text detection model based on DBnet, and the text recognition model based on DenseNet+CTC.

[0016] The steps of setting up a camera above the medicine box and collecting the appearance image information of the medicine box include:

[0017] The medicine box is placed in a medicine box information collection box, and a camera is set up above it to take a bird's-eye view of the medicine box. The inner wall of the medicine box information collection box is made of non-reflective black material and provides a stable light source;

[0018] Use the pre-trained yolov5-based medicine box recognition model to detect and identify the medicine box, and obtain the medicine box edge and the unique medicine number based on the appearance of the medicine box;

[0019] Crop the original image according to the obtained medicine box edges to obtain multiple images in which each image contains only a single medicine box;

[0020] For each cropped medicine box image, perform text detection using a pre-trained DBNet-based text detection model to obtain the area containing the medicine box text.

[0021] For the recognized text area, the pre-trained DenseNet and CTC-based text recognition model is used to perform text recognition and identify the text information of the medicine box.

[0022] Furthermore, the training of the medicine box appearance image recognition model based on yolov5 includes the following steps:

[0023] The medicine box is placed in a medicine box information collection box, and a movable camera is set up above it. The inner wall of the medicine box information collection box is made of non-reflective black material and provides a stable light source;

[0024] The camera takes pictures of all the medicines to be collected from a bird's-eye view to obtain sufficient pictures of the appearance of the medicine boxes. To ensure the quality of training, no less than 20 valid pictures should be collected for each medicine box.

[0025] Use the labeling tool to label the obtained medicine box images with regions and categories;

[0026] Load yolov5s pre-trained model;

[0027] During the training process, the appearance data of the medicine box is enhanced through mosaic data enhancement and other methods. After appropriate training rounds, the neural network model based on YOLOv5 reaches convergence.

[0028] Furthermore, the steps of training the DBnet-based text detection model and the DenseNet+CTC-based text recognition model include:

[0029] A public dataset containing approximately 3.64 million images was used, split into a training set and a validation set at a ratio of 99:1. The data was randomly generated from a Chinese corpus (news + classical Chinese texts) by varying font size, grayscale, blur, perspective, and stretching. The dataset contains 5,990 characters, including Chinese characters, English letters, numbers, and punctuation. Each sample consists of 10 characters, randomly selected from sentences in the corpus. The image resolution is uniformly 280×3.

[0030] Use text synthesis programs to synthesize text images of commonly used medicines to increase the amount of training data;

[0031] Load pre-trained models trained on other large text recognition datasets;

[0032] The dataset is used to train the DBnet text detection model and the DenseNet+CTC-based text recognition model until the models converge.

[0033] Furthermore, the pre-trained yolov5-based medicine box recognition model is used to detect and identify the medicine box, and the yolov5 neural network model in the medicine box recognition results based on the edge and appearance is obtained. The model is constructed using yolov5s, which consists of five parts: input, backbone, neck, head and output.

[0034] The input image size is 640×640×3, and the image is preprocessed using strategies such as Mosaic data enhancement, Anchor adaptive anchor box calculation, and image scaling;

[0035] In Yolov5, CSPDarknet53 is used as the backbone network of the model, including the Focus module, Conv module, C3 module, and SPP module. Its function is to extract rich semantic features from the input image; the neck uses FPN and PAN to generate a feature pyramid to enhance the detection of multi-scale targets; the head predicts the features transmitted from the neck and generates three feature maps of different scales;

[0036] The Conv module has a structure of Conv2d+BN+SiLU, which consists of a convolutional layer, a normalization operation, and an activation function. The Focus module aims to reduce the computational complexity of the model and speed up network training. First, the input image of size 3×640×640 is divided into four slices, each of which is 3×320×320. Then, the four slices are concatenated along the channel dimension using a splicing operation, resulting in a feature map of size 12×320×320. After another convolution operation, the final feature map is 32×320×320.

[0037] The C3 module consists of two branches. In the first branch, the input feature map passes through three consecutive Conv modules and multiple stacked Bottleneck modules. In the second branch, the feature map passes through only one Conv module, and the two branches are finally spliced together channel by channel. The Bottleneck module can better extract high-level features of the target and mainly consists of two consecutive convolution operations and a residual operation.

[0038] The SPP module is a spatial pyramid pooling module used to expand the receptive field of the network. In yolov5s, the input feature map of the SPP module is 512×20×20 in size, and the number of channels is halved after passing through a Conv module. Then, the maximum pooling operation with convolution kernels of 5×5, 9×9, and 13×13 is applied to the feature map, and the three feature maps are concatenated with the input feature map by channel and then passed through a Conv module. The final output feature map size is 512×20×20.

[0039] Furthermore, the DBNet text detection model used in the pre-trained DBNet-based text detection is composed of a DBNet text detection network. DBNet mainly uses differentiable binarization as a text detector based on a simple segmentation network. The differentiable binarization formula is shown in Equation 1:

[0040] (1)

[0041] Where B represents the approximate binary image, T is the threshold feature map learned by the network, and k represents the magnification factor; the empirical value here is 50, which can better distinguish the foreground and background.

[0042] Furthermore, a pre-trained DenseNet and CTC-based text recognition model is used for text recognition. The DenseNet and CTC text recognition model consists of a DenseNet neural network consisting of a CNN layer, an RNN layer, and a CTC transcription layer;

[0043] The functions of each layer of the text recognition model from bottom to top are as follows: Convolutional layer (CNN) extracts features of the input image; Recurrent layer (RNN) predicts the distribution of the input feature sequence of the CNN layer; Convolutional layer (CTC) filters redundant data of the sequence label;

[0044] The DenseNet network uses Relu as the activation function and uses three Dense Block layers for calculation. Each Dense Block is connected together through the Transition structure to form a DenseNet network. It is trained with CTC_loss to obtain the final data model.

[0045] The DenseNet network uses skip splicing to retain the original features and reduce the occurrence of gradient vanishing. However, as the network depth continues to deepen, the number of channels and parameters increases, making it difficult for the model to extract deep features. Therefore, DenseNet sets up a Transition conversion module; this module is used after the Dense Block and is mainly used to reduce the number of channels; at the same time, in order to reduce the number of channels, a bottleneck structure is added before each DenseBlock splicing to reduce the number of channels; the parameters are reduced by pooling and then transmitted to the Dense Block structure of the lower layer, thereby achieving higher accuracy.

[0046] Furthermore, each word or phrase in the text recognition result is sequentially subjected to a low-threshold fuzzy search and matching in the drug database. The fuzzy search and matching content is the name, manufacturer, dosage and other attributes of the drug. The steps for obtaining one or more drug numbers are as follows:

[0047] Taking into account the possibility that incomplete text may be recognized in medicine box text recognition, the matching threshold of fuzzy matching is low, and the low threshold is 0.5, that is, a match is considered successful if half of the characters are identical; text recognition can obtain multiple words or sentences, and a low-threshold fuzzy search and match is performed in the drug database. The content of the fuzzy search and match is first the name of the drug. If the drug name matches, the manufacturer, dosage and other attributes are continued to be matched in the matched drugs. If one of the attributes does not match any drug, it is considered a match failure; a match failure will not narrow the matching range, and the matching of the attribute will be directly skipped; finally, one or more drug numbers can be obtained through fuzzy search and matching.

[0048] Furthermore, the unique drug number obtained through appearance recognition is searched against the drug number retrieved through text recognition. If the search is successful, the medicine box recognition result is directly output; if the search fails, each word or phrase in the text recognition result is intelligently matched against the medicine database in turn, and the medicine box recognition result is finally obtained:

[0049] Considering the situation where no drugs can be found in the text recognition results due to errors in the appearance recognition results, the information of the medicine box can be confirmed entirely by text recognition. First, each word or phrase in the text recognition results is matched with the name attributes of the drug in the drug database using an intelligent matching algorithm. After obtaining the matching drugs, the intelligent matching algorithm is used to match the brand, manufacturer, dosage and other attributes of these drugs, and finally a drug number is obtained.

[0050] The intelligent matching algorithm is as follows:

[0051] The first attribute to be matched is the drug name. The initial matching threshold is 1, meaning that an exact match is considered a successful match. If no drug is matched, the match is considered a failure. If a match fails, the matching threshold is lowered by 0.1 until a match is successful. After the first successful match with a threshold other than 1, the matching threshold is increased by 0.01 until a match fails. After a match fails, the previous successful match is considered the final match result.

[0052] The present invention also claims protection for a real-time medicine box recognition system based on deep learning, comprising an image acquisition device and a data processing device.

[0053] The image acquisition device is used to acquire appearance image information of the medicine box;

[0054] The data processing device is used to realize real-time recognition of the collected medicine box appearance image information based on yolov5 and text recognition. The data processing device includes a medicine box recognition module based on yolov5, a text detection module based on DBNet, a text recognition module based on DenseNet and CTC, and a medicine box recognition module based on text recognition and appearance recognition;

[0055] The system can implement the above-mentioned real-time medicine box recognition method based on deep learning.

[0056] The beneficial effects achieved by the present invention are:

[0057] The present invention discloses a real-time medicine box recognition method based on deep learning. The appearance picture of the medicine box is obtained through the camera; the pre-trained yolov5-based medicine box recognition model is used to detect and identify the medicine box, and the medicine box edge and appearance-based medicine box recognition results are obtained; the original image is cropped according to the obtained medicine box edge to obtain multiple images in which each image contains only a single medicine box; for each cropped medicine box image, text detection is performed using a pre-trained DBNet-based text detection model to obtain the area of the medicine box text; for the identified text area, text recognition is performed using a pre-trained DenseNet and CTC-based text recognition model to identify the text information of the medicine box; each word or phrase of the text recognition result is subjected to a low-threshold fuzzy search and matching in the drug database in turn, and the fuzzy search and matching content is the drug name, brand, manufacturer, and dosage attributes of the drug to obtain one or more drug numbers; the unique drug number identified based on appearance is searched in the drug number queried based on text recognition; if the search is successful, the recognition result is output; if the search fails, each word or phrase of the text recognition result is subjected to an intelligent fuzzy search and matching in the drug database in turn, and finally the medicine box recognition result is obtained. The present invention integrates image recognition and text recognition technologies to greatly ensure recognition accuracy and ensure the safety of patients' medication. The present invention can effectively solve the problem of manual verification required when dispensing medications in pharmacies. Moreover, the present method greatly ensures the accuracy of recognition, reduces the workload of pharmacy workers, and ensures the safety of patients' medication. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] Figure 1 This is a flowchart of a method for real-time identification of medicine boxes based on deep learning provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0059] The present invention will be further described below with reference to the accompanying drawings and specific embodiments so that those skilled in the art can better understand the present invention and implement it. However, the embodiments given are not intended to limit the present invention. Unless there is a conflict, the embodiments of the present invention and the technical features in the embodiments may be combined with each other.

[0060] like Figure 1 As shown, the present invention proposes a real-time drug recognition method based on deep learning, which includes the following steps:

[0061] S100, setting up a camera above the medicine box to collect appearance image information of the medicine box;

[0062] The medicine box is placed in a medicine box information collection box, and a camera is set up above it. In order to reduce the interference of platform reflection on the recognition result, the inner wall of the medicine box information collection box is made of non-reflective black material and a stable industrial light source is provided inside.

[0063] S200, using a pre-trained yolov5-based medicine box recognition model to detect and recognize the medicine box, and obtain the medicine box edge and the unique medicine number of the medicine box based on the appearance.

[0064] The yolov5 neural network model is constructed using yolov5s, which consists of five parts: input, backbone, neck, head, and output.

[0065] The input image size is 640×640×3, and the image is preprocessed using strategies such as Mosaic data enhancement, Anchor adaptive anchor box calculation, and image scaling;

[0066] In yolov5, CSPDarknet53 is used as the backbone network of the model, including Focus module, Conv module, C3 module and SPP module, and FPN and PAN are used to generate feature pyramid in the neck;

[0067] The Conv module has a structure of Conv2d+BN+SiLU, which consists of a convolution layer, a normalization operation, and an activation function. The Focus module first divides the input image of size 3×640×640 into four slices, where each slice is 3×320×320. It then uses a splicing operation to splice the four slices together along the channel dimension, resulting in a feature map of size 12×320×320. After another convolution operation, the final feature map is 32×320×320.

[0068] The C3 module consists of two branches. In the first branch, the input feature map passes through three consecutive Conv modules and multiple stacked Bottleneck modules. In the second branch, the feature map passes through only one Conv module, and finally the two branches are spliced together channel by channel. The Bottleneck module mainly consists of two consecutive convolution operations and a residual operation.

[0069] The SPP module is a spatial pyramid pooling module. The input feature map of the SPP module is 512×20×20 in size, and the number of channels is halved after passing through a Conv module. The maximum pooling operation with convolution kernels of 5×5, 9×9, and 13×13 is then performed on the feature map. The three feature maps are concatenated with the input feature map by channel and then passed through a Conv module. The final output feature map size is 512×20×20.

[0070] The steps to train the yolov5-based medicine box appearance image recognition model are as follows:

[0071] The medicine box is placed in a medicine box information collection box, and a movable camera is set up above it. The inner wall of the medicine box information collection box is made of non-reflective black material and provides a stable light source;

[0072] Considering image quality, a camera with good product performance was selected. The camera took photos of all the medications to be collected from a bird's-eye view, obtaining sufficient images of the medicine boxes' exteriors. To ensure training quality, no fewer than 20 valid images were collected for each medicine box. To ensure effective medicine box recognition, the angles of the collected medicine box images should match the angles used when capturing the photos in step 100.

[0073] Use the labeling tool to label the area and category of the obtained medicine box image; the labeling should be close to the edge of the medicine box.

[0074] Load yolov5s pre-trained model;

[0075] During the training process, the appearance data of the medicine box is enhanced by mosaic data enhancement and other methods. After appropriate training rounds, the neural network model based on YOLOv5 reaches convergence.

[0076] After completing the model training, in this step, the trained yolov5-based medicine box recognition model is used to detect and identify the collected medicine box photos to obtain the medicine box edge and the unique medicine number of the medicine box based on the appearance.

[0077] S300: Crop the original image according to the obtained medicine box edge to obtain multiple images, each of which contains only a single medicine box.

[0078] According to the edges of the medicine box detected by yolov5, the original image is cropped to obtain multiple medicine box images, and each image contains only a single medicine box.

[0079] S400: Perform text detection on each cropped medicine box image using a pre-trained DBNet-based text detection model to obtain the area of the medicine box text.

[0080] The DBNet text detection model consists of the DBNet text detection network, which mainly uses differentiable binarization as a text detector based on a simple segmentation network. The differentiable binarization formula is as follows:

[0081] (1)

[0082] Where B represents the approximate binary image, T is the threshold feature map learned by the network, and k represents the magnification factor.

[0083] S500 : For the recognized text area, use a pre-trained DenseNet and CTC-based text recognition model to perform text recognition to identify the text information of the medicine box.

[0084] The DenseNet and CTC text recognition model consists of a DenseNet neural network consisting of a CNN layer, an RNN layer, and a CTC transcription layer;

[0085] The functions of each layer of the text recognition model from bottom to top are as follows: Convolutional layer (CNN) extracts features of the input image; Recurrent layer (RNN) predicts the distribution of the input feature sequence of the CNN layer; Convolutional layer (CTC) filters redundant data of the sequence label;

[0086] The DenseNet network uses Relu as the activation function and uses three Dense Block layers for calculation. Each Dense Block is connected together through the Transition structure to form a DenseNet network. It is trained with CTC_loss to obtain the final data model.

[0087] The DenseNet network uses skip splicing to retain the original features and reduce the occurrence of gradient vanishing. However, as the network depth continues to deepen, the number of channels and parameters increases, making it difficult for the model to extract deep features. Therefore, DenseNet sets up a Transition conversion module; this module is used after the Dense Block and is mainly used to reduce the number of channels; at the same time, in order to reduce the number of channels, a bottleneck structure is added before each DenseBlock splicing to reduce the number of channels; the parameters are reduced by pooling and then transmitted to the Dense Block structure of the lower layer, thereby achieving higher accuracy.

[0088] The steps for training the DBnet-based text detection model and the DenseNet+CTC-based text recognition model include:

[0089] A public dataset containing approximately 3.64 million images was used, split into a training set and a validation set at a ratio of 99:1. The data was randomly generated from a Chinese corpus (news + classical Chinese texts) by varying font size, grayscale, blur, perspective, and stretching. The dataset contains 5,990 characters, including Chinese characters, English letters, numbers, and punctuation. Each sample consists of 10 characters, randomly selected from sentences in the corpus. The image resolution is uniformly 280×3.

[0090] Use text synthesis programs to synthesize text images of commonly used medicines to increase the amount of training data;

[0091] Load pre-trained models trained on other large text recognition datasets;

[0092] The dataset is used to train the DBnet text detection model and the DenseNet+CTC-based text recognition model until the models converge.

[0093] S600. Perform a low-threshold fuzzy search and match on each word or phrase in the text recognition result in the drug database in turn. The fuzzy search and match content is the drug name, brand, manufacturer, and dosage attributes to obtain one or more drug numbers.

[0094] Taking into account the possibility that incomplete text may be recognized in medicine box text recognition, the matching threshold of fuzzy matching is low, and the low threshold is 0.5, that is, a match is considered successful if half of the characters are identical; text recognition can obtain multiple words or sentences, and a low-threshold fuzzy search and match is performed in the drug database. The content of the fuzzy search and match is first the name of the drug. If the drug name matches, the manufacturer, dosage and other attributes are continued to be matched in the matched drugs. If one of the attributes does not match any drug, it is considered a match failure; a match failure will not narrow the matching range, and the matching of the attribute will be directly skipped; finally, one or more drug numbers can be obtained through fuzzy search and matching.

[0095] S700: Search the unique drug number obtained through appearance recognition among the drug numbers found through text recognition:

[0096] The unique drug number based on appearance recognition is searched among the drug numbers retrieved based on text recognition. If the search is successful, the medicine box recognition result is directly output.

[0097] S800: Search the unique drug number obtained through appearance recognition against the drug numbers found through text recognition. If the search fails, intelligently match each word or phrase in the text recognition result against the drug database in sequence, ultimately obtaining the medicine box recognition result.

[0098] Considering the situation where no drugs can be found in the text recognition results due to errors in the appearance recognition results, the information of the medicine box can be confirmed entirely by text recognition. First, each word or phrase in the text recognition results is matched with the name attributes of the drug in the drug database using an intelligent matching algorithm. After obtaining the matching drugs, the intelligent matching algorithm is used to match the brand, manufacturer, dosage and other attributes of these drugs, and finally a drug number is obtained.

[0099] The intelligent matching algorithm is as follows:

[0100] The first attribute to be matched is the drug name. The initial matching threshold is 1, meaning that an exact match is considered a successful match. If no drug is matched, the match is considered a failure. If a match fails, the matching threshold is lowered by 0.1 until a match is successful. After the first successful match with a threshold other than 1, the matching threshold is increased by 0.01 until a match fails. After a match fails, the previous successful match is considered the final match result.

[0101] An embodiment of the present invention also provides a real-time medicine box recognition system based on deep learning, including an image acquisition device and a data processing device. The system can implement the real-time medicine box recognition method based on deep learning described in the above embodiment.

[0102] The image acquisition device is used to acquire appearance image information of the medicine box.

[0103] The data processing device is used to realize real-time recognition of the collected medicine box appearance image information based on YOLOv5 and text recognition. The data processing device includes a medicine box recognition module based on YOLOv5, a text detection module based on DBNet, a text recognition module based on DenseNet and CTC, and a medicine box recognition module based on text recognition and appearance recognition.

[0104] The step of setting up a camera above the medicine box to collect image information of the appearance of the medicine box includes:

[0105] Before using the neural network model for medicine box appearance recognition and text recognition, all models need to be trained, including the medicine box appearance image recognition model based on yolov5, the text detection model based on DBnet, and the text recognition model based on DenseNet+CTC.

[0106] The steps of setting up a camera above the medicine box and collecting the appearance image information of the medicine box include:

[0107] The medicine box is placed in a medicine box information collection box, and a camera is set up above it to take a bird's-eye view of the medicine box. The inner wall of the medicine box information collection box is made of non-reflective black material and provides a stable light source.

[0108] Use the pre-trained yolov5-based medicine box recognition model to detect and identify the medicine box, and obtain the medicine box edge and the unique medicine number based on the appearance of the medicine box;

[0109] Crop the original image according to the obtained medicine box edges to obtain multiple images in which each image contains only a single medicine box;

[0110] For each cropped medicine box image, perform text detection using a pre-trained DBNet-based text detection model to obtain the area containing the medicine box text.

[0111] For the identified text area, use the pre-trained DenseNet and CTC-based text recognition model to perform text recognition and identify the text information on the medicine box;

[0112] Each word or phrase in the text recognition result is sequentially subjected to a low-threshold fuzzy search and matching in the drug database. The fuzzy search and matching content is the drug name, brand, manufacturer, and dosage attributes, and one or more drug numbers are obtained;

[0113] Search the unique drug number identified by appearance among the drug numbers found by text recognition; if the search is successful, output the recognition result;

[0114] If the search fails, each word or phrase in the text recognition result will be subjected to intelligent fuzzy search and matching in the drug database in turn, and finally the medicine box recognition result will be obtained.

[0115] The training of the medicine box appearance image recognition model based on yolov5 includes the following steps:

[0116] The medicine box is placed in a medicine box information collection box, and a movable camera is set up above it. The inner wall of the medicine box information collection box is made of non-reflective black material and provides a stable light source;

[0117] The camera takes pictures of all the medicines to be collected from a bird's-eye view to obtain sufficient pictures of the appearance of the medicine boxes. To ensure the quality of training, no less than 20 valid pictures should be collected for each medicine box.

[0118] Use the labeling tool to label the obtained medicine box images with regions and categories;

[0119] Load yolov5s pre-trained model;

[0120] During the training process, the appearance data of the medicine box is enhanced through mosaic data enhancement and other methods. After appropriate training rounds, the neural network model based on YOLOv5 reaches convergence.

[0121] The steps for training the DBnet-based text detection model and the DenseNet+CTC-based text recognition model include:

[0122] A public dataset containing approximately 3.64 million images was used, split into a training set and a validation set at a ratio of 99:1. The data was randomly generated from a Chinese corpus (news + classical Chinese texts) by varying font size, grayscale, blur, perspective, and stretching. The dataset contains 5,990 characters, including Chinese characters, English letters, numbers, and punctuation. Each sample consists of 10 characters, randomly selected from sentences in the corpus. The image resolution is uniformly 280×3.

[0123] Use text synthesis programs to synthesize text images of commonly used medicines to increase the amount of training data;

[0124] Load pre-trained models trained on other large text recognition datasets;

[0125] The dataset is used to train the DBnet text detection model and the DenseNet+CTC-based text recognition model until the models converge.

[0126] Using the pre-trained yolov5-based medicine box recognition model, the medicine box is detected and recognized, and the medicine box edge and appearance-based medicine box recognition results are obtained. The yolov5 neural network model is constructed using yolov5s and consists of five parts: input, backbone, neck, head, and output.

[0127] The input image size is 640×640×3, and the image is preprocessed using strategies such as Mosaic data enhancement, Anchor adaptive anchor box calculation, and image scaling;

[0128] In Yolov5, CSPDarknet53 is used as the backbone network of the model, including the Focus module, Conv module, C3 module, and SPP module. Its function is to extract rich semantic features from the input image; the neck uses FPN and PAN to generate a feature pyramid to enhance the detection of multi-scale targets; the head predicts the features transmitted from the neck and generates three feature maps of different scales;

[0129] The Conv module has a structure of Conv2d+BN+SiLU, which consists of a convolutional layer, a normalization operation, and an activation function. The Focus module aims to reduce the computational complexity of the model and speed up network training. First, the input image of size 3×640×640 is divided into four slices, each of which is 3×320×320. Then, the four slices are concatenated along the channel dimension using a splicing operation, resulting in a feature map of size 12×320×320. After another convolution operation, the final feature map is 32×320×320.

[0130] The C3 module consists of two branches. In the first branch, the input feature map passes through three consecutive Conv modules and multiple stacked Bottleneck modules. In the second branch, the feature map passes through only one Conv module, and the two branches are finally spliced together channel by channel. The Bottleneck module can better extract high-level features of the target and mainly consists of two consecutive convolution operations and a residual operation.

[0131] The SPP module is a spatial pyramid pooling module used to expand the receptive field of the network. In yolov5s, the input feature map of the SPP module is 512×20×20 in size, and the number of channels is halved after passing through a Conv module. Then, the maximum pooling operation with convolution kernels of 5×5, 9×9, and 13×13 is applied to the feature map, and the three feature maps are concatenated with the input feature map by channel and then passed through a Conv module. The final output feature map size is 512×20×20.

[0132] The DBNet text detection model used in the text detection is composed of a DBNet text detection network. DBNet mainly uses differentiable binarization as a text detector based on a simple segmentation network. The differentiable binarization formula is shown in Equation 1:

[0133] (1)

[0134] Where B represents the approximate binary image, T is the threshold feature map learned by the network, and k represents the magnification factor; the empirical value here is 50, which can better distinguish the foreground and background.

[0135] The DenseNet and CTC-based text recognition model used for text recognition is composed of a DenseNet neural network consisting of a CNN layer, an RNN layer, and a CTC transcription layer.

[0136] The functions of each layer of the text recognition model from bottom to top are as follows: Convolutional layer (CNN) extracts features of the input image; Recurrent layer (RNN) predicts the distribution of the input feature sequence of the CNN layer; Convolutional layer (CTC) filters redundant data of the sequence label;

[0137] The DenseNet network uses Relu as the activation function and uses three Dense Block layers for calculation. Each Dense Block is connected together through the Transition structure to form a DenseNet network. It is trained with CTC_loss to obtain the final data model.

[0138] The DenseNet network uses skip splicing to retain the original features and reduce the occurrence of gradient vanishing. However, as the network depth continues to deepen, the number of channels and parameters increases, making it difficult for the model to extract deep features. Therefore, DenseNet sets up a Transition conversion module; this module is used after the Dense Block and is mainly used to reduce the number of channels; at the same time, in order to reduce the number of channels, a bottleneck structure is added before each DenseBlock splicing to reduce the number of channels; the parameters are reduced by pooling and then transmitted to the Dense Block structure of the lower layer, thereby achieving higher accuracy.

[0139] Each word or phrase in the text recognition result is sequentially subjected to a low-threshold fuzzy search and matching in the drug database. The fuzzy search and matching content is the drug name, manufacturer, dosage and other attributes. The steps to obtain one or more drug numbers are as follows:

[0140] Taking into account the possibility that incomplete text may be recognized in medicine box text recognition, the matching threshold of fuzzy matching is low, and the low threshold is 0.5, that is, a match is considered successful if half of the characters are identical; text recognition can obtain multiple words or sentences, and a low-threshold fuzzy search and match is performed in the drug database. The content of the fuzzy search and match is first the name of the drug. If the drug name matches, the manufacturer, dosage and other attributes are continued to be matched in the matched drugs. If one of the attributes does not match any drug, it is considered a match failure; a match failure will not narrow the matching range, and the matching of the attribute will be directly skipped; finally, one or more drug numbers can be obtained through fuzzy search and matching.

[0141] The unique drug number obtained through appearance recognition is searched against the drug number retrieved through text recognition. If the search is successful, the medicine box recognition result is directly output; if the search fails, each word or phrase in the text recognition result is intelligently matched against the medicine database in turn, and the medicine box recognition result is finally obtained:

[0142] Considering the situation where no drugs can be found in the text recognition results due to errors in the appearance recognition results, the information of the medicine box can be confirmed entirely by text recognition. First, each word or phrase in the text recognition results is matched with the name attributes of the drug in the drug database using an intelligent matching algorithm. After obtaining the matching drugs, the intelligent matching algorithm is used to match the brand, manufacturer, dosage and other attributes of these drugs, and finally a drug number is obtained.

[0143] The intelligent matching algorithm is as follows:

[0144] The first attribute to be matched is the drug name. The initial matching threshold is 1, meaning that an exact match is considered a successful match. If no drug is matched, the match is considered a failure. If a match fails, the matching threshold is lowered by 0.1 until a match is successful. After the first successful match with a threshold other than 1, the matching threshold is increased by 0.01 until a match fails. After a match fails, the previous successful match is considered the final match result.

[0145] Although preferred embodiments of the present invention have been described, those skilled in the art may make additional changes and modifications to these embodiments once they are aware of the basic inventive concepts. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present invention. Obviously, those skilled in the art may make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, the present invention is intended to include such changes and modifications as fall within the scope of the claims and their equivalents.

Claims

1. A real-time medicine box recognition method based on deep learning, characterized in that: The following steps are involved: A camera is installed above the medicine box to collect image information of the appearance of the medicine box; Use the pre-trained yolov5-based medicine box recognition model to detect and identify medicine boxes, and obtain medicine box edge and appearance-based medicine box recognition results; The original image is cropped according to the obtained medicine box edges to obtain multiple images, each containing only a single medicine box. For each cropped medicine box image, text detection is performed using a pre-trained DBNet-based text detection model to obtain the area containing the medicine box text. For the identified text area, use the pre-trained DenseNet and CTC-based text recognition model to perform text recognition and identify the text information on the medicine box; Each word or phrase in the text recognition result is sequentially subjected to a low-threshold fuzzy search and matching in the drug database. The fuzzy search and matching content includes the drug name, brand, manufacturer, and dosage attributes to obtain one or more drug numbers; Search the unique drug number based on appearance recognition among the drug numbers retrieved based on text recognition; If the search is successful, the recognition result is output; If the search fails, each word or phrase in the text recognition result is subjected to intelligent fuzzy search and matching in the drug database in turn, and finally the medicine box recognition result is obtained; Each word or phrase in the text recognition result is subjected to intelligent fuzzy search and matching in the drug database in turn, and the steps to finally obtain the medicine box recognition result are as follows: Each word or phrase in the text recognition result is matched with the drug name attribute in the drug database using an intelligent matching algorithm. After the matching drugs are screened out, the brand, manufacturer, and dosage attributes of these drugs are matched in turn using an intelligent matching algorithm to finally obtain a drug number. The intelligent matching algorithm is as follows: The first thing to be matched is the drug name attribute. The initial matching threshold is 1, that is, if they are exactly the same, it is considered a successful match; if any drug is not matched, it is considered a match failure; if the match fails, the matching threshold is lowered by 0.1 until the match is successful; after the first match is successful with a threshold not equal to 1, the matching threshold is increased by 0.01 in sequence until the match fails; after a match fails, the result of the last successful match before the match failure is regarded as the final match result.

2. A method for real-time identification of medicine boxes based on deep learning according to claim 1, characterized in that: The step of setting up a camera above the medicine box to collect appearance image information of the medicine box includes: Before using the neural network model for medicine box appearance recognition and text recognition, all models are trained, including the medicine box appearance image recognition model based on yolov5, the text detection model based on DBnet, and the text recognition model based on DenseNet+CTC.

3. The method for real-time identification of medicine boxes based on deep learning according to claim 2, characterized in that: The training of the medicine box appearance image recognition model based on yolov5 includes the following steps: The medicine box is placed in a medicine box information collection box, and a movable camera is set up above it. The inner wall of the medicine box information collection box is made of non-reflective black material and provides a stable light source; The camera takes pictures of all the medicines to be collected from a bird's-eye view to obtain the required number of pictures of the medicine box appearance. To ensure the quality of training, the number of valid pictures collected for each medicine box should not be less than 20; Use the labeling tool to label the obtained medicine box images with regions and categories; Load yolov5s pre-trained model; During the training process, the appearance data of the medicine box is enhanced, and appropriate training rounds are performed until the neural network model based on yolov5 reaches convergence.

4. The method for real-time identification of medicine boxes based on deep learning according to claim 2, characterized in that: The steps of training the DBnet-based text detection model and the DenseNet+CTC-based text recognition model include: A public dataset was used, split into a training set and a validation set at a ratio of 99:

1. The data was extracted from a Chinese corpus, and a total of 5,990 characters, including Chinese characters, English letters, numbers, and punctuation, were randomly generated by varying parameters such as font, size, grayscale, blur, perspective, and stretching. Each sample contained 10 characters, randomly extracted from sentences in the Chinese corpus, and the image resolution was uniformly 280×3. Use text synthesis programs to synthesize text images of commonly used medicines to increase the amount of training data; Load pre-trained models trained on other large text recognition datasets; The dataset is used to train the DBnet text detection model and the DenseNet+CTC-based text recognition model until the models converge.

5. The method for real-time identification of medicine boxes based on deep learning according to claim 1, characterized in that: The pre-trained yolov5-based medicine box recognition model is used to detect and identify medicine boxes, and the yolov5 neural network model in the medicine box recognition results based on the edge and appearance is constructed using yolov5s. It consists of five parts: input end, backbone network, neck and head, and output end. composition: The input image size is 640×640×3, and the image is preprocessed using Mosaic data enhancement, Anchor adaptive anchor box calculation, and image scaling strategy; In yolov5, CSPDarknet53 is used as the backbone network of the model, including Focus module, Conv module, C3 module and SPP module; FPN and PAN are used to generate feature pyramid in the neck; The Conv module has a structure of Conv2d+BN+SiLU, which consists of a convolution layer, a normalization operation, and an activation function. The Focus module first divides the input image of size 3×640×640 into four slices, where each slice is 3×320×320. It then uses a splicing operation to splice the four slices together along the channel dimension, resulting in a feature map of size 12×320×320. After another convolution operation, the final feature map is 32×320×320. The C3 module consists of two branches. In the first branch, the input feature map passes through three consecutive Conv modules and multiple stacked Bottleneck modules. In the second branch, the feature map passes through only one Conv module. Finally, the two branches are spliced together channel by channel. The Bottleneck module includes two consecutive convolution operations and one residual operation. The SPP module is a spatial pyramid pooling module. The input feature map of the SPP module is 512×20×20 in size, and the number of channels is halved after passing through a Conv module. The maximum pooling operation with convolution kernels of 5×5, 9×9, and 13×13 is then performed on the feature map. The three feature maps are concatenated with the input feature map by channel and then passed through a Conv module. The final output feature map size is 512×20×20.

6. The method for real-time identification of medicine boxes based on deep learning according to claim 1, characterized in that: The DBNet text detection model used in the text detection using the pre-trained DBNet-based text detection model is composed of a DBNet text detection network, which uses differentiable binarization as a text detector based on a simple segmentation network. The differentiable binarization formula is shown in Formula 1: （1） Where B represents the approximate binary image, T is the threshold feature map learned by the network, and k represents the magnification factor.

7. The method for real-time identification of medicine boxes based on deep learning according to claim 1, characterized in that: The structure of the DenseNet and CTC text recognition model for text recognition using the pre-trained DenseNet and CTC-based text recognition model is as follows: The DenseNet and CTC text recognition models consist of CNN layers, RNN layers, and CTC transcription layers; The DenseNet network uses Relu as the activation function and uses three Dense Block layers for calculation. Each DenseBlock is connected together through the Transition structure to form a DenseNet network. It is trained with CTC_loss to obtain the final data model. DenseNet sets up a Transition conversion module, which is used after the Dense Block. It adds a bottleneck structure before each DenseBlock splicing, reduces the parameters through pooling, and transmits them to the Dense Block structure of the lower layer.

8. The method for real-time identification of medicine boxes based on deep learning according to claim 1, characterized in that: The steps of performing a low-threshold fuzzy search and matching on each word or phrase in the text recognition result in turn in the drug database, wherein the fuzzy search and matching content is the drug name, brand, manufacturer, and dosage attribute of the drug, and obtaining one or more drug numbers are as follows: The low threshold is 0.5, that is, a match is considered successful if half of the characters are identical; text recognition is used to obtain multiple words or sentences, and a low-threshold fuzzy search and match is performed in the drug database. The content of the fuzzy search and match is first the name of the drug. If the drug name matches, the brand, manufacturer, and dosage attributes are continued to be matched in the matched drugs. If one of the attributes does not match any drug, the match is considered a failure; a failed match will not narrow the matching range, and the matching of the attribute will be directly skipped; finally, one or more drug numbers can be obtained through fuzzy search and matching.

9. A real-time medicine box recognition system based on deep learning, characterized in that: including an image acquisition device and a data processing device, The image acquisition device is used to acquire appearance image information of the medicine box; The data processing device is used to realize real-time recognition of the collected medicine box appearance image information based on yolov5 and text recognition. The data processing device includes a medicine box recognition module based on yolov5, a text detection module based on DBNet, a text recognition module based on DenseNet and CTC, and a medicine box recognition module based on text recognition and appearance recognition; The system can implement the deep learning-based real-time medicine box identification method described in any one of claims 1 to 8.

Images