Drug label recognition method based on image and text embedding based on deep learning

Through deep learning image and text embedding methods, combined with optical character recognition and CTPN network, the problem of relying on manual input and historical record library in the recognition of illegal drugs is solved, and the rapid identification and accurate judgment of illegal drugs is achieved, and the investigation efficiency is improved. and the timeliness of the dataset.

CN113920335BActive Publication Date: 2024-11-22SUZHOU LENGWANG NETWORK TECH CO LTD
View PDF 7 Cites 0 Cited by

Patent Information

Application Number
CN202111167848.X
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2021-09-28
Publication Date
2024-11-22
Estimated Expiration
2041-09-28

AI Technical Summary

Technical Problem

The prior art relies on manual input and historical record library for the identification of illegal drugs, and cannot quickly identify drugs outside the record library, resulting in low investigation efficiency and difficulty in analyzing similarity based on image content.

Method used

Using deep learning-based image and text embedding methods, combining Tesseract OCR and CTPN network structures for drug image recognition, using cosine similarity and perceived hashing algorithms to process image content and text information, and integrating similarity ranking and model training to improve recognition accuracy sex.

Benefits of technology

The accuracy of drug label identification and investigation and judgment efficiency are improved, the timeliness and accuracy of the data set are ensured, and the work efficiency of investigators is significantly improved by combining picture content and text information.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN113920335B_ABST
    Figure CN113920335B_ABST
Patent Text Reader

Abstract

The present invention discloses a drug label recognition method based on image and text embedding of deep learning, comprising the steps of: obtaining drug label information data, the label information data including drug picture information; extracting the input drug picture information in combination with optical character recognition, the optical character recognition using Tesseract OCR engine to complete the recognition and extraction of image text; processing the extracted picture text information based on a similarity module; processing the extracted picture content based on a similarity module; comparing the top K pictures with the highest similarity ranking obtained respectively based on a picture content text recognition result integration module, and performing model training after the comparison. The present invention improves the accuracy of judgment and the work efficiency of investigators by using a method combining picture content information and image text recognition information.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of medical image text recognition, and in particular to a drug label recognition method based on image and text embedding based on deep learning. Background Art

[0002] Illegal, unapproved, counterfeit and potentially risky drugs can cause serious harm to medical patients. The role of the law in drug regulation is limited and depends on people's compliance with the law. At present, the verification of illegal and banned drugs requires manual input of drug names for verification and is highly dependent on the historical illegal verification record library. In addition, it is impossible to make an immediate judgment on drugs outside the illegal drug name record library. It often takes a long time to investigate drugs outside the record library, which greatly reduces the efficiency of investigators. For regulatory investigators, how to immediately decide whether to investigate drugs outside the illegal drug record library is a very complicated issue.

[0003] At present, the image recognition technology based on deep learning is relatively mature. By comparing the similarity of image content, illegal drug images can be identified. However, due to the small size of the existing illegal drug data set and the high environmental requirements for similarity analysis based on image content, it is not suitable to use only image content retrieval. It is more feasible to use text recognition to identify text information such as drug names in images and compare text similarity to make a decision on whether to investigate. The main text recognition engine currently used by the industry is Tesseract OCR, but there are still major problems in text recognition in different backgrounds. Therefore, how to design a drug label recognition method that makes better use of image and text information is a very difficult task. Summary of the invention

[0004] The purpose of the present invention is to provide a drug label recognition method based on image and text embedding based on deep learning to solve the problems raised in the above background technology.

[0005] To achieve the above object, the present invention provides the following technical solution: a drug label recognition method based on image and text embedding based on deep learning, comprising the steps of:

[0006] S1: Obtain drug label information data, which includes drug image information;

[0007] S2: Extract the input drug image information by combining optical character recognition. Optical character recognition uses the Tesseract OCR engine to complete the recognition and extraction of image text;

[0008] S3: Process the extracted image text information based on the similarity module;

[0009] S4: Process the extracted image content based on the similarity module;

[0010] S5: The module integrating the results of text recognition based on the image content compares the top K images with the highest similarity rankings, and performs model training after the comparison.

[0011] Preferably, in S2, before the image is processed by the Tesseract OCR engine, the image background is also uniformly processed by the CTPN network structure.

[0012] Preferably, in S3, the effective information of each image that has been identified by CTPN and OCR is embedded and encoded to obtain a vector and the cosine similarity is calculated with the image label vector in the reference data set. The cosine similarity calculation is: Similarity(A,B)=AB / ||A||.||B||, A and B represent a sentence respectively, and the top K images ranked by similarity and the corresponding drug labels are found.

[0013] Preferably, in S4, a perceptual hashing algorithm is used to compare the similarities of the images. The perceptual hashing algorithm uses a discrete cosine transform to obtain the low-frequency part of the image. After the hash value of the image is calculated by perceptual hashing, the Hamming distance of the hash value is compared. If it is 0, it means that the two images are very similar, thereby obtaining the top K images ranked by similarity and the corresponding drug labels.

[0014] Preferably, in S5, the indicator accuracy P@K is set to represent the proportion of drug labels with the same labels as the retrieved drug labels and the test drug labels, where P1@K represents the label recognition accuracy obtained by using only the image content similarity, and P2@K represents the label recognition accuracy obtained by using the similarity calculated by embedding the image text information, and the indicator calculation method of P@K=0.5*P1@K+0.5*P2@K is used.

[0015] Preferably, after the model training is completed in S5, an accurate judgment is made as to whether the drug shown in the input drug image needs to be investigated, and the classification label result of the input drug image is added to the reference data set after manual verification, and the reference data set is updated.

[0016] Compared with the prior art, the present invention has the following beneficial effects:

[0017] The present invention utilizes the content information and text information of the image to perform label recognition on the image. Compared with the method of only using the image content similarity to retrieve the image to obtain the image label, the accuracy of drug label recognition is greatly improved; and it can make a judgment in a relatively short time whether the drug represented by a certain image needs to be investigated. The method of combining the image content information and the image text recognition information improves the accuracy of the judgment and the work efficiency of the investigators. The reference image data set is continuously updated during the continuous input of drug images, thereby ensuring timeliness; at the same time, the present invention manually verifies the retrieved images and adds them to the reference data set, which can further improve the accuracy and field applicability of the method. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 It is a logic block diagram of the method of the present invention;

[0019] Figure 2 4 is a model architecture diagram of CTPN in an embodiment of the present invention. DETAILED DESCRIPTION

[0020] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0021] See also Figure 1-2 The present invention provides a technical solution: a drug label recognition method based on image and text embedding based on deep learning, comprising the steps of:

[0022] S1: Obtain drug label information data, which includes drug image information;

[0023] S2: Extract the input drug image information by combining optical character recognition. Optical character recognition uses the Tesseract OCR engine to complete the recognition and extraction of image text;

[0024] S3: Process the extracted image text information based on the similarity module;

[0025] S4: Process the extracted image content based on the similarity module;

[0026] S5: The module integrating the results of text recognition based on the image content compares the top K images with the highest similarity rankings, and performs model training after the comparison.

[0027] In this embodiment, before the Tesseract OCR engine processes the image, S2 also uses the CTPN network structure to unify the image background. CTPN mainly includes VGG16 and bidirectional LSTM. Finally, the fully connected layer is used to obtain blocks of different information of the image. The image text information in these different blocks is extracted using optical character recognition to complete the recognition and extraction of image text.

[0028] In this embodiment, S3 will embed the effective information of each image that has been identified by CTPN and OCR, encode the effective information to obtain a vector, and perform cosine similarity calculation with the image label vector in the reference data set. The cosine similarity calculation is: Similarity(A,B)=AB / ||A||.||B||, A and B represent a sentence respectively, by finding the top K images ranked by similarity and the corresponding drug labels.

[0029] In this embodiment, the perceptual hashing algorithm is used in S4 to compare the similarity of the images. The perceptual hashing adopts the discrete cosine transform to obtain the low-frequency part of the image. After the hash value of the image is calculated by perceptual hashing, the Hamming distance of the hash value is compared. If it is 0, it means that the two images are very similar, thereby obtaining the top K images ranked by similarity and the corresponding drug labels.

[0030] In this embodiment, S5 sets the indicator accuracy rate P@K to represent the proportion of drug labels with the same labels as the test drug labels, where P1@K represents the label recognition accuracy obtained by using only the image content similarity, and P2@K represents the label recognition accuracy obtained by using the similarity calculated by embedding the image text information, and the indicator calculation method of P@K=0.5*P1@K+0.5*P2@K is used.

[0031] In this embodiment, after the model training is completed in S5, an accurate judgment is made as to whether the drug shown in the input drug image needs to be investigated, and the classification label result of the input drug image is added to the reference data set after manual verification, and the reference data set is updated.

[0032] In this embodiment, the method of the present invention is to first use scene text detection and recognition combined with an optical character recognition method to extract information from the input drug picture, and then use a text similarity-based method to calculate the cosine similarity between the vector represented by the picture text information and the label vector of the picture in the reference data set, so as to find the K candidate pictures with the highest similarity. Then, the input drug picture is used to find the K candidate pictures with the highest similarity from the reference data set using a similarity recognition method based on the picture content, and then the results obtained by the two methods are compared with the picture label to make a judgment on whether to investigate and whether to update the reference data set.

[0033] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A drug label recognition method based on image and text embedding based on deep learning, characterized in that: Includes steps: S1: Obtain drug label information data, which includes drug image information; S2: Extract the input drug image information by combining optical character recognition. Optical character recognition uses the TesseractOCR engine to complete the recognition and extraction of image text; S3: Process the extracted image text information based on the similarity module; S4: Process the extracted image content based on the similarity module; S5: The module integrating the results of text recognition based on the image content compares the top K images with the highest similarity rankings, and performs model training after the comparison.

2. The drug label recognition method based on image and text embedding based on deep learning according to claim 1 is characterized in that: In S2, before the image is processed by the Tesseract OCR engine, the image background is also uniformly processed by the CTPN network structure.

3. The drug label recognition method based on image and text embedding based on deep learning according to claim 1, characterized in that: In S3, the effective information of each image that has been identified by CTPN and OCR is embedded and encoded to obtain a vector and the cosine similarity is calculated with the image label vector in the reference data set. The cosine similarity calculation is: Similarity(A,B)=AB / ||A||.||B||, A and B represent a sentence respectively, and the top K images ranked by similarity and the corresponding drug labels are found.

4. The drug label recognition method based on image and text embedding based on deep learning according to claim 1, characterized in that: In S4, the perceptual hashing algorithm is used to compare the similarity of the images. The perceptual hashing adopts discrete cosine transform to obtain the low-frequency part of the image. After the hash value of the image is calculated by perceptual hashing, the Hamming distance of the hash value is compared. If it is 0, it means that the two images are very similar, thereby obtaining the top K images ranked by similarity and the corresponding drug labels.

5. The drug label recognition method based on image and text embedding based on deep learning according to claim 1, characterized in that: In S5, the indicator precision rate P@K is set to represent the proportion of drug labels with the same labels as the test drug labels, where P1@K represents the label recognition precision rate obtained by using only the image content similarity, and P2@K represents the label recognition precision rate obtained by using the similarity calculated by embedding the image text information, and the indicator calculation method of P@K=0.5*P1@K+0.5*P2@K is used.

6. The drug label recognition method based on image and text embedding based on deep learning according to claim 1, characterized in that: After the model training is completed in S5, an accurate judgment is made as to whether the drug shown in the input drug image needs to be investigated, and the classification label result of the input drug image is added to the reference data set after manual verification, so as to update the reference data set.

Citation Information

Patent Citations

  • Image retrieval method and image retrieval system

    CN103473327A

  • Drug identification system based on deep learning and implementation method thereof

    CN111382622A

  • Image similarity detection method and device, storage medium and electronic equipment

    CN111694978A

  • High-level semantic image retrieval method based on deep learning

    CN111782852A

  • Multimedia data processing method and device, electronic equipment and storage medium

    CN112085120A