A deep learning-based photographing medicine identification system and a use method thereof

Abstract

The application discloses a kind of based on deep learning's photograph identification medicine system and its use method, belong to medicine identification technical field.A kind of based on deep learning's photograph identification medicine system, including camera module, image pre-processing module, image segmentation module, target detection module and OCR identification module.The application discloses a kind of based on deep learning's photograph identification medicine system, easy to operate, efficient and accurate, and can be identified by photographing distinguishing the substance to be measured, and can obtain drug information and drug instruction, effectively prevent the old people from being cheated by false advertisement, mistake health products as medicine use.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of medicine identification, and particularly relates to a photographing medicine identification system based on deep learning and a use method thereof. BACKGROUND

[0002] In recent years, the market scale of the health product industry in China has been expanding year by year, and the market demand is booming. In this era of health product prevalence, some manufacturers over-promote, leading to the increasingly low ability of people to distinguish health products from medicines. Especially some middle-aged and elderly people often mistake health products for medicines through exaggerated introductions by sales personnel or channels such as advertisements.

[0003] In view of the above problems, a medicine helper-Dingxiangyuan has appeared on the market at present, but Dingxiangyuan only has a query function and cannot directly photograph and identify medicines, which is quite inconvenient to use, especially some imported medicines, which are difficult for the elderly to distinguish and query. Although Taobao and Jingdong have the function of photographing and identifying objects, the identification effect is poor, the medicine resource information is incomplete, and the health products and medicines cannot be accurately distinguished, only having a preliminary object identification function, and the medicine information and medicine instruction manual cannot be further obtained.

[0004] Therefore, a tool that can help users efficiently and accurately distinguish medicines is urgently needed. SUMMARY

[0005] Therefore, the purpose of the present application is to provide a photographing medicine identification system based on deep learning and a use method thereof. The present application aims to solve the problem that the existing medicine identification system cannot efficiently and accurately identify medicines and distinguish medicines, health products and medical devices.

[0006] To achieve the above purpose, the present application provides a photographing medicine identification system based on deep learning, comprising:

[0007] A camera module for photographing and recording medicines, health products and medical devices to be identified;

[0008] An image preprocessing module for preprocessing digital images photographed by the camera module;

[0009] An image segmentation module for foreground and background segmentation of the preprocessed digital images, so as to extract the foreground subject;

[0010] A target detection module for target detection of the foreground subject;

[0011] The target detection module is provided with a detection model, which can classify the foreground subject into medicines, health products, medical devices or others;

[0012] An OCR recognition module is configured to perform OCR recognition on the to-be-recognized object determined as a medicine or other, so as to obtain medicine-related information of the to-be-recognized object.

[0013] The OCR recognition module is associated with a database, and can further obtain medicine information and a medicine instruction manual.

[0014] OCR (Optical Character Recognition) refers to a process of analyzing and processing an image file after scanning text materials to obtain text and layout information.

[0015] Further, the preprocessing includes image graying, filter denoising, sharpening and correction.

[0016] Further, the detection model is constructed based on training data of a medicine feature database.

[0017] Further, the training data includes product images and feature images of medicines, health products and medical devices.

[0018] Further, the feature images include the following three types:

[0019] Medicine feature images: electronic supervision code, Class A OTC mark and Class B OTC mark;

[0020] Health product feature images: blue hat mark;

[0021] Medical device feature images: various instrument marks.

[0022] Further, the medicine-related information includes a medicine name, an approval number and a registration number.

[0023] The application also provides a use method of a photographing medicine identification system based on deep learning, including the following steps:

[0024] S1. Constructing a detection model through a medicine feature database;

[0025] S2. Capturing a to-be-recognized object by using a camera module;

[0026] S3. Preprocessing an image captured in step S2;

[0027] S4. Performing foreground-background segmentation on the digital image after preprocessing in step S3 to obtain a foreground subject;

[0028] S5. Performing target detection on the foreground subject by using a deep learning detection model, and classifying the foreground subject as a medicine, a health product, a medical device or other;

[0029] If the foreground subject of the object to be identified is determined to be medicine or other, then OCR recognition is performed to identify the medicine-related information in the OCR text;

[0030] If the foreground subject of the object to be identified is determined to be a health product or a medical device, then stop and derive the judgment result;

[0031] S6. Based on the drug-related information identified in step S5, query the yz_instruct table and export the final result. There are three possible outcomes:

[0032] A. The medicine was found, and the corresponding medicine instructions were available;

[0033] B. The drug was found, but there was no corresponding drug instruction manual;

[0034] C. If the query result is empty, it means that it is not a drug.

[0035] This invention provides a drug identification system based on OCR recognition technology, comprising:

[0036] The camera module is used to capture and record images of the medicines, health products, and medical devices to be identified.

[0037] The OCR recognition module is used to extract text information from images captured by the camera module;

[0038] The keyword matching module is used to match the text information extracted by the OCR recognition module, thereby classifying the images captured by the camera module into medicines, health products, medical devices, or others.

[0039] This invention also provides a method for using a photo-based drug identification system based on OCR recognition technology, comprising the following steps:

[0040] S1. Use the camera module to photograph the object to be identified;

[0041] S2. Use the OCR recognition module to extract the text information from the images captured by the camera module;

[0042] S3. Use the keyword matching module to match the text information extracted by the OCR recognition module, thereby classifying the object to be identified as: medicine, health products, medical devices or others;

[0043] If the output matching result of the object to be identified is medicine or other, then query the yz_instruct table and export the final result;

[0044] If the output matching result of the object to be identified is health products or medical devices, then stop and export the judgment result.

[0045] The beneficial effects of this invention are as follows:

[0046] 1. This invention provides a deep learning-based image-based drug identification system, which is more convenient to use. It can identify and distinguish the test object by taking a picture. If the test object is a drug, it can also obtain relevant information such as the drug instruction manual, which can effectively prevent the elderly from being deceived by false advertisements and mistakenly using health products as medicines.

[0047] 2. This invention provides a deep learning-based image-based drug identification system, which includes a target detection module and a detection model. The detection model is based on a drug feature database, covering a wider and more comprehensive range of drugs, and is more professional. This invention also includes an image preprocessing module and an image segmentation module, which extract clearer product images and feature images from the object to be identified. The combination of the two makes drug identification more efficient and accurate, and can distinguish between drugs, health products, and medical devices.

[0048] Other advantages, objectives, and features of the invention will be set forth in part in the description which follows, and in part will be apparent to those skilled in the art from the following examination, or may be learned from practice of the invention. The objectives and other advantages of the invention can be realized and obtained through the following description. Attached Figure Description

[0049] Figure 1 This is a flowchart of a deep learning-based image-based drug identification system according to the present invention;

[0050] Figure 2 This is a functional module diagram of a deep learning-based image-based drug identification system according to the present invention;

[0051] Figure 3 This is a flowchart of a drug identification system based on OCR recognition technology according to the present invention;

[0052] Figure 4 This is a functional module diagram of a drug identification system based on OCR recognition technology according to the present invention;

[0053] Figure 5 This is a packaging diagram of Fenbid in Example 1;

[0054] Figure 6 This is a packaging diagram of Chinese Family Complete Nutritional Supplement (Example 2);

[0055] Figure 7 This is a packaging diagram of the oral examination kit in Example 3. Detailed Implementation

[0056] To make the technical solutions, advantages, and objectives of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. All other embodiments obtained by those skilled in the art based on the described embodiments of the present invention without creative effort are within the protection scope of this application.

[0057] like Figure 1 and Figure 2 As shown, the present invention provides a deep learning-based image-based drug identification system, including a camera module, an image preprocessing module, an image segmentation module, an object detection module, and an OCR recognition module.

[0058] The camera module is used to capture and record images of the medicines, health products, and medical devices to be identified.

[0059] The image preprocessing module is used to preprocess the digital images captured by the camera module. The preprocessing includes image grayscale conversion, filtering and noise reduction, sharpening and correction.

[0060] The image segmentation module is used to segment the foreground and background of the preprocessed digital image, thereby extracting the foreground subject;

[0061] The target detection module is used to detect targets in the foreground subject. The target detection module can classify the foreground subject into medicines, health products, medical devices or others.

[0062] The target detection module includes a detection model built upon training data from a drug feature database. The training data comprises product images and feature images of drugs, health products, and medical devices. The feature images fall into three categories:

[0063] Drug characteristic images: electronic regulatory code, Class A OTC mark and Class B OTC mark;

[0064] Characteristic image of health supplements: blue hat logo;

[0065] Medical device feature images: logos of various instruments.

[0066] The OCR recognition module is used to perform OCR recognition on objects identified as "drugs" or "others" to obtain drug-related information, including drug name, approval number, and registration certificate number.

[0067] The OCR recognition module is associated with a database. In this embodiment, the database is the yz_instruct table, which can be used to further obtain drug information and drug instructions.

[0068] Example 1

[0069] This embodiment uses a deep learning-based image-based drug identification system to identify Fenbid. The identification process is as follows:

[0070] The Fenbid packaging box was photographed using a camera module, and the captured images are as follows: Figure 5 As shown, the captured image is then preprocessed; after preprocessing, foreground and background segmentation is performed on the digital image to obtain the foreground subject; subsequently, a deep learning detection model is used to detect the foreground subject, classifying Fenbid as a drug. Figure 5 The text contains phrases such as "National Drug Approval Number" and "Indications"; then, OCR recognition is performed to identify drug-related information in the OCR text; finally, the yz_instruct table is queried to export the drug information and the corresponding drug instructions.

[0071] Example 2

[0072] This embodiment uses a deep learning-based image-based medicine recognition system to identify Guoshi Complete Nutritional Supplements. The recognition process is as follows:

[0073] The packaging box of Guoshi Complete Nutritional Supplements was photographed using a camera module. The captured images are as follows: Figure 6 As shown, the captured images are then preprocessed; after preprocessing, foreground and background segmentation is performed on the digital images to obtain the foreground subject; subsequently, a deep learning detection model is used to detect the foreground subject, which allows Guoshi Complete Nutrition to be classified as a health supplement. Figure 6 The data contains terms such as "health food" or "health food product"; then the judgment result can be exported.

[0074] like Figure 3 and Figure 4 As shown, the present invention provides a drug identification system based on OCR recognition technology, comprising:

[0075] The camera module is used to capture and record images of the medicines, health products, and medical devices to be identified.

[0076] The OCR recognition module is used to extract text information from images captured by the camera module;

[0077] The keyword matching module is used to match the text information extracted by the OCR recognition module, thereby classifying the images captured by the camera module into medicines, health products, medical devices, or others.

[0078] Example 3

[0079] This embodiment uses a deep learning-based image-based drug recognition system to identify oral examination kits. The recognition process is as follows:

[0080] The packaging of the dental examination kit was photographed using a camera module, and the captured images are as follows: Figure 7 As shown, the OCR recognition module is then used to extract text information from the image captured by the camera module. Next, a keyword matching module is used to match the text information extracted by the OCR recognition module, thereby classifying the object to be identified. For example, the dental examination kit can be classified as a medical device. Figure 7 The data contains phrases such as "food and drug administration" and "medical equipment"; the results can then be exported.

[0081] When using the drug identification system of this invention, the identification is highly efficient and accurate. It can quickly distinguish the test substance, obtain drug information and drug instructions, and effectively prevent the elderly from being deceived by false advertisements and mistakenly using health products as medicines.

[0082] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the present invention, and all such modifications or substitutions should be covered within the protection scope of the present invention.

Claims

1. A deep learning-based image-based drug identification system, characterized in that, include: The camera module is used to capture and record images of the medicines, health products, and medical devices to be identified. The image preprocessing module is used to preprocess the digital images captured by the camera module; the preprocessing includes image grayscale conversion, filtering and noise reduction, sharpening and correction. The image segmentation module is used to segment the foreground and background of the preprocessed digital image, thereby extracting the foreground subject; The target detection module is used to detect targets in the foreground subject. The target detection module is equipped with a detection model that can classify foreground subjects as medicines, health products, medical devices, or others. The OCR recognition module is used to perform OCR recognition on objects identified as "medicine" or "other" to obtain relevant drug information of the objects to be identified. The OCR recognition module is associated with a database and can further obtain drug information and drug instructions; The detection model is constructed based on training data from a drug feature database; the training data includes product images and feature images of drugs, health products, and medical devices; the feature images include the following three categories: Drug characteristic images: electronic regulatory code, Class A OTC mark and Class B OTC mark; Characteristic image of health supplements: blue hat logo; Medical device feature images: logos of various instruments.

2. The deep learning-based image-based drug identification system according to claim 1, characterized in that: The drug-related information includes the drug name, approval number, and registration certificate number.

3. A method for using a deep learning-based image-based drug identification system, characterized in that, Includes the following steps: S1. Construct a detection model using a drug characteristic database; S2. Use the camera module to photograph the object to be identified; S3. Preprocess the images captured in step S2; S4. Perform foreground and background segmentation on the digital image preprocessed in step S3 to obtain the foreground subject; S5. Detect objects in the foreground subject using a deep learning detection model and classify the foreground subject as: medicine, health product, medical device, or others; If the foreground subject of the object to be identified is determined to be medicine or other, then OCR recognition is performed to identify the medicine-related information in the OCR text; If the foreground subject of the object to be identified is determined to be a health product or a medical device, then stop and derive the judgment result; S6. Based on the drug-related information identified in step S5, query the yz_instruct table and export the final result. There are three possible outcomes: A. The medicine was found, and the corresponding medicine instructions were available; B. The drug was found, but there was no corresponding drug instruction manual; C. If the query result is empty, it means that it is not a drug.

4. A drug identification system based on OCR recognition technology, characterized in that, include The camera module is used to capture and record images of the medicines, health products, and medical devices to be identified. The OCR recognition module is used to extract text information from images captured by the camera module; The keyword matching module is used to match the text information extracted by the OCR recognition module, thereby classifying the images captured by the camera module into medicines, health products, medical devices, or others.

5. A method for using a photo-based drug identification system based on OCR recognition technology, characterized in that, Includes the following steps: S1. Use the camera module to photograph the object to be identified; S2. Use the OCR recognition module to extract the text information from the images captured by the camera module; S3. Use the keyword matching module to match the text information extracted by the OCR recognition module, thereby classifying the object to be identified as: medicine, health products, medical devices or others; If the output matching result of the object to be identified is medicine or other, then query the yz_instruct table and export the final result; If the output matching result of the object to be identified is health products or medical devices, then stop and export the judgment result.

Images