A medicine bottle label character recognition and medicine name extraction method with semantic error correction capability

Abstract

This invention discloses a method for recognizing text on medicine bottle labels and extracting drug names with semantic error correction capabilities. The method includes: acquiring and preprocessing medicine bottle images to locate the label area; obtaining the original text sequence through optical character recognition (OCR); inputting the complete text string, retaining all information and without filtering, into a trained large language model; the large language model, based on its internally pre-trained pharmaceutical knowledge graph, simultaneously performing semantic error correction and extracting core drug information during the same reasoning process, correcting character-level errors generated by OCR and outputting key information such as generic names and specifications; and finally verifying the output results through a standard library. This invention embeds the pharmaceutical knowledge semantic parsing capability of the large language model into the post-processing stage of OCR, achieving end-to-end processing from recognition to error correction to extraction, significantly improving the recognition accuracy and error tolerance of pharmaceutical terminology in drug verification scenarios.

Description

Technical Field

[0001] This invention relates to the field of pharmaceutical finished product verification technology, and in particular to a method for recognizing text on medicine bottle labels and extracting drug names with semantic error correction capabilities. Background Technology

[0002] In the pharmaceutical product verification process, accurately identifying information such as the drug name on the medicine bottle label is a core step in ensuring medication safety and preventing errors. However, the medicine bottle labels themselves and the collection environment present numerous challenges, resulting in insufficient accuracy and stability of existing technologies.

[0003] In existing technologies, a common approach is to combine specific target detection with information matching. For example, Chinese patent CN117671705A discloses a method for identifying and extracting drug keywords. This method uses YOLOv7 to locate the liquid bag, PP-OCRv3 to recognize all text, and then performs similarity matching with medical order information using jieba word segmentation and a TF-IWF model. When faced with drugs with highly similar names such as "glucose injection," "sodium chloride injection," and "glucose sodium chloride injection," the technical solution is to adjust the word segmentation weights using a custom jieba dictionary and then perform keyword matching using a statistical model.

[0004] However, the aforementioned technical approach has a limitation: it lacks semantic error correction capabilities. When the optical character recognition (OCR) front-end produces character-level errors, such as misidentifying "penicillin sodium" as "amycin sodium," jieba segmentation will fail to correctly segment "amycin sodium," a word not found in the dictionary. Furthermore, the subsequent TF-IWF statistical model will be unable to associate it with the correct drug name. This is because the design philosophy of this approach addresses segmentation ambiguity and statistical matching issues, rather than correcting front-end recognition errors. Faced with OCR errors, this approach will lead to keyword extraction failure and prevent drug identification.

[0005] Furthermore, Chinese patent CN107958250A is a method for identifying vials, using SIFT feature matching to locate specific strings for single-point recognition. However, it cannot extract complete information such as the drug name, limiting its applicability. Chinese patent CN121305584A focuses on solving the problems of curved surfaces and multi-angle acquisition on vials using multiple cameras and YOLOv8. Its text recognition backend still uses general OCR and does not involve semantic error correction of the recognized content. Chinese patent CN118470722A uses both OCR and image recognition for verification, but the combination is loose. Optical character recognition and natural language processing run as two independent modules, failing to couple semantic error correction and information extraction in the same inference process.

[0006] In summary, existing technologies for addressing optical character recognition errors generally focus on improving the quality of front-end image processing, while lacking a technical solution for proactively correcting recognition results at the back end using medical semantic knowledge. Summary of the Invention

[0007] This invention addresses the problem that existing drug bottle label text recognition schemes lack semantic error correction capabilities when faced with optical character recognition errors. It proposes a drug bottle label text recognition and drug name extraction method with semantic error correction capabilities. This method embeds the pharmaceutical knowledge semantic parsing capabilities of a large language model into the post-processing stage of optical character recognition, achieving the coupling of semantic error correction and core drug information extraction in the same reasoning process. This significantly improves the recognition accuracy and error tolerance in pharmaceutical product verification scenarios.

[0008] To achieve the above objectives, the following technical solution is proposed: A method for recognizing text and extracting drug names from medicine bottle labels with semantic error correction capabilities includes the following steps: Acquire images of medicine bottles, preprocess the images, and locate the label area; Optical character recognition is performed on the label area to obtain the original text line information containing location information; The original text lines are sorted and concatenated according to a preset spatial order to form a complete text string that retains all text information. No text information is removed or filtered during this process. The complete text string is input into the trained large language model. The large language model performs semantic correction on character-level errors caused by optical character recognition errors in the text string based on the medical knowledge graph obtained through pre-training. In the same reasoning process, the core information of the drug is extracted from the corrected text. The core information of the drug includes at least the generic name, brand name and specifications of the drug. The extracted core drug information is verified for compliance, and the names of the drugs that pass the verification are output as the identification results.

[0009] By constructing a processing link that is completely different from the existing word segmentation and statistical path, the large language model is directly used as the backend of optical character recognition. This allows the model to actively correct front-end recognition errors by utilizing medical expertise obtained through pre-training. Therefore, it overcomes the inherent defects of statistical methods such as jieba word segmentation and TF-IWF in handling character-level errors. It adaptively completes the two coupled tasks of error correction and extraction in the same inference process, and outputs structured key information that can be directly used for review and decision-making.

[0010] As a preferred option, the step of inputting the complete text string into the trained large language model further includes: providing the original image information corresponding to the label region and the complete text string together as multimodal input to the large language model, so that the large language model can perform joint reasoning using text semantic features and image visual features simultaneously during semantic error correction and information extraction.

[0011] By incorporating the original image as a supplementary modality in addition to the text sequence, the large language model can achieve cross-validation of text and image information during the reasoning process. This not only serves error correction and extraction but also enables the discovery of contradictions between textual and visual features at the reasoning level, generating independent anomaly warning signals.

[0012] As a preferred approach, the joint reasoning process of the large language model includes: the large language model utilizes its internal cross-modal attention mechanism to map the image features extracted by the visual encoder and the semantic features extracted by the text encoder to the joint embedding space, calculates the alignment score between the two in this space, determines whether the image and text information are consistent based on the alignment score, and uses the consistency judgment result as one of the bases for generating the core drug information and subsequent confidence assessment; when the image and text information are determined to be inconsistent, regardless of whether the core drug information matches the standard drug name database, the current medicine bottle is marked as an abnormal state.

[0013] Through the above technical solutions, since the cross-modal feature alignment calculation is directly linked to the final decision output within the model, the text-image consistency judgment is no longer a post-processing verification step outside the model, but is deeply embedded within the reasoning process, becoming a security mechanism that can independently trigger anomaly marking.

[0014] As a preferred approach, the internal pharmaceutical knowledge graph upon which the large language model is based is obtained through pre-training on drug naming standards, pharmacopoeias, and drug instruction manuals. Semantic error correction includes using the combination rules of generic drug names, trade names, dosage forms, and specifications in the pharmaceutical knowledge graph to correct similar-looking character errors and contextual inconsistency errors generated by optical character recognition.

[0015] Through the above technical solutions, since the knowledge source is clearly defined as authoritative medical texts, and the objects of error correction are specifically defined as including similar-looking characters and contextual inconsistencies, the large language model does not perform general text proofreading, but uses structured domain knowledge rules to accurately correct professional errors in medical texts.

[0016] As a preferred option, the step of verifying the compliance of the extracted core drug information includes: matching the extracted generic name, brand name and specifications of the drug with a preset standard drug name database; if the match is successful, the verification is deemed to have passed.

[0017] Through the above technical solutions, since an additional round of comparison based on authoritative standard sources is added after semantic error correction and information extraction, even if the large language model has completed error correction, its results still need to be confirmed a second time by the pharmacopoeia standard, forming a double guarantee and effectively preventing the leakage of hallucination or biased information that may be generated by the model.

[0018] As a preferred option, after the step of verifying the compliance of the extracted core drug information, the following steps are also included: generating a confidence assessment value for the identification result; when the confidence assessment value is lower than a preset threshold, the current medicine bottle is marked as an abnormal state awaiting manual review.

[0019] By using the above technical solutions, a quantitative confidence index is generated for each identification result, and a hierarchical handling rule based on threshold triage is established. Therefore, the system can establish a clear boundary between automated decision-making and human intervention, ensuring that no high-risk cases are missed.

[0020] As a preferred option, the original text line information containing location information is obtained, specifically: the content of each text segment and its bounding box coordinate information are obtained.

[0021] By employing the above technical solutions, since both text content and spatial location information are recorded simultaneously during the optical character recognition stage, a data foundation is provided for subsequent text sequence reconstruction, ensuring the accuracy of the conversion from two-dimensional image information to one-dimensional text sequence.

[0022] As a preferred method, the original text line information is sorted and concatenated according to a preset spatial order. Specifically, based on the bounding box coordinates of each text segment, all text segments are sorted in a spatial order from top to bottom and from left to right, and then line breaks and character splicing are performed according to the coordinate relationship to form a complete text string.

[0023] By employing the above technical solutions, a row and column sorting method based on spatial coordinates is used instead of a semantic judgment method. This allows for the accurate restoration of the physical layout order of tags without relying on the understanding of text content, providing downstream large language models with a complete context that preserves the original reading order.

[0024] Preferably, the steps of preprocessing the medicine bottle image and locating the label area include: obtaining a clear image by adjusting the light source, angle and focal length through hardware; performing grayscale conversion, de-reflection, tilt correction and contrast enhancement on the clear image in sequence to obtain a preprocessed image; and locating and generating a label area mask image on the preprocessed image based on preset area recognition rules.

[0025] By employing the above technical solutions, a systematic front-end processing flow that includes hardware control and multi-step software algorithms is constructed. This systematically suppresses various interferences in the imaging process, providing low-noise input for subsequent optical character recognition. This is the prerequisite for ensuring the effectiveness of subsequent semantic error correction.

[0026] As a preferred option, the internal medical knowledge graph upon which the large language model is based is obtained through pre-training on drug naming standards, pharmacopoeias, and drug instruction manuals.

[0027] Through the above technical solutions, since the domain knowledge of the large language model comes from the systematic pre-training of authoritative medical texts, the model is embedded with the semantic judgment ability equivalent to that of a professional pharmacist. It can accurately identify drug naming norms and professionally deconstruct ambiguities, which is the source of its ability to achieve semantic error correction of medical texts.

[0028] As a preferred option, after verifying the compliance of the extracted core drug information and outputting the names of the verified drugs as the identification results, the system further includes: outputting the identification results to the finished product verification system in a standardized data format and storing the original drug bottle image in conjunction with it, thus forming a complete verification data closed loop.

[0029] Through the above technical solutions, since standardized data output specifications have been defined and a correlation storage mechanism between results and original evidence has been established, this invention can be seamlessly embedded into existing hospital information systems as an intelligent module, while providing a complete audit traceability link for each review decision, thus meeting the data integrity requirements of the pharmaceutical industry.

[0030] Therefore, the present invention has at least the following beneficial effects: 1. Breaking the dependence on optical character recognition errors, unlike existing technologies that rely on jieba word segmentation and TF-IWF statistical matching, this invention uses a large language model as the core of optical character recognition post-processing, and actively corrects front-end character-level errors by using medical knowledge obtained through pre-training, fundamentally solving the problem that existing statistical methods cannot handle character errors.

[0031] 2. Error correction and extraction are coupled within a single inference process. Semantic error correction and structured information extraction are no longer two independent and sequential steps, but are coupled and completed within the same inference process of the large language model. Error correction serves extraction, and extraction depends on the results of error correction. The two promote each other, improving the overall efficiency and accuracy of processing.

[0032] 3. A security detection mechanism for text-image consistency reasoning has been established. The text-image consistency judgment results generated during cross-modal joint reasoning are embedded in the model decision-making process. This not only serves error correction but also proactively detects security risks that cannot be detected by text recognition alone, such as mislabeling, thus expanding the security coverage dimension of drug review. Attached Figure Description

[0033] Figure 1 This is a flowchart of the method in Embodiment 1 of the present invention.

[0034] Figure 2 This is a flowchart of the method in Embodiment 2 of the present invention.

[0035] Figure 3 This is a flowchart of the method in Embodiment 3 of the present invention. Detailed Implementation Example

[0036] This embodiment describes the identification of an ambroxol hydrochloride injection vial under ideal lighting and orientation. This embodiment proposes a method for recognizing text on the vial label and extracting the drug name with semantic error correction capabilities, referencing... Figure 1 Specifically, it includes the following steps: Step 1, Image Acquisition and Preprocessing, and Label Region Localization: First, at the image acquisition station equipped with a ring-shaped LED light source and a 5-megapixel industrial camera, a labeled vial is conveyed to the designated position. The light source brightness, camera angle, and focal length are adjusted to ensure the vial is centered in the field of view and the image is clear. In this embodiment, the light source brightness value L is set to 70%, the camera exposure time T is set to 5 milliseconds, and the gain value G is set to 0 dB.

[0037] The original color image obtained from the capture is denoted as ImageRaw. First, ImageRaw undergoes grayscale conversion, transforming the three-channel color image into a single-channel grayscale image, ImageGray. The conversion formula is: ValueGray(x,y) = 0.299 * ValueR(x,y) + 0.587 * ValueG(x,y) + 0.114 *ValueB(x,y); Where: ValueR, ValueG, and ValueB are the brightness values ​​of the pixel in the red, green, and blue channels, respectively; ValueGray(x,y) is the grayscale value of the pixel (x,y) after conversion.

[0038] Next, the ImageGray image is de-glare processed. An image inpainting algorithm based on the fast-walking method is used to repair the locally bright areas caused by reflections from the curved surface of the bottle. The highlight threshold Thigh is set to 220, and the gradient threshold Tgrad is set to 20. Pixel regions with brightness values ​​greater than Thigh and neighborhood gradient changes less than Tgrad are repaired, generating the de-glare image ImageDeGlared.

[0039] Next, the ImageDeGlared image is tilted. Hough line detection is used to find the edge lines of the label, and their tilt angle Theta is calculated. In this embodiment, Theta is detected to be 1.5 degrees. A rotation matrix is ​​constructed based on Theta, and the image is subjected to an inverse rotation transformation to obtain the corrected image ImageCorrected.

[0040] Finally, contrast enhancement is performed on ImageCorrected. A contrast-limited adaptive histogram equalization algorithm is used to segment the image into M×N grids. In this embodiment, both M and N are 16, and the histogram cropping limit C for each grid is set to 2.0. Independent histogram equalization is performed on each grid, and bilinear interpolation is used to eliminate inter-grid boundary effects, resulting in the final preprocessed image ImageProcessed used for recognition.

[0041] In ImageProcessed, the label is located using a pre-defined region recognition rule. This rule is based on a normalized cross-correlation template matching algorithm, which pre-stores the outline template of the target vial. By calculating the normalized cross-correlation coefficient between the template and ImageProcessed, the position with the highest matching degree is found, thus determining the position of the vial in the image. Based on prior knowledge, the proportion Rh of the label region to the height of the vial is set to 60%, and the proportion Rb of the starting position from the bottom of the vial is set to 15%. The system automatically segments the label region RoiLabel accordingly and generates a binary mask image MaskLabel, where the pixel value of the label region is 1, and the pixel value of the remaining regions is 0.

[0042] Step 2, Optical Character Recognition and Acquisition of Original Text Sequence: The image data covered by the regions with a MaskLabel value of 1 is input into a pre-trained optical character recognition model. This model includes a DBNet-based text detection network and a CRNN-based text recognition network. The detection network outputs the bounding box coordinates of N text segments, and the recognition network outputs the specific character content of each segment.

[0043] In this embodiment, the model detects and identifies a total of 15 text fragments, which are stored in the form of a list of tuples. Each tuple has the structure (Texti, Bi), where Texti is the string of the i-th text fragment, and Bi is the bounding box of the fragment, represented by an array of coordinates of the four vertices, Bi = {(x1i,y1i), (x2i,y2i), (x3i,y3i), (x4i,y4i)}.

[0044] Part of the output is as follows: Text1 = “Ambroxol Hydrochloride Injection”, B1 = {(20, 30), (180, 30), (180, 60), (20,60)}; Text2 = “2ml:15mg”, B2 = {(190, 30), (280, 30), (280, 60), (190, 60)}; Text3 = “

Approval Number

[0045] Step 3: Clean and regularize the text sequence based on the full retention strategy: First, sorting and concatenation are performed. All text fragments `Texti` are traversed, and the top-left corner coordinates (xmini, ymini) of their bounding boxes `Bi` are extracted. All fragments are first sorted in ascending order by their ymini values, i.e., from top to bottom. If the absolute value of the difference in ymini between two fragments is less than a set line height threshold `Trow` (20 pixels in this embodiment), they are determined to belong to the same line. For multiple fragments determined to be in the same line, they are then sorted in ascending order by their xmini values, i.e., from left to right. After sorting, fragments in the same line are directly concatenated with an empty string, and texts in different lines are concatenated with a newline character, forming a complete text string `FullText` that retains all the original information. In this step, no text information is removed or filtered; even if the text fragments may contain recognition errors or interference, they are completely preserved, providing complete contextual information for subsequent large language models. This full-preservation strategy is a necessary prerequisite for large language models to effectively perform semantic error correction, because large language models rely on complete context to judge and correct local errors.

[0046] The FullText example in this embodiment is as follows: Ambroxol hydrochloride injection 2ml:15mg; [Approval Number] National Drug Approval Number H20031314; [Manufacturer] A pharmaceutical company; Step 4, semantic parsing and error correction of the large language model, coupling error correction and extraction in the same reasoning process: The original image data of the FullText and RoiLabel regions are used as multimodal input and submitted to the Qwen3.5 large language model. The model's processing result is expressed through a set of key-value pairs, for example: {Key1:Value1, Key2:Value2,...}. The internal logical processing steps of the model are as follows: Knowledge Activation: Based on the input texts such as "Ambroxol Hydrochloride Injection" and "2ml:15mg", the model activates relevant nodes in its internal medical knowledge graph obtained through pre-training on drug naming specifications, pharmacopoeias, and drug instruction texts. The knowledge base of the model contains structured professional knowledge such as "Ambroxol Hydrochloride" being the generic name of the drug, "Injection" being the dosage form, and the common representation format of the specification being "X volume:Y mass".

[0047] Text Error Correction: The model proofreads the FullText and verifies the correctness of fixed-format terms such as "National Medical Product Approval Number". In this embodiment, the original recognized text has no errors, but the model still performs the verification step to confirm that "Ambroxol Hydrochloride" conforms to the naming specification of the drug generic name, and "2ml:15mg" conforms to the representation format of the injection specification.

[0048] Information Extraction: Based on semantic roles and the medical knowledge graph, the model performs extraction operations during the same inference process. That is, the model executes a function F such that (GenericName, TradeName, Specification) = F(FullText). Since error correction and extraction are carried out in the same inference process, the model has utilized the semantic understanding of the text in the error correction stage during extraction, and these two subtasks are coupled rather than independent and serial. The extracted results are: GenericName = "Ambroxol Hydrochloride", TradeName = null value, Specification = "2ml:15mg".

[0049] Multi-modal Consistency Verification: The model uses its internal cross-modal attention mechanism to map the image features extracted by the visual encoder and the semantic features extracted by the text encoder to the joint embedding space, and calculates the alignment score Salign between them in this space. The alignment score Salign is compared with a preset matching threshold. When the alignment score meets the preset matching condition, it is determined that the text and image information is consistent. In this embodiment, the two are highly consistent, and the result of this consistency judgment will affect the subsequent confidence evaluation.

[0050] Structured Output: The model organizes and outputs the parsing results in a preset key-value pair data structure. The core information output includes the following keys and their corresponding values: GenericName: “Ambroxol Hydrochloride”; TradeName: null; Specification: “2ml:15mg”; Confidence: 0.98; Consistency: “matched”; Step 5, Drug Name Verification and Result Output: First, perform rule verification. Integrate the value of GenericName as "Ambroxol Hydrochloride" with the value of Specification as "2ml:15mg" to obtain the string to be verified Scheck = "Ambroxol Hydrochloride 2ml:15mg". Retrieve Scheck in the preset "Standard Drug Name Library" and perform an exact match. There is an entry in the library that is exactly the same as this string, and the match result is true.

[0051] Then, perform confidence evaluation. The system reads the value of Confidence as 0.98 and the value of Consistency as "matched". The preset confidence threshold Tconf is 0.9. Since 0.98 is greater than 0.9 and the consistency result is "matched", the system comprehensively determines that this result is a high-confidence result and passes it.

[0052] Finally, the system sends the recognition result "Ambroxol Hydrochloride 2ml:15mg" to the finished product review system in a standardized message format, and stamps the corresponding ImageProcessed with a timestamp and a result label, and stores it in the disk array for audit and traceability. The total process of this example takes 2.3 seconds. Embodiment

[0053] This embodiment describes the recognition of "Sodium Penicillin G for Injection" on an ampoule bottle with faint printing, tilt, and slight reflection interference. This embodiment proposes a method for recognizing the text on the drug bottle label and extracting the drug name with semantic error correction ability. Refer to Figure 2 , which specifically includes the following steps: Step 1, Image Acquisition, Preprocessing, and Label Region Location: When collecting the image, the ampoule bottle is placed with a 5-degree tilt, and there is faint printing on the "Sodium Penicillin G" part of the bottle label, resulting in the missing strokes of the lower half of the character "青". At the same time, there are dot-like reflections on the bottle body.

[0054] The preprocessing process is the same as that in Embodiment 1. Among them, in the tilt correction step, Theta is detected as 5.3 degrees and corrected accordingly. In the contrast enhancement step, the clipping limit value C of the algorithm is set to 2.5 to enhance the local contrast of the faint printing text to a greater extent, so that the blurred stroke contour is restored to a certain extent.

[0055] Step 2, Optical Character Recognition and Obtaining the Original Text Sequence: The optical character recognition model recognizes the preprocessed label region to obtain a text sequence. Due to faint printing and reflection interference, character-level errors occur during the recognition process: Text1 = "For Injection", B1 = {...}; Text2 = "Amitomycin Sodium", B2 = {...}; Here, the character '青' was misrecognized as '霭' due to the missing strokes in the lower part, which is a typical error of similar-shaped characters. Text3 = "0.48g", B3 = {...}; Here, the digit '4' was misrecognized as '.' due to adhesion to the reflection point, which is an error of context inconsistency because the '.' in "0.48g" appears in an unreasonable position. ... And so on.

[0056] Step 3, cleaning and regularization of the text sequence by retaining all error-containing texts: After sorting and splicing, the complete text string FullText obtained is: 0 Injection; Amitomycin Sodium; 0.48g; ...... In this step, although it is already known that there are recognition errors, the proposed solution in this application does not exclude or filter any text information, which is a key technical choice. If jieba word segmentation is performed at this step as in the prior art, "Amitomycin Sodium", which is not a word in the medical dictionary, will either be segmented into single characters "霭 / 霉 / 素 / 钠" or be marked as an unknown word. In either case, the subsequent TF-IWF statistical model will not be able to establish an association with the correct "Penicillin Sodium". The all-retaining strategy of this application ensures that the large language model can receive complete context information, including the dosage form prefix "0 Injection" and the common suffix "mycin sodium" for antibiotics. It is these context information that provides clues for the large language model to correct errors.

[0057] Step 4, semantic parsing and error correction by the large language model, character-level error correction driven by medical knowledge: Input the text string FullText and the corresponding image area into the large language model. The model processing process is as follows: Knowledge activation and context analysis: The model reads in fields such as "0 Injection", "mycin sodium", and "0.48g". Its internally pre-trained medical knowledge graph is activated. The model uses medical professional knowledge for reasoning: First, "0 Injection" is a common dosage form prefix, usually followed by the generic name of a drug; second, "mycin sodium" is a common suffix for the generic name of antibiotic drugs and appears frequently in combination with "0 Injection" in the medical context; third, "0.48g", as the drug specification value, is common in penicillin and cephalosporin drugs.

[0058] Semantic Error Correction: The model analyzes the incorrect text using the drug naming rules in the medical knowledge graph. The model recognizes that the character "霭" in "霭霉素钠" is extremely rare in common drug names, while "青霉素钠" is a standard common drug name, composed of "青霉素" plus "钠", which fully complies with the drug naming specifications. At the same time, the position of the "." in the specification "0.48g" is abnormal in the specification representation format. Combining with the fact that the common specifications of penicillin sodium in the knowledge graph include "0.48g", the model speculates that the "." should be a misrecognition of "4". Based on the comprehensive analysis of the above medical professional knowledge, the model uses the error correction function Fcorrect to correct "霭霉素钠" to "青霉素钠" and correct "0.48g" to "0.48g". That is, CorrectedText = Fcorrect(FullText, K), where K is the medical knowledge graph inside the model.

[0059] Information Extraction: In the same reasoning process after semantic error correction, the model performs extraction operations on the corrected text CorrectedText and obtains: GenericName = “Penicillin Sodium”; Specification = “0.48g”; Multi-modal Consistency Verification: The image features extracted by the model exhibit low-quality imaging characteristics such as low contrast and blurred edges. As a context signal, this visual feature reduces the model's direct confidence in the original optically recognized text, prompting the model to rely more on its internal medical knowledge graph for prediction during semantic error correction. The model calculates the alignment score Salign in the joint embedding space through a cross-modal attention mechanism. Since the corrected text "Penicillin Sodium" semantically points to the same overall appearance features of the ampoule in the image, the alignment score still meets the matching conditions. Finally, it is confirmed that the corrected text and the image are semantically consistent.

[0060] Structured Output: The model outputs a key-value pair structure: GenericName: “Penicillin Sodium”; TradeName: null; Specification: “0.48g”; Confidence: 0.91; Consistency: “matched”; It is worth noting that two significant character-level errors occurred during the optical character recognition stage, and these errors happened precisely on the core part of the drug name, "sodium penicillin." Although the model successfully corrected the errors based on the pharmaceutical knowledge graph, the model's self-evaluation of the error correction results reflected this uncertainty due to the inherent uncertainty in the original recognition signal. Therefore, a relatively low but still higher than the conventional threshold confidence value of 0.91 was given.

[0061] Step 5, Drug Name Verification and Result Output: The integrated string to be verified, Scheck = "Penicillin Sodium 0.48g", was successfully matched in the Standard Drug Name Database. The Confidence value was read as 0.91, which is greater than the threshold Tconf = 0.9, and the Consistency was "matched". The result was deemed successful and output to the verification system. The entire process demonstrates that even if obvious errors involving core drug names occur in the optical character recognition stage, the semantic error correction capability based on the medical knowledge graph of the large language model can still accurately recover and output the correct target information. Example

[0062] This embodiment simulates a serious anomaly: a medicine bottle is labeled "Sodium Chloride Injection," but it is incorrectly labeled "Ceftriaxone Sodium for Injection." This embodiment proposes a method for recognizing text on medicine bottle labels and extracting drug names with semantic error correction capabilities, referencing... Figure 3 Specifically, it includes the following steps: Steps 1 and 2, image acquisition, preprocessing, and optical character recognition: everything proceeded as normal. The optical character recognition model accurately identified the text on the incorrect label.

[0063] Step 3, Text Sequence Cleaning and Normalization: The resulting FullText string is: Ceftriaxone sodium for injection; 1.0g; ...... Step 4, Semantic parsing and error correction of the large language model: The model processes the multimodal input of FullText and a medicine bottle image as follows: Information extraction: The model performs an extraction operation on the text, and correctly extracts: GenericName = “Ceftriaxone Sodium”; TradeName = null; Specification = “1.0g”; Multimodal consistency verification: The model utilizes a cross-modal attention mechanism to calculate the alignment score Salign between text features and image features in the joint embedding space. The calculation results show that Salign is far below the preset matching threshold. Further analysis by the model reveals that the global visual features extracted from the image, such as the shape, size ratio, and cap color of the medicine bottle, do not match the typical appearance of large-capacity vials with blue caps for "Ceftriaxone Sodium" type drugs in its medical knowledge graph. Instead, they highly match the appearance of specific-sized ampoules with orange caps for "Sodium Chloride Injection" type drugs. Based on this significant contradiction between text and image features, the model determines that the text and image information are inconsistent.

[0064] In this application, multimodal consistency verification is performed in the joint embedding space within a large language model. The model examines text and images simultaneously during the same reasoning process, enabling it to detect contradictions that cannot be detected by relying on a single modality.

[0065] Structured output: The model ultimately outputs key-value pairs: GenericName: “Ceftriaxone Sodium”; TradeName: null; Specification: “1.0g”; Confidence: 0.99; Consistency: "unmatched"; Since the text itself is clearly and unambiguously recognized, the model has a high degree of confidence in its own text recognition capabilities, hence the confidence score of 0.99. However, the consistency flag is set to "unmatched," a field that independently reflects anomalous signals discovered based on cross-modal comparisons.

[0066] Step 5: Drug name verification and final processing; consistency anomalies trigger a security mechanism independent of confidence level. The first hurdle is standard library matching: the integration yields Scheck = "Ceftriaxone Sodium 1.0g", which is successfully matched in the "Standard Drug Name Database".

[0067] The second step is security assessment: The system first checks the value of the Consistency field. Built-in rules stipulate that if the Consistency value is "unmatched," regardless of the Confidence value or whether the standard library match is successful, it is immediately classified as a top-level security alert. This is because the anomaly of "label information not matching the physical appearance of the drug" means that the drug may have been incorrectly sorted, the label incorrectly affixed, or the drug itself may be counterfeit, posing a far greater risk than simply identifying uncertainty (low confidence).

[0068] Therefore, the system will not use the recognition results for automatic verification. Instead, the system will execute an anomaly fallback procedure: a visual and audible alarm will be generated on the control interface, indicating "Label information does not match the appearance of the medicine, suspected mislabeling, please handle manually," while simultaneously controlling a pneumatic robotic arm to move the medicine bottle from the conveyor belt into the anomaly handling area. The original image and key-value pairs of all processing data are packaged into a complete audit log and stored in the log system for subsequent detailed traceability.

[0069] This example demonstrates the cross-modal consistency verification function implemented in this application using a multimodal large language model. As an independent security detection dimension, it can proactively discover and intercept the security vulnerability of mislabeling, which is extremely difficult to detect in traditional verification systems. This capability stems from the ingenious technical concept of integrating semantic error correction, information extraction, and multimodal consistency verification into the same inference process, a feature not found in any single-modal or loosely coupled solution.

Claims

1. A method for recognizing text and extracting drug names from medicine bottle labels with semantic error correction capability, characterized in that, Includes the following steps: Acquire images of medicine bottles, preprocess the images, and locate the label area; Optical character recognition is performed on the label area to obtain the original text line information containing location information; The original text lines are sorted and concatenated according to a preset spatial order to form a complete text string that retains all text information. No text information is removed or filtered during this process. The complete text string is input into the trained large language model. The large language model performs semantic correction on character-level errors caused by optical character recognition errors in the text string based on the medical knowledge graph obtained through pre-training. In the same reasoning process, the core information of the drug is extracted from the corrected text. The core information of the drug includes at least the generic name, brand name and specifications of the drug. The extracted core drug information is verified for compliance, and the names of drugs that pass the verification are output as the identification result.

2. The method for recognizing text and extracting drug names from medicine bottle labels with semantic error correction capability according to claim 1, characterized in that, The step of inputting the complete text string into the trained large language model further includes: providing the original image information corresponding to the label region and the complete text string together as multimodal input to the large language model, so that the large language model can perform joint reasoning using text semantic features and image visual features simultaneously during semantic error correction and information extraction.

3. The method for recognizing text and extracting drug names from medicine bottle labels with semantic error correction capability according to claim 2, characterized in that, The joint reasoning process of the large language model includes: the large language model uses its internal cross-modal attention mechanism to map the image features extracted by the visual encoder and the semantic features extracted by the text encoder to the joint embedding space, calculates the alignment score between the two in this space, determines whether the image and text information are consistent through the alignment score, and uses the consistency judgment result as one of the bases for generating the core information of the drug and subsequent confidence assessment; when the image and text information are determined to be inconsistent, regardless of whether the core information of the drug matches the standard drug name database, the current medicine bottle is marked as an abnormal state.

4. The method for recognizing text and extracting drug names from medicine bottle labels with semantic error correction capability according to claim 1, characterized in that, The internal pharmaceutical knowledge graph upon which the large language model is based is obtained through pre-training on drug naming standards, pharmacopoeias, and drug instruction manuals. The semantic error correction includes using the combination rules of generic drug names, trade names, dosage forms, and specifications in the pharmaceutical knowledge graph to correct similar-looking character errors and contextual inconsistency errors generated by optical character recognition.

5. The method for recognizing text and extracting drug names from medicine bottle labels with semantic error correction capability according to claim 1, characterized in that, The step of verifying the compliance of the extracted core drug information includes: matching the extracted generic name, trade name and specifications of the drug with a preset standard drug name database; if the match is successful, the verification is deemed to have passed.

6. The method for recognizing text and extracting drug names from medicine bottle labels with semantic error correction capability according to claim 5, characterized in that, After the step of verifying the compliance of the extracted core drug information, the method further includes: generating a confidence assessment value for the identification result; when the confidence assessment value is lower than a preset threshold, marking the current medicine bottle as an abnormal state requiring manual review.

7. The method for recognizing text and extracting drug names from medicine bottle labels with semantic error correction capability according to claim 1, characterized in that, The process of obtaining the original text line information containing location information specifically involves obtaining the content of each text segment and its bounding box coordinates.

8. The method for recognizing text and extracting drug names from medicine bottle labels with semantic error correction capability according to claim 7, characterized in that, The step of sorting and splicing the original text line information according to a preset spatial order is as follows: based on the bounding box coordinates of each text segment, all text segments are sorted in a spatial order from top to bottom and from left to right, and then spliced ​​into lines and characters according to the coordinate relationship to form the complete text string.

9. The method for recognizing text and extracting drug names from medicine bottle labels with semantic error correction capability according to claim 1, characterized in that, The steps of preprocessing the image of the medicine bottle and locating the label area specifically include: obtaining a clear image by adjusting the light source, angle and focal length through hardware; sequentially performing grayscale conversion, de-reflection, tilt correction and contrast enhancement processing on the clear image to obtain a preprocessed image; and locating and generating a label area mask image on the preprocessed image based on preset area recognition rules.

10. The method for recognizing text and extracting drug names from medicine bottle labels with semantic error correction capability according to claim 1, characterized in that, After the step of verifying the extracted core information of the drug and outputting the name of the drug that has passed the verification as the identification result, the method further includes: outputting the identification result to the finished product verification system in a standardized data format and storing the original drug bottle image in association to form a complete verification data closed loop.

Images