A multi-scale OCR recognition and matching method for drug packaging facing complex background

Abstract

The application discloses a kind of complex background-oriented medicine packaging multi-scale OCR identification and matching method, it is related to medicine packaging identification field;Through hand-held machine RFID scanning unit, medicine packaging image is collected, and enhanced image is generated after multi-channel adaptive filtering preprocessing;YOLOv8 is used to locate character area, and improved CNN-Transformer hybrid network outputs preliminary character prediction sequence;BERT model corrects semantics, and verifies entity in combination with medicine field library;Scanning two-dimensional code calls medical order information comparison, and outputs matching result;Record artificial correction sample, and fine-tune model when accumulating to 50, realize that OCR identification accuracy continues to improve.The application improves the accuracy of medicine packaging OCR identification under complex background, reduces misidentification by semantic correction and field verification;Fast and accurate matching medical order, abnormal information highlights alarm;Model is continuously optimized through feedback, adapts to multiple medical scenarios, improves medicine management efficiency and drug safety.

Description

Technical Field

[0001] This invention relates to the field of pharmaceutical packaging identification technology, and in particular to a multi-scale OCR identification and matching method for pharmaceutical packaging with complex backgrounds. Background Technology

[0002] In the medical field, accurate identification of drug packaging information is crucial for ensuring medication safety and standardizing drug management. This is especially true in scenarios such as drug dispensing, inventory verification, and prescription execution, where the rapid and accurate acquisition of core information like drug name, concentration, and dosage is essential to provide reliable data for medical staff. Currently, Optical Character Recognition (OCR) technology, with its non-contact and efficient advantages, has become the mainstream method for extracting drug packaging information, widely used in handheld terminal scanning and recognition, and automated pharmacy management processes. However, drug packaging recognition in medical settings faces complex and diverse interference factors, making it difficult for existing OCR technologies to meet the demands for high-precision identification.

[0003] The diversity of pharmaceutical packaging and environmental interference directly limit the accuracy of traditional OCR technology. On the one hand, pharmaceutical packaging designs vary greatly, with some pharmaceuticals using irregular fonts, artistic printing, or adding textures and patterns to the packaging surface as decoration, creating complex backgrounds that reduce the contrast between characters and background. On the other hand, in actual operation, there are often problems such as uneven lighting, scanning angle deviations, packaging wear or stains, which further increase the difficulty of character recognition.

[0004] The technical architectural flaws of traditional OCR systems further limit their applicability in medical scenarios. Existing systems mostly rely on a single rule engine or basic neural network model, lacking targeted optimization for image quality. They process images only through simple threshold segmentation or edge detection, failing to effectively reduce background interference or repair blurred characters. At the same time, they do not incorporate professional medical knowledge and contextual semantic logic, resulting in insufficient verification capabilities for recognition results.

[0005] Furthermore, traditional systems lack continuous optimization capabilities and cannot dynamically adjust the model based on actual recognition errors. This results in poor recognition performance stability and weak generalization ability when facing new types of drug packaging or complex scenarios. This not only increases the workload of medical staff for manual verification but may also lead to medication safety risks due to misidentification of information. It is difficult to meet the stringent requirements of the medical field for high-precision and high-reliability character recognition. Summary of the Invention

[0006] The present invention proposes a multi-scale OCR recognition and matching method for pharmaceutical packaging with complex backgrounds to solve the problems mentioned in the prior art.

[0007] To achieve the above objectives, the present invention adopts the following technical solution: a multi-scale OCR recognition and matching method for pharmaceutical packaging with complex backgrounds, comprising:

[0008] Image acquisition and preprocessing steps: Acquire images of drug packaging using a handheld device, perform multi-channel adaptive filtering preprocessing, use Gaussian bilateral filtering for noise reduction, enhance contrast using the Ret Inex algorithm, use Otsu adaptive binarization, and correct tilted images using Hough transform.

[0009] Character region detection steps: Use the YOLOv8 network to locate the character regions on the drug packaging, input the enhanced image, and output candidate character boxes; filter the candidate boxes, perform non-maximum suppression processing, and output the coordinates and region types of the effective candidate boxes;

[0010] Multi-scale feature extraction and recognition steps: Construct an improved CNN-Transformer hybrid network. The CNN uses ResNet50 to extract features, and the Transformer fuses the features; input character box features, and output preliminary prediction sequences and confidence scores;

[0011] Semantic language model correction steps: Introduce a pre-trained BERT-based language model to construct a semantic correction window; calculate the reasonableness score of the input character sequence and correct semantically unreasonable sequences; output the corrected character sequence and semantic correction confidence.

[0012] Domain knowledge verification steps: Construct a standard thesaurus for the pharmaceutical domain, and use the BiLSTM-CRF model to extract character sequence entities; compare the entities with the domain thesaurus, mark non-matching entities as anomalies, and output a list of anomaly entities and standard matching suggestions;

[0013] Drug information and medical order matching steps: Obtain the unique identifier of the drug by scanning the code with a handheld device, retrieve the corresponding medical order information, compare the character sequence with the medical order information by field; output the matching results, and mark some matching fields as mismatched.

[0014] Active learning and feedback correction steps: Display matching results and anomaly markers via handheld device, record correction content, and label error types; fine-tune the model when the accumulation of error samples exceeds the threshold, and redeploy after fine-tuning.

[0015] Furthermore, it also includes:

[0016] Adaptive filter weight dynamic adjustment steps: Calculate the filter weight using the formula W=α·W1+β·W2, where W is the final filter weight, α is the denoising weight coefficient, W1 is the Gaussian bilateral filter weight, β is the contrast enhancement weight coefficient, and W2 is the Retinex algorithm weight.

[0017] Furthermore, it also includes:

[0018] Semantic correction result confidence evaluation steps: Calculate the comprehensive confidence of the corrected character sequence using the formula C=C1·γ+C2·(1-γ), where C is the comprehensive confidence, C1 is the initial recognition confidence output by the CNN-Transformer hybrid network, γ is the recognition confidence weight, C2 is the semantic reasonableness score output by the BERT semantic model, and 1-Y is the semantic score weight; when the comprehensive confidence is lower than the threshold, the system automatically triggers secondary semantic correction.

[0019] Furthermore, in the multi-scale feature extraction and recognition steps, the improved CNN-Transformer hybrid network uses an attention weighting mechanism to fuse features; the CNN outputs character details, local structure, and global contour feature maps, assigns attention weights, and fuses features through matrix multiplication; the Transformer calculates the fused feature sequence through self-attention and outputs the character probability distribution; the character with the highest probability is selected to form the preliminary prediction sequence, and the prediction confidence of a single character is the corresponding probability value.

[0020] Furthermore, in the domain knowledge verification step, through formulas Calculate entity matching similarity, where S is the matching similarity and N is the number of similarities. correct To ensure the correct number of entities are matched with the domain thesaurus, N total The total number of entities to be extracted; when the similarity is below the threshold, the system outputs candidate entities for selection.

[0021] Furthermore, in the active learning and feedback correction steps, an error type weighting strategy is used to screen samples; error types are classified, weighted error scores of error samples are calculated, and samples with weighted error scores exceeding the threshold are added to the training dataset; when fine-tuning the model, an incremental training method is used, and the proportion of new samples in the training set is not less than 30%.

[0022] Furthermore, in the process of matching drug information with medical orders, a tiered processing mechanism is adopted for some matching scenarios. When the drug name field does not match, it is judged as a level 1 mismatch, and the system automatically retrieves the drug alias database for a second match. The second match is updated to a complete match. When the concentration and dosage fields do not match, it is judged as a level 2 mismatch. The system calculates the concentration and dosage deviation rate. If the deviation rate is lower than the threshold, the user is notified. If the deviation rate exceeds the threshold, it is marked as abnormal. When the drug name does not match the concentration and dosage, it is judged as a level 3 mismatch, marked as abnormal, and subsequent operations are prohibited.

[0023] Furthermore, it also includes:

[0024] Visualization and alarm steps for matching results: The handheld display unit uses a segmented display. Completely matched results are displayed in green with the words "Match Successful". Partially matched results are displayed in yellow with the words "Partial Match" and the non-matching fields are highlighted. Non-matching results are displayed in red with the words "Match Failed" and an alarm prompt box pops up. The alarm prompt box provides vibration feedback and records the matching result log.

[0025] Furthermore, in the active learning and feedback correction steps, a dynamic learning rate strategy is adopted to fine-tune the model; an initial learning rate is set for fine-tuning, and the model recognition accuracy is calculated iteratively; when the accuracy improvement exceeds the threshold, the current learning rate is maintained; when the accuracy improvement is lower than the threshold, the learning rate is reduced to 0.5 times the original value and fine-tuning is stopped.

[0026] Furthermore, in the image acquisition and preprocessing steps, supplementary preprocessing operations are used for special complex backgrounds; polarization filtering algorithm is used for reflective areas, image restoration algorithm is used for stained areas, and edge enhancement algorithm is used for blurred character printing areas. After supplementary preprocessing, multi-channel adaptive filtering is re-executed.

[0027] Compared with existing technologies, the beneficial effects of this invention are:

[0028] In the image preprocessing stage, multi-channel adaptive filtering technology is adopted, combined with operations such as noise reduction, contrast enhancement, and tilt correction, to effectively handle problems such as reflection, blurring, and tilt, weaken complex background interference, make character areas clearer, lay a high-quality image foundation for subsequent recognition, and reduce recognition deviations caused by poor image quality from the source.

[0029] In the recognition and correction stage, multi-scale character features are extracted through an improved CNN-Transformer hybrid network, taking into account both character details and global structural information. It accurately captures and discriminates features when dealing with irregular fonts and deformed characters, significantly improving the initial recognition effect. At the same time, a pre-trained language model and domain knowledge verification mechanism are introduced to double-verify the recognition results from both semantic logic and professional rules, correcting problems such as "misrecognition of similar-looking characters" and "semantic inconsistencies," and identifying domain deviations such as "incorrect units" and "abnormal drug names," further reducing the false recognition rate and the missed recognition rate, and ensuring that the recognition results meet the professional standards of medical scenarios.

[0030] In terms of system usability and sustainability, the active learning and feedback correction mechanism can record human error correction information and dynamically optimize model parameters, so that the system's recognition performance can be continuously improved as the usage scenarios expand and adapt to the constantly updated types of drug packaging. At the same time, through linkage and matching with the pharmacy system and the medical order system, the recognition results can be accurately compared with drug information and medical order content, and the matching results can be displayed in a visual way and abnormal information can be highlighted, so that medical staff can quickly detect problems and reduce the workload of manual verification.

[0031] In addition, the entire recognition process is efficient and convenient, adaptable to mobile operation scenarios of handheld terminals, and can meet the real-time recognition needs of multiple scenarios such as pharmacy inventory and clinical dispensing, improve drug management efficiency, and avoid medication safety risks caused by misidentification of information from a technical perspective, providing a reliable solution for high-precision character recognition in the medical field. Attached Figure Description

[0032] Figure 1 This is a schematic block diagram of a multi-scale OCR recognition and matching method for pharmaceutical packaging with complex backgrounds proposed in this invention.

[0033] Figure 2 A diagram showing the comparison of OCR recognition accuracy of pharmaceutical packaging under different backgrounds;

[0034] Figure 3 A diagram showing the time consumption of each step in OCR recognition;

[0035] Figure 4 This is a diagram showing the comparison of OCR recognition performance before and after model fine-tuning;

[0036] Figure 5 This is a diagram showing the comparison between drug information and medical orders. Detailed Implementation

[0037] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0038] In the description of this invention, it should be understood that the terms "center," "longitudinal," "lateral," "length," "width," "thickness," "upper," "lower," "front," "rear," "left," "right," "vertical," "horizontal," "top," "bottom," "inner," "outer," "clockwise," and "counterclockwise," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are only for the convenience of describing this invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on this invention.

[0039] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, features defined with "first" and "second" may explicitly or implicitly include one or more of the stated features. In the description of this invention, "a plurality of" means two or more, unless otherwise explicitly specified. Furthermore, the terms "installed," "connected," and "linked" should be interpreted broadly; for example, they may refer to a fixed connection, a detachable connection, or an integral connection; they may refer to a mechanical connection or an electrical connection; they may refer to a direct connection or an indirect connection through an intermediate medium; and they may refer to the internal connection of two components. Those skilled in the art can understand the specific meaning of the above terms in this invention based on the specific circumstances. The invention will now be described in further detail with reference to the accompanying drawings.

[0040] Reference Figures 1 to 5 A multi-scale OCR recognition and matching method for pharmaceutical packaging with complex backgrounds, comprising:

[0041] Image acquisition and preprocessing steps: Relying on the RFID scanning unit integrated in the handheld device, image acquisition is triggered synchronously. The resolution of the drug packaging image is set to 1280×960 pixels, and the acquisition frequency is stabilized at 1 frame / second to match the manual operation rhythm of drug circulation, ensuring acquisition efficiency and data integrity. The preprocessing stage employs a multi-channel adaptive filtering strategy for layer-by-layer optimization. Gaussian bilateral filtering is used for noise reduction, with a filter kernel, spatial domain standard deviation, and value domain standard deviation set. The spatial-value domain joint weight calculation utilizes bilateral filtering to suppress noise caused by reflection and uneven ink distribution on the pharmaceutical packaging surface, while preserving key features such as barcode edges and anti-counterfeiting patterns. The Retinex algorithm is introduced to enhance contrast, with a scale parameter of 800. By simulating the color constancy of human vision, the algorithm decomposes the image's illumination and reflection components, adaptively enhancing low-contrast areas to clarify details in dark areas and prevent overexposure in bright areas. Otsu adaptive binarization is then initiated, automatically analyzing the image's grayscale histogram and dynamically determining the threshold to convert the color image into a black-and-white binary image, highlighting detection targets such as barcodes and text. Hough transform is used to correct image tilt, constructing a polar coordinate accumulator with an angle correction range of -15° to +15°. Based on the straight-line features extracted from edge detection, the tilt angle is calculated and rotated for correction with an accuracy of ±0.5°, ensuring that the input images for subsequent OCR and barcode recognition algorithms are upright and distortion-free. Through the entire process, enhanced images with no noise interference, high contrast, and geometric correction are output, laying a solid foundation for accurate identification and information verification of drug packaging.

[0042] Character region detection steps: A lightweight YOLOv8 object detection network is used. The network structure is customized and optimized for the small target and multi-class characteristics of character regions on pharmaceutical packaging. Depthwise separable convolutions are introduced in the backbone part to reduce computation while enhancing the extraction of fine-grained features such as character edges and strokes. The neck layer uses a fusion structure of feature pyramid network and path aggregation network to enhance the fusion effect of multi-scale character features, adapting to the detection of character regions with different font sizes and layouts on pharmaceutical packaging. The network input is an enhanced image preprocessed with multi-channel adaptive filtering, providing a high-quality data foundation for character detection. In the network output stage, a candidate box confidence threshold of ≥0.7 is set. Based on the classification confidence obtained through training with the cross-entropy loss function, low-confidence candidate boxes are filtered out, retaining effective candidate boxes with high probability of character region recognition. To address the potential dense arrangement and overlap of character regions on pharmaceutical packaging, non-maximum suppression is performed on candidate boxes with an overlap greater than 50%. By calculating the intersection-union ratio (IUU) of candidate boxes, the candidate box with the highest score and most accurate position is retained, suppressing duplicate detection results. Output the coordinates of each valid candidate box, and restore the position of the character region in the original drug packaging image through coordinate mapping; based on the character region category labeled during network training, output the corresponding character region type, providing accurate region positioning and category guidance for subsequent character recognition and information verification, ensuring the accuracy and practicality of drug packaging character detection, and supporting the whole process traceability and quality control of drugs.

[0043] Multi-scale feature extraction and recognition steps: An improved CNN-Transformer hybrid network is constructed. The CNN part uses ResNet50 as the backbone network, and the feature extraction process is deeply optimized. Sixteen convolutional layers are deployed sequentially, employing an alternating design of 1×1 and 3×3 convolutional kernels. The 1×1 convolutional kernels compress and upscale the channel dimensions, reducing computation while reshaping feature representation. The 3×3 convolutional kernels focus on spatial feature extraction, using a dynamic stride of 1-2 to gradually reduce the feature map size at different levels, extracting low, medium, and high-dimensional features, ultimately outputting a 512-dimensional feature map that comprehensively covers the multi-scale visual information of pharmaceutical packaging characters. The Transformer part features a 6-layer encoder structure, with 8 attention heads per layer and a hidden layer dimension of 512. A multi-head self-attention mechanism is used to perform multi-scale feature fusion on the CNN output feature map. By adjusting the attention weights, the network's receptive field flexibly covers a range from 1×1 to 16×16 pixels, adapting to the recognition needs of pharmaceutical packaging characters, from individual numbers and letters to continuous text paragraphs. During the recognition task, the features corresponding to the candidate character boxes output from the character region detection step are extracted by RoIAlign and then input into the hybrid network. The network is trained end-to-end and optimized based on cross-entropy loss and CTC loss to output a preliminary character prediction sequence, such as "Amoxicillin Capsules 0.25g 24 Capsules" which fully presents the drug information. The predicted sequence length is strictly constrained to ≤100 characters to adapt to the text capacity of drug packaging labels; the single character prediction confidence is ≥0.6. The probability value of the character in the Softmax output is calculated to ensure the reliability of recognition, laying a solid foundation for subsequent accurate OCR verification and drug information parsing, and helping the efficient operation of drug traceability and quality control processes.

[0044] Semantic language model correction steps: A pre-trained BERT-base language model is introduced, with the training dataset deeply covering 100,000 real pharmaceutical packaging texts. Based on the characteristics of short texts and a high proportion of specialized vocabulary on pharmaceutical packaging characters, a contextual semantic correction window is constructed, with a window size of 5 characters. A sliding window mechanism is used to capture local semantic relationships within the character sequence. The character sequence output from the multi-scale feature extraction and recognition steps is input into the model. BERT-base utilizes the multi-head attention mechanism of the Transformer encoder to semantically encode the character sequence and calculate the semantic reasonableness score of the sequence's context. The score is based on the semantic understanding ability obtained from training the masked language model and the next sentence prediction task, mapped to the 0-1 interval using the Softmax function, and a threshold of ≥0.8 is set for semantic reasonableness. For semantically unreasonable sequences, the probability distribution output by the model is called, and the semantic vulnerability location is analyzed based on the attention weights. Combined with a pharmaceutical specialized vocabulary database, erroneous characters are corrected to semantically reasonable expressions. During the correction process, the correction strategy is optimized through backpropagation using cross-entropy loss to ensure the semantic accuracy of the output corrected character sequence. Simultaneously, the semantic correction confidence score is calculated based on the probability change of characters in the language model before and after correction. The confidence score after correction is required to be ≥0.85 to ensure the reliability of the correction results and provide high-precision text data support for drug packaging information collection, traceability and supervision.

[0045] Domain knowledge verification steps: A standard thesaurus for the pharmaceutical domain is constructed, based on authoritative sources such as the *Chinese Pharmacopoeia* and the *National Essential Medicines List*. It integrates 3000 commonly used drug names, 200 concentration units, and 150 dosage expressions. The thesaurus is dynamically updated based on drug regulatory data and new product specifications added by pharmaceutical companies, with automatic monthly updates to ensure its timeliness. A BiLSTM-CRF named entity recognition model is used for entity extraction. The model is trained and optimized using pharmaceutical packaging text corpus. The BiLSTM layer uses forward and backward recurrent neural networks to capture bidirectional semantic dependencies in character sequences; the CRF layer introduces a transition probability matrix to constrain the rationality of entity labeling, extracting three core entities: drug name, concentration, and dosage, and outputting entity boundaries and category labels. The extracted entities are compared and verified against the domain thesaurus from multiple dimensions. Based on string matching and semantic similarity calculation, mismatched entities are identified. When a concentration expression of "0.25kg" is encountered, the thesaurus verification module triggers a unit rationality check, marks it as an abnormal entity, and simultaneously calls the thesaurus association rules to output a standard matching suggestion, "It is recommended to correct it to 0.25g". This forms a complete closed loop from entity extraction to abnormal verification and standard correction, providing accurate domain knowledge support for drug packaging information entry and quality traceability.

[0046] Drug information and prescription matching steps: Using the QR code scanning unit integrated into the handheld device, based on a CMOS image sensor and QR code decoding algorithm, the QR code on the drug bottle label is scanned. The decoding module follows the ISO / IEC 18004 standard, parsing the QR code data through processes such as grayscale conversion, binarization, contour extraction, and position detection image recognition to obtain the unique drug identifier. Relying on the hospital information system interface, data from the prescription system is retrieved based on the unique drug identifier. The system retrieves the corresponding prescription information from the prescription database through the HL7 medical information interaction protocol, including drug name, concentration, dosage, and frequency of administration. During field comparison, for the character sequence after domain knowledge verification, a combined strategy of precise matching and intelligent verification is adopted for the three core fields: drug name, concentration, and dosage. The drug name is matched using string equality; the concentration field incorporates unit conversion logic and verifies the rationality of the value; the dosage field distinguishes between single dose and frequency association. After the comparison is completed, the matching results are output as complete match, partial match, and no match. When partial matching occurs, the system automatically marks the mismatched fields and provides conflict prompts based on the drug instructions and medical order rules, realizing intelligent medical order verification in the drug dispensing process and ensuring medication safety throughout the entire process from barcode recognition to information verification.

[0047] Active learning and feedback correction steps: Utilizing a handheld display unit, the matching results of drug information and medical orders are presented in a visual interface. Interactive methods such as highlighting abnormal fields in red and pop-up correction prompts guide users to confirm or manually correct errors. User correction actions are recorded in real time, not only retaining numerical changes but also labeling error types. Based on a predefined error classification system, error samples are accurately categorized, providing clear direction for model optimization. When the accumulated error samples reach ≥50, the model iteration process is initiated. The labeled error samples are divided into training and validation sets at an 8:2 ratio and integrated into the original model training dataset. For the CNN-Transformer hybrid network, the backbone network layers were frozen during the fine-tuning phase, and gradient updates were applied only to the output layer and the Transformer encoder layer. The AdamW optimizer was used, with a learning rate of 1e-5 and 10 iterations, focusing on optimizing the accuracy of character feature extraction and sequence prediction. For the BERT semantic model, domain-adaptive fine-tuning was performed based on error samples from the pharmaceutical domain, building upon the pre-trained weights. The semantic understanding of pharmaceutical terminology was enhanced through the Masked Language Modeling task. A gradient accumulation strategy was used during the fine-tuning process to balance the stability and convergence of small-sample training. After fine-tuning, the optimized model underwent full-process testing and validation. A / B testing was used to compare the accuracy, recall, and F1 score of the new and old models in the pharmaceutical packaging recognition task. After successful validation, the new model was redeployed to a handheld terminal, and lightweight inference acceleration was achieved through an edge computing framework to ensure recognition efficiency.

[0048] This invention also includes the following steps:

[0049] The adaptive filtering weight dynamic adjustment steps are as follows: The filtering weight is calculated using the formula W = α·W1 + β·W2, where W is the final filtering weight (value 0-1), α is the denoising weight coefficient (value 0.3-0.7, 0.7 when image noise intensity > 30dB, and 0.3 when ≤ 30dB), W1 is the Gaussian bilateral filtering weight (fixed at 0.6), β is the contrast enhancement weight coefficient (value 0.3-0.7, summed with α to 1), and W2 is the Retinex algorithm weight (fixed at 0.4). This weight adjustment prioritizes denoising in high-noise images and contrast enhancement in low-contrast images, ensuring optimal enhancement for images of different quality. For example, when a pharmaceutical packaging image has a noise intensity of 35dB due to reflection, α = 0.7, β = 0.3, and W = 0.7 × 0.6 + 0.3 × 0.4 = 0.54, the filtering process focuses more on denoising and reducing reflection interference.

[0050] This invention also includes the following steps:

[0051] Semantic correction result confidence evaluation steps: Calculate the overall confidence of the corrected character sequence using the formula C=C1·γ+C2·(1-γ), where C is the overall confidence (value 0-1, ≥0.8 is considered a reliable result), C1 is the initial recognition confidence output by the CNN-Transformer hybrid network (value 0-1), γ is the recognition confidence weight (fixed at 0.6), C2 is the semantic reasonableness score output by the BERT semantic model (value 0-1), and 1-γ is the semantic score weight (fixed at 0.4). When the overall confidence level is <0.8, the system automatically triggers secondary semantic correction (expanding the context window to 8 characters). For example, the initial identification sequence "amoxicillin capsules" has a C1 of 0.75, and the BERT model determines the semantic rationality with a C2 of 0.6. The calculated value is <0.8. During the secondary correction, "amoxicillin" is corrected to "amoxicillin". After correction, C1 = 0.88, C2 = 0.92, and C = 0.88 × 0.6 + 0.92 × 0.4 = 0.896 ≥ 0.8, making the correction result reliable.

[0052] In this invention, the feature fusion of the improved CNN-Transformer hybrid network in the multi-scale feature extraction and recognition steps adopts an attention weighting mechanism. The CNN backbone network outputs three scale feature maps (16×16, 32×32, and 64×64 pixels), corresponding to character detail features, local structure features, and global contour features, respectively. Attention weights are assigned to each scale feature map (0.4 for detail features, 0.3 for local structure features, and 0.3 for global contour features). The multi-scale features are fused into a 512×1×L dimension feature sequence (L is the length of the character sequence) through matrix multiplication. The Transformer encoder performs self-attention calculation on the fused feature sequence (attention masking excludes invalid background regions), and outputs the probability distribution of each character (the dimension is the size of the character set, including 5000 commonly used Chinese characters such as numbers, letters, and numbers). The character with the highest probability is selected to form the initial prediction sequence, and the prediction confidence of a single character is the corresponding probability value. This fully utilizes the multi-scale features and improves the recognition accuracy in complex backgrounds (such as blurred character edges and background pattern interference).

[0053] In this invention, the domain knowledge verification step is performed using a formula. Calculate entity matching similarity, where S is the matching similarity (value from 0-100%, ≥90% is considered a successful match), and N... correct To determine the number of entities that correctly match the domain thesaurus (e.g., the number of correct matches in drug name, concentration, and dosage), N total This represents the total number of entities extracted (fixed for name, concentration, and dosage). When the similarity is <90%, the system outputs the three candidate entities with the highest similarity in the standard dictionary for the user to choose from. For example, if the extracted entity is "Amoxicillin Capsules 0.25kg 24 capsules", and the dictionary matches "Amoxicillin Capsules" (correct match, corresponding to the alias "Amoxicillin Capsules"), "0.25kg" (not a match, the standard is "g"), and "24 capsules" (correct match), then N... correct =2, N total =3, S = (2 / 3) × 100% ≈ 66.7% < 90%, the system recommends candidate concentrations "0.25g", "0.5g" and "0.125g" to help users make quick corrections.

[0054] In this invention, the selection of error samples in the active learning and feedback correction steps adopts an error type weighting strategy. Error types are categorized into semantic misidentification (weight 0.4), domain mismatch (weight 0.3), character ambiguity misidentification (weight 0.2), and other errors (weight 0.1). A weighted error score is calculated for each error sample (score = error type weight × error impact level, impact level: 1 for single-field errors, 2 for multi-field errors). Samples with a weighted error score ≥ 0.5 are added to the training dataset, prioritizing high-value error samples for model fine-tuning. Incremental training is used during fine-tuning, with new samples accounting for a certain percentage of the training set. ≤30% to avoid the model forgetting existing knowledge; for example, a sample misidentifies "0.1g" as "0.7g" due to character ambiguity (error type: character ambiguity misidentification, single field error), the weighted error score = 0.2×1 = 0.2 < 0.5, and is not added to the training set for the time being; another sample misidentifies "cefixime" as "cefixime filth" due to semantic misidentification (multi-field error, affecting the matching of drug name and medical order), the score = 0.4×2 = 0.8 ≥ 0.5, and is added to the training set to improve the model's ability to recognize easily confused characters.

[0055] In this invention, the drug information and prescription matching step also includes a tiered processing mechanism for some matching scenarios. When only the drug name field does not match, it is judged as a first-level mismatch. The system automatically retrieves the drug alias database (containing 500 commonly used drug aliases, such as "Amoxicillin" corresponding to "Amoxicillin") for a second match. If the second match is successful, it is updated to a complete match. When only the concentration / dosage field does not match, it is judged as a second-level mismatch. The system calculates the concentration / dosage deviation rate (deviation rate = |identification value - prescription value| / prescription value × 100%). If the deviation rate is ≤10%, the user is prompted to confirm whether it is a labeling error; if it is >10%, the user is prompted to confirm whether it is a labeling error. When the percentage is not specified, it is marked as abnormal; when the drug name and concentration / dosage do not match, it is judged as a level three mismatch, directly marked as abnormal and subsequent operations are prohibited; for example, if the drug name is "Amoxicillin Capsules" and the concentration is "0.25g", and the prescription name is "Amoxicillin Capsules" and the concentration is "0.2g", in the case of a level one mismatch, a second match is made through the alias library to match the names. The concentration deviation rate = (0.25-0.2) / 0.2×100% = 25% > 10%, so it is marked as a level two mismatch, and the user is prompted to check the concentration.

[0056] This invention also includes the following steps:

[0057] Visualization and alarm steps for matching results: The handheld display unit uses a segmented display (upper area: drug packaging image and candidate character box markers, middle area: recognized and corrected character sequence, lower area: matching results); a complete match result is displayed in green as "Match Successful", a partial match result is displayed in yellow as "Partial Match" and the mismatched field is highlighted (highlight color is orange, font is bold), and a mismatch result is displayed in red as "Match Failed" and an alarm prompt box pops up (prompt content: "The following field does not match: [Drug Concentration], please check again"); the alarm prompt box is accompanied by vibration feedback (vibration frequency 2 times / second, lasting 1 second) to help users detect abnormalities in a timely manner; at the same time, a matching result log (including timestamp, drug code, recognition result, and matching status) is recorded, and the log is stored for ≥3 months for subsequent traceability.

[0058] In this invention, the active learning and feedback correction steps for model fine-tuning employ a dynamic learning rate strategy. The initial learning rate is set to 1e-5. Every two iterations, the model's recognition accuracy on the validation set (containing 1000 drug packaging samples) is calculated. When the accuracy improvement is ≥0.5%, the current learning rate is maintained. When the accuracy improvement is <0.5% but ≥0.2%, the learning rate is reduced to 0.8 times the original value. When the accuracy improvement is <0.2%, the learning rate is reduced to 0.5 times the original value, and fine-tuning stops (maximum 10 iterations). For example, in the first round of fine-tuning, the accuracy improves by 1.2%, maintaining 1e-5; in the third round, it improves by 0.4%, and the learning rate is adjusted to 8e-6; in the fifth round, it improves by 0.1%, and the learning rate is adjusted to 4e-6, and fine-tuning stops. This avoids overtraining leading to a decrease in the model's generalization ability, ensuring that the fine-tuned model can still stably recognize new scenarios.

[0059] In this invention, the image acquisition and preprocessing steps include supplementary preprocessing operations for special complex backgrounds (such as reflective or stained pharmaceutical packaging). For reflective areas, a polarization filtering algorithm is used (polarization angle adjustment range 0°–90°, step size 15°, selecting the angle with the least reflectivity); for stained areas, an image restoration algorithm is used (based on context pixel prediction, restoration window size 5×5, iteration count 3); for blurred character printing areas, an edge enhancement algorithm is used (Laplacian operator convolution kernel size 3×3, enhancement coefficient 1.2). After supplementary preprocessing, multi-channel adaptive filtering is re-executed to ensure clear identification of character areas even in special backgrounds. For example, when the character area on pharmaceutical packaging is too bright due to plastic film reflection, polarization filtering reduces the reflectivity from 200 cd / m². 2 Reduced to 80 cd / m 2 After edge enhancement, the clarity of the character edges is improved by 60%, meeting the requirements for subsequent recognition.

[0060] Reference Figure 2This indicates the advantages of this invention in recognizing complex backgrounds, specifically addressing the problem of low accuracy in traditional OCR technology in scenarios such as reflection, blur, and complex patterns.

[0061] Reference Figure 3 This indicates the efficiency advantages of the present invention in each processing stage, and also highlights the necessity of semantic correction and domain verification stages.

[0062] Reference Figure 4 This indicates the effectiveness of the active learning and feedback correction mechanism of the present invention, which solves the problems of traditional OCR systems being unable to continuously optimize and having weak generalization ability.

[0063] Reference Figure 5 This indicates the advantages of the present invention in terms of accuracy and intelligence in the process of matching drug information with medical orders, solving the problems of low accuracy and weak anomaly identification ability of traditional manual + simple comparison methods.

[0064] The above are merely preferred embodiments of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A multi-scale OCR recognition and matching method for pharmaceutical packaging with complex backgrounds, characterized in that, Includes the following steps: Image acquisition and preprocessing steps: Acquire images of drug packaging using a handheld device, perform multi-channel adaptive filtering preprocessing, use Gaussian bilateral filtering for noise reduction, enhance contrast using the Retinex algorithm, use Otsu adaptive binarization, and correct tilted images using Hough transform. Character region detection steps: Use the YOLOv8 network to locate the character regions on the drug packaging, input the enhanced image, and output candidate character boxes; filter the candidate boxes, perform non-maximum suppression processing, and output the coordinates and region types of the effective candidate boxes; Multi-scale feature extraction and recognition steps: Construct an improved CNN-Transformer hybrid network. The CNN uses ResNet50 to extract features, and the Transformer fuses the features; input character box features, and output preliminary prediction sequences and confidence scores; Semantic language model correction steps: Introduce a pre-trained BERT-based language model to construct a semantic correction window; calculate the reasonableness score of the input character sequence and correct semantically unreasonable sequences; output the corrected character sequence and semantic correction confidence. Domain knowledge verification steps: Construct a standard thesaurus for the pharmaceutical domain, and use the BiLSTM-CRF model to extract character sequence entities; compare the entities with the domain thesaurus, mark non-matching entities as anomalies, and output a list of anomaly entities and standard matching suggestions; Drug information and medical order matching steps: Obtain the unique identifier of the drug by scanning the code with a handheld device, retrieve the corresponding medical order information, and compare the character sequence with the medical order information by field; Output the matching results, indicating that some matched fields are not matched. Active learning and feedback correction steps: Display matching results and anomaly markers on a handheld device, record correction content, and label error types; When the accumulated error samples exceed the threshold, fine-tune the model and redeploy it. It also includes: the adaptive filter weight dynamic adjustment step: the filter weight is calculated by the formula W=α·W1+β·W2, where W is the final filter weight, α is the denoising weight coefficient, W1 is the Gaussian bilateral filter weight, β is the contrast enhancement weight coefficient, and W2 is the Retinex algorithm weight. It also includes: a semantic correction result confidence assessment step: the comprehensive confidence of the corrected character sequence is calculated using the formula C=C1·γ+C2·(1−γ), where C is the comprehensive confidence, C1 is the initial recognition confidence output by the CNN-Transformer hybrid network, γ is the recognition confidence weight, C2 is the semantic reasonableness score output by the BERT semantic model, and 1−γ is the semantic score weight; when the comprehensive confidence is lower than the threshold, the system automatically triggers secondary semantic correction; In the multi-scale feature extraction and recognition steps, the improved CNN-Transformer hybrid network uses an attention weighting mechanism to fuse features; the CNN outputs character details, local structure, and global contour feature maps, assigns attention weights, and fuses features through matrix multiplication; the Transformer calculates the fused feature sequence through self-attention and outputs the character probability distribution; the character with the highest probability is selected to form the preliminary prediction sequence, and the prediction confidence of a single character is the corresponding probability value; In the domain knowledge verification step, through formulas Calculate entity matching similarity, where S is the matching similarity and N is the number of similarities. correct To ensure the correct number of entities are matched with the domain thesaurus, N total The total number of entities to be extracted; when the similarity is below the threshold, the system outputs candidate entities for selection.

2. The multi-scale OCR recognition and matching method for pharmaceutical packaging with complex backgrounds according to claim 1, characterized in that, In the active learning and feedback correction steps, an error type weighting strategy is used to screen samples; error types are classified, weighted error scores of error samples are calculated, and samples with weighted error scores exceeding the threshold are added to the training dataset; when fine-tuning the model, incremental training is used, and the proportion of new samples in the training set is not less than 30%.

3. The multi-scale OCR recognition and matching method for pharmaceutical packaging with complex backgrounds according to claim 1, characterized in that, In the process of matching drug information with medical orders, a tiered processing mechanism is used for some matching scenarios. When the drug name field does not match, it is judged as a level 1 mismatch. The system automatically retrieves the drug alias database for a second match, and the second match is updated to a complete match. When the concentration and dosage fields do not match, it is judged as a level 2 mismatch. The system calculates the concentration and dosage deviation rate. If the deviation rate is lower than the threshold, the system prompts the user. If the deviation rate exceeds the threshold, it is marked as abnormal. When the drug name does not match the concentration and dosage, it is judged as a level 3 mismatch. It is marked as abnormal and subsequent operations are prohibited.

4. The multi-scale OCR recognition and matching method for pharmaceutical packaging with complex backgrounds according to claim 1, characterized in that, Also includes: Visualization and alarm steps for matching results: The handheld display unit uses a segmented display. Completely matched results are displayed in green font as "Match Successful", partially matched results are displayed in yellow font as "Partial Match" with the non-matching fields highlighted, and non-matching results are displayed in red font as "Match Failed" with an alarm prompt box popping up; the alarm prompt box provides vibration feedback and records the matching result log.

5. The multi-scale OCR recognition and matching method for pharmaceutical packaging with complex backgrounds according to claim 1, characterized in that, In the active learning and feedback correction steps, a dynamic learning rate strategy is used to fine-tune the model; an initial learning rate is set for fine-tuning, and the model recognition accuracy is calculated iteratively; when the accuracy improvement exceeds the threshold, the current learning rate is maintained; when the accuracy improvement is lower than the threshold, the learning rate is reduced to 0.5 times the original value and fine-tuning is stopped.

6. The multi-scale OCR recognition and matching method for pharmaceutical packaging with complex backgrounds according to claim 1, characterized in that, In the image acquisition and preprocessing steps, supplementary preprocessing is used for special and complex backgrounds; polarization filtering algorithm is used for reflective areas, image restoration algorithm is used for stained areas, and edge enhancement algorithm is used for blurred character printing areas. After supplementary preprocessing, multi-channel adaptive filtering is re-executed.

Images