A method and apparatus for text recognition and drug name extraction on medicine bottle labels based on line scan imaging

Abstract

This invention discloses a method and apparatus for text recognition and drug name extraction on medicine bottle labels based on line scanning imaging. The apparatus includes: a rotating stage for carrying and rotating the medicine bottle; a line scanning camera that acquires line-by-line image data at a constant line frequency while the medicine bottle rotates; an image stitching unit that stitches the line-by-line images in real time according to encoder signals to generate a cylindrical unfolded image, eliminating surface distortion at the hardware level; a text recognition unit that performs OCR text detection and recognition on the unfolded image and outputs a fully preserved text sequence; and a semantic parsing unit that uses a pre-trained large language model to perform semantic-level error correction on the text sequence and extracts the generic name, brand name, and specifications of the drug. This invention actively eliminates distortion and reflection interference through line scanning imaging and achieves semantic-level text understanding by combining a large language model, significantly improving the accuracy and robustness of drug name extraction. It is suitable for automated finished product verification scenarios of cylindrical medicine bottles.

Description

Technical Field

[0001] This invention relates to the field of pharmaceutical finished product verification technology, and in particular to a method and apparatus for recognizing text on medicine bottle labels and extracting drug names based on line scanning imaging. Background Technology

[0002] In pharmaceutical manufacturing, warehousing, and compounding centers, finished product verification is a crucial step in ensuring medication safety. The automatic recognition of label information (generic name, trade name, specifications, etc.) on cylindrical medicine bottles (such as ampoules, vials, and oral liquid bottles) is an important prerequisite for achieving automated verification.

[0003] Currently, relevant technical solutions have been proposed in the industry. Patent document CN121305584A discloses a method for recognizing the content of medicine bottle labels based on multiple cameras and YOLOv8. This method uses multiple cameras to acquire images of the medicine bottle from different angles, uses the YOLOv8 model to detect the label area, performs text recognition through OCR, and combines chessboard calibration for spatial positioning. It then selects the optimal result from multiple OCR results using the minimum reprojection error.

[0004] However, this existing technical solution has the following shortcomings in practical applications: First, the hardware imaging architecture has inherent flaws. This solution uses multiple fixed-array cameras for image acquisition, and the surface curvature of the cylindrical medicine bottle causes stretching, compression, and perspective distortion of the label text in the captured images. Although it corrects these issues using a "perspective correction" algorithm in image preprocessing, the algorithm inevitably introduces interpolation errors and edge blurring, and the correction effect is limited when the label tilt angle is large or the surface curvature changes drastically. Furthermore, the multi-camera architecture faces the risk of information redundancy in overlapping fields of view and missed captures in non-overlapping areas, resulting in high system complexity and hardware costs.

[0005] Second, the solution to the reflection problem is passive and its effectiveness is limited. This scheme relies on data augmentation during the training phase (such as randomly adjusting brightness, contrast, and saturation to simulate different lighting conditions) and contrast enhancement during image preprocessing to address the reflection problem on the medicine bottle surface. This is a "passive adaptation" strategy, meaning that model training and image processing aim to increase tolerance to existing reflective images, rather than suppressing reflection at its source. When there is strong specular reflection on the medicine bottle surface, locally overexposed areas will permanently lose text information, which cannot be recovered by later algorithms.

[0006] Third, the post-processing of text recognition lacks semantic understanding capabilities. This solution uses a "minimum reprojection error" method based on spatial geometry to select the best OCR result from multiple cameras. Essentially, this method "selects" the text that best matches the others in geometric coordinates, lacking the ability to understand the semantic content of the text itself. It cannot correct semantically for errors such as ambiguity, misidentification, or omissions in OCR. For example, misidentifying "sodium chloride injection" as "sodium chloride injection" cannot be corrected based on medical knowledge. Furthermore, this method only outputs the raw OCR text on the label and cannot directly extract structured key information such as the generic name, brand name, and specifications of the drug from lengthy label text, requiring additional database mapping or manual intervention.

[0007] Fourth, end-to-end processing efficiency needs improvement. The solution involves steps such as multi-camera calibration, 3D coordinate calculation, multi-view data fusion, and reprojection error optimization, which has high computational complexity. When performing high-speed verification of large batches of medicines, the system's processing speed may become a bottleneck.

[0008] Therefore, there is an urgent need for a solution for recognizing text on medicine bottle labels and extracting drug names that can eliminate surface distortion and reflection interference at the hardware level, possess semantic-level text understanding capabilities, and meet the processing efficiency requirements of high-speed production lines. Summary of the Invention

[0009] This invention addresses the problems of high system complexity and limited processing efficiency in existing medicine bottle label recognition schemes based on multi-faceted camera arrays. It proposes a method and device for medicine bottle label text recognition and drug name extraction based on line scan imaging, which eliminates surface distortion, actively suppresses reflection, and utilizes a large language model to achieve semantic-level error correction and information extraction, thereby significantly improving recognition accuracy and processing efficiency.

[0010] To achieve the above objectives, the following technical solution is proposed: A device for recognizing text and extracting drug names from medicine bottle labels based on line scanning imaging, comprising: A rotating stage, configured to hold and rotate medicine bottles; A line scan camera, fixedly mounted and with its field of view covering the surface of the medicine bottle on the rotating stage, is configured to scan at a constant line frequency as the medicine bottle rotates to acquire line-by-line image data of the surface of the medicine bottle. The image stitching unit is coupled to the encoder of the line scan camera and the rotating stage, and is configured to stitch the line-by-line image data in real time according to the signal of the encoder to generate the cylindrical unfolded image of the medicine bottle label; The text recognition unit is configured to perform text detection and recognition on the cylindrical unfolded image and output a text sequence; and The semantic parsing unit is configured to use a pre-trained large language model to perform semantic-level parsing and error correction on the text sequence in order to extract structured drug name information.

[0011] Through the above technical solution, by using a rotating stage in conjunction with a line-scan camera for line-by-line scanning imaging, and with the image stitching unit generating the cylindrical unfolded image in real time based on encoder signals, the text stretching, compression, and perspective distortion caused by the cylindrical curved surface are fundamentally eliminated at the hardware imaging level. Complete, distortion-free label planar images can be obtained without relying on perspective correction algorithms. Simultaneously, the line-by-line acquisition characteristic of line scanning imaging ensures that only a narrow band area is illuminated and imaged at any given time. Combined with the optimized design of the light source angle, specular reflection light paths can be effectively avoided, thus actively suppressing local reflections during the imaging process and avoiding the strip-shaped or arc-shaped overexposed areas common in area scan cameras. Furthermore, by incorporating a text recognition unit to perform OCR recognition on the distortion-free, uniformly illuminated cylindrical unfolded image, the completeness and accuracy of the original text sequence are significantly improved. Furthermore, by setting up a semantic parsing unit to perform semantic-level parsing and error correction on text sequences using a large language model, it can not only correct errors such as homographs and missing characters generated by OCR, but also intelligently extract core verification fields such as generic drug name, brand name, and specifications from the entire tag text, achieving a leap from "recognition" to "understanding". The output results can be directly used by the verification system for automated verification, greatly reducing the need for manual intervention.

[0012] Preferably, the system also includes a light source, which is a line light source, and the direction of its emitted light is configured to keep the specular reflected light away from the receiving light path of the line scanning camera, so as to suppress local reflections on the surface of the medicine bottle during the imaging process.

[0013] By employing the above technical solution, and by configuring the light source as a line source, and through spatial geometric design to deflect specular reflections from the camera's receiving light path, the specular reflections caused by the curvature of the cylindrical surface of the medicine bottle can be effectively prevented from entering the lens during line-by-line scanning. This achieves active suppression of reflections from an optical perspective. Compared to existing technologies that perform de-reflection processing after image acquisition or rely on data augmentation training models to passively adapt to reflective scenes, this solution ensures image quality at the source, preventing the permanent loss of text information due to overexposure caused by reflections.

[0014] Preferably, the character recognition unit includes: The label region positioning module is configured to automatically locate the effective text region of the label and generate a mask on the cylindrical unfolded image after eliminating surface distortion, based on the grayscale difference and texture difference between the medicine bottle label and the bottle background in the cylindrical unfolded image, using the region recognition rules of vertical projection analysis and / or edge gradient statistics. The OCR recognition module is configured to perform text recognition on the area defined by the mask and output the original text fragment containing the position coordinates.

[0015] Through the above technical solution, the label region localization module fully utilizes the inherent advantages of cylindrical unfolded images obtained by line scanning imaging—namely, the absence of surface distortion and uniform illumination—and employs traditional image processing rules such as vertical projection analysis and / or edge gradient statistics for label region localization. Therefore, it can ensure localization accuracy while avoiding the computational resources and annotation data costs required for deploying complex deep learning object detection models. There is an inherent synergy between this technical choice and the line scanning imaging architecture: line scanning imaging provides high-quality, standardized input images, allowing relatively simple rule-based methods to achieve ideal results; while the low complexity of traditional rule-based methods further reduces the overall computational overhead and deployment threshold of the system.

[0016] Preferably, the text recognition unit is further configured to sort and concatenate the original text fragments according to the position coordinates in a top-to-bottom and left-to-right order to form a complete, fully preserved text string, which serves as the text sequence input to the semantic parsing unit; the full preservation provides the large language model with a complete context without information loss.

[0017] By employing the above technical solutions, the character recognition unit fully preserves and spatially sorts all text fragments output by the OCR, rather than filtering or selecting based on confidence thresholds. This allows it to provide subsequent large language models with original text information containing complete context. For large language models, complete contextual information is crucial for their semantic understanding and error correction capabilities. Discarding low-confidence text fragments during the input stage can lead to the loss of key semantic information, preventing the large model from accurately determining the direction of error correction. This solution, through its "full preservation" strategy, enables large language models to perform error correction and information extraction within the complete semantic context, significantly improving the accuracy and robustness of the final extraction results.

[0018] Preferably, the semantic parsing unit is configured to: input the text sequence and the preset drug naming standard into the large language model in a structured manner, and instruct the large language model to output the extracted generic name, trade name and corresponding specification information of the drug.

[0019] Through the above technical solution, because the semantic parsing unit inputs drug naming specifications as part of the structured prompts into the large language model, the large model performs targeted information extraction under the constraints of pharmaceutical professional knowledge. Therefore, it can accurately distinguish different fields such as generic names, brand names, and specifications in the label text, avoiding misidentification of trademark names as generic names or confusion between packaging specifications and drug specifications. Compared with simply outputting the raw OCR text and then relying on external database mapping, this solution has stronger scenario adaptability and robustness, and is especially suitable for dynamic scenarios such as adding new drugs, changing label formats, and databases not being updated in a timely manner.

[0020] Preferably, a verification unit is also included, coupled to the semantic parsing unit, configured to access a standard drug name database, and perform rule verification on the drug name information extracted by the semantic parsing unit based on the standard drug name database. The rule verification includes: performing fuzzy matching between the extracted generic drug name and the standard drug name database, triggering verification failure when the matching degree is lower than a preset threshold; marking the results of verification failure or parsing abnormality as pending manual review, and simultaneously associating and saving the results with the corresponding cylindrical unfolded image for iterative training of the large language model.

[0021] Through the above technical solutions, a multi-layered quality assurance mechanism is formed by setting up a verification unit to perform rule-based verification of the semantic extraction results of the large language model based on a standard drug name database, and saving abnormal results along with the original image for model iterative training. This mechanism includes "OCR recognition, large model semantic extraction, knowledge base verification, and anomaly feedback closed loop." This closed-loop iterative mechanism allows the system to continuously accumulate boundary cases during operation, which are then used for fine-tuning and optimizing the large language model. This gradually improves the system's ability to handle rare drug names and blurred printed text, achieving self-evolution of system performance.

[0022] A method for recognizing text and extracting drug names from medicine bottle labels based on line scan imaging includes: The rotating stage is controlled to rotate the medicine bottle, and at the same time, a fixedly installed line scan camera scans the surface of the medicine bottle at a constant line frequency to acquire line-by-line image data. Based on the encoder signal of the rotating stage, the line-by-line image data is stitched together in real time to generate a cylindrical unfolded image of the medicine bottle label; The cylindrical unfolded image is subjected to text detection and recognition to obtain a text sequence; The text sequence is semantically parsed and corrected using a pre-trained large language model, and structured drug name information is extracted and output.

[0023] By employing the above technical solution, the cylindrical unfolded image is generated using a line-scan rotational scanning method, eliminating surface distortion from the imaging stage and providing high-quality image input for subsequent text recognition. Simultaneously, the use of a large language model for semantic-level parsing and error correction of the OCR recognition results enables the system to possess a deep understanding of the label text content, intelligently extracting key information about the drug name from complete text sequences and semantically correcting recognition errors. This method organically combines hardware imaging optimization with artificial intelligence semantic understanding, achieving high-quality automated processing throughout the entire process from image acquisition to structured information output.

[0024] Preferably, the step of acquiring line-by-line image data further includes: configuring a line light source and setting the illumination direction of the line light source and the scanning direction of the line scan camera, so as to guide the specular reflection light from the surface of the medicine bottle to outside the receiving field of view of the line scan camera.

[0025] By using the above technical solutions, the reflected light from the mirror is guided outside the camera's field of view through optical path geometry design during the imaging stage. This effectively avoids interference from the reflective area on text recognition without relying on software post-processing, thus ensuring the integrity and clarity of the text information in the label image.

[0026] Preferably, the steps for text detection and recognition in the cylindrical unfolded image specifically include: Based on the grayscale and texture differences between the medicine bottle label and the bottle background in the cylindrical unfolded image, and taking advantage of the characteristics of the cylindrical unfolded image being free of surface distortion and having uniform illumination, the effective text area of ​​the label is automatically located and a mask is generated by using the region recognition rules of vertical projection analysis and / or edge gradient statistics. The OCR text recognition model is invoked to recognize the masked area and output the original text fragment containing the position coordinates. Based on the position coordinates, the original text fragments are sorted and spliced ​​in order from top to bottom and from left to right to form a complete text string that retains all information, which serves as the text sequence and provides the large language model with a complete semantic context without information loss.

[0027] By fully utilizing the high-quality image characteristics of cylindrical unfolded images, which are distortion-free and uniformly illuminated, efficient and accurate label region localization can be achieved using traditional image processing rules. Furthermore, the text splicing strategy that retains all text provides complete semantic context information for the large language model. Therefore, it can reduce the complexity of the algorithm while ensuring the input quality of the subsequent semantic parsing stage.

[0028] Preferably, after extracting and outputting structured drug name information, the process also includes: The extracted drug name information is then subjected to fuzzy matching verification with the standard drug name database. For results that fail verification or whose parsing confidence is below a threshold, a label is generated for manual review, and the result is associated with and stored in relation to the corresponding cylindrical unfolded image for iterative optimization training of the large language model.

[0029] By employing the above technical solutions, and by adding a fuzzy matching verification step based on a standard drug name database and establishing a feedback loop from abnormal results to model training, the system can continuously optimize the extraction capabilities of the large language model by utilizing boundary samples accumulated during daily operation, thus ensuring the security of the verification process and enabling the system to continuously improve its performance over long-term operation.

[0030] Therefore, the present invention has at least the following beneficial effects: First, surface distortion is eliminated at the hardware level, significantly improving text integrity. A line-scan camera combined with a rotating stage acquires the full-circumference cylindrical image of the medicine bottle, obtaining a distortion-free planar image of the label without relying on perspective correction algorithms. The label text integrity is close to that of a planar label, resulting in a significant improvement in OCR recognition accuracy. In a test containing 4000 bottles of medicine, the overall drug name recognition accuracy reached 99.94%.

[0031] Second, it actively suppresses reflection interference to ensure image quality. Through the spatial geometry design of the optical path of the line light source and the line scan camera, the specular reflected light is guided outside the camera's receiving field of view, actively avoiding overexposure due to reflection during the imaging process. Compared with the passive adaptation strategy of data enhancement in existing technologies, this approach can fundamentally ensure the integrity of text information.

[0032] Third, it is compatible with cylindrical medicine bottles of various diameters and materials. The line scan camera, in conjunction with the encoder, can adapt to medicine bottles of different diameters (such as ampoules ranging from 2ml to 20ml) and materials (such as transparent glass, brown glass, plastic, etc.). It can be adapted by adjusting parameters such as the rotation speed of the rotating stage, the line scan camera's line frequency, and the brightness of the light source. There is no need to change hardware or frequently adjust parameters, making the system highly versatile.

[0033] Fourth, semantic-level error correction and key information extraction enhance recognition robustness. A pre-trained large language model is introduced to perform semantic analysis and error correction on the original OCR text sequence. Combined with medical expertise and drug naming standards, it can not only correct OCR recognition errors, but also accurately extract core verification fields such as generic drug name, brand name, and specifications from lengthy label text, achieving a leap from "recognition" to "understanding".

[0034] Fifth, the system boasts high processing efficiency, meeting the cycle time requirements of high-speed production lines. Single-bottle line scan imaging and stitching takes no more than 2 seconds, and the overall algorithm process takes no more than 6 seconds, satisfying the production cycle time requirements of pharmaceutical finished product verification and dispensing machines. It is suitable for high-speed automated verification scenarios involving large volumes of pharmaceuticals.

[0035] Sixth, the system architecture is simple and the deployment is flexible. It uses a single-line scanning camera instead of a multi-camera array, eliminating the need for complex multi-camera calibration and spatial coordinate alignment steps. The system integration is simpler and can be directly embedded into existing finished product review pipelines, interfacing with the standard interface of the review system. Attached Figure Description

[0036] Figure 1 This is a schematic diagram of the structural connection of the medicine bottle label text recognition and drug name extraction device based on line scanning imaging provided in an embodiment of the present invention.

[0037] Figure 2 The flowchart illustrates a method for recognizing text on medicine bottle labels and extracting drug names based on line scanning imaging, provided in an embodiment of the present invention. Detailed Implementation

[0038] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention. Example

[0039] Please see Figure 1 , Figure 1 This is a schematic diagram illustrating the structural connections of a medicine bottle label text recognition and drug name extraction device based on line scanning imaging, provided in an embodiment of the present invention. The device includes: a rotating stage, a line scanning camera, an encoder, an image stitching unit, a text recognition unit, a semantic parsing unit, and a verification unit. Specifically, the line scanning camera and the image stitching unit are coupled via a data interface; the encoder and the image stitching unit are coupled via a signal interface; the image stitching unit and the text recognition unit are coupled via a data bus; the text recognition unit and the semantic parsing unit are communicatively connected; and the semantic parsing unit and the verification unit are communicatively connected.

[0040] The rotating stage is configured to hold and rotate medicine bottles. Driven by a servo motor, the rotation speed is adjustable from 0.5 rpm to 3 rpm depending on the bottle diameter to accommodate different bottle sizes. For common 2ml ampoules (approximately 10mm in diameter), the speed is set to approximately 2 rpm; for 20ml ampoules (approximately 22mm in diameter), the speed is set to approximately 1 rpm. The stage's axis of rotation is perpendicular to the horizontal plane, ensuring that the label remains within the depth of field of the line-scan camera during rotation. The rotating stage is equipped with an adaptive clamping mechanism compatible with cylindrical medicine bottles from 8mm to 30mm in diameter, with an adjustable clamping force ranging from 0.5N to 5N to prevent damage to the bottles or slippage during rotation.

[0041] The line scan camera is fixedly mounted on one side of the rotating stage, with its optical axis aligned horizontally with the surface of the medicine bottle. In this embodiment, the line scan camera is a monochrome CMOS line scan camera with a resolution of 4096 pixels and a pixel size of 7μm × 7μm. The line frequency can be adjusted within the range of 1kHz to 80kHz. The line frequency of the line scan camera is matched with the rotational linear speed of the medicine bottle, ensuring that the resolution corresponding to the arc length of the medicine bottle surface covered by each scan is not less than 300dpi (dots per inch), meeting the requirements for clear imaging of label text. The lens of the line scan camera is a telecentric lens with a magnification of 0.5x, a field of view of approximately 57 mm, and a depth of field of approximately ±3 mm. The use of a telecentric lens ensures that the object-side magnification remains constant within a certain diameter tolerance and installation eccentricity range, and the image size does not change with the object distance.

[0042] The encoder, an incremental rotary encoder mounted on the rotating stage shaft, outputs 4096 pulse signals per revolution. The encoder's A-phase and B-phase signals are connected to the image stitching unit to trigger line acquisition by the line scan camera and provide the bottle rotation angle information for image stitching. Using A / B phase orthogonal encoded signals allows for higher precision angle subdivision in subsequent processing, improving stitching accuracy.

[0043] The light source is a line light source with a narrow emitting surface approximately 3mm wide and 100mm long, parallel to the scanning line of the line scanner. The light source is installed at an angle of approximately 15° to 25° to the line scanner, with the specific angle fine-tuned based on the curvature of the medicine bottle surface and the label material. This geometric configuration causes the specular reflection direction of the light source on the medicine bottle surface to deviate from the receiving light path of the line scanner, effectively suppressing local reflections during the imaging process. The light source uses a high-brightness LED array with a color temperature of 5500K, close to natural light, and a color rendering index greater than 90, ensuring accurate color reproduction for label printing. The brightness of the light source can be adjusted via a PWM (Pulse Width Modulation) signal, with a duty cycle adjustable from 10% to 100% to accommodate medicine bottle materials with different reflectivities (such as transparent glass, amber glass, and plastic).

[0044] The image stitching unit is coupled to the line scan camera and encoder. In this embodiment, the image stitching unit is implemented in FPGA (Field Programmable Gate Array) hardware, specifically using a Xilinx Kintex-7 series FPGA chip. It receives image data from the line scan camera via a CameraLink interface and encoder signals via an RS-422 differential interface. The image stitching unit internally includes: a line buffer for caching image data of the current scan line; an encoder counting module for accumulating encoder pulses and generating line trigger signals; and a stitching control state machine for determining the write position of each line based on encoder pulses, writing the line-by-line data sequentially to the frame buffer to generate a complete cylindrical unfolded image. The FPGA's integrated DDR3 SDRAM serves as the frame buffer memory, with a capacity of 512MB, capable of storing multiple frames of unfolded images for subsequent processing. The image stitching unit transmits the stitched cylindrical unfolded image to a host industrial computer via a PCIe 2.0 x4 interface.

[0045] During image acquisition, by maintaining a constant rotational speed of the rotating stage (with speed fluctuations controlled within ±1%), a constant PWM duty cycle of the line light source, and constant parameters such as exposure time and gain of the line scan camera, the cylindrical unfolded image output each time is guaranteed to have repeatable clarity and brightness consistency, meeting the requirements of image quality stability for final product verification.

[0046] The host industrial control computer serves as the central computing platform for this device, equipped with a character recognition unit, a semantic parsing unit, and a verification unit. The industrial control computer configuration includes: an Intel Core i7-12700 processor (12 cores, 20 threads, base frequency 2.1GHz), 32GB DDR4 memory, an NVIDIA GeForce RTX 3070 GPU (8GB GDDR6 video memory), and a 512GB NVMe SSD. The operating system is Ubuntu 20.04 LTS.

[0047] The text recognition unit operates as a software module on the host industrial control computer, including a label area positioning module and an OCR recognition module.

[0048] The label region localization module is implemented based on the OpenCV library, and its workflow is as follows: First, the input cylindrical unfolded image is converted to grayscale; second, a 5×5 Gaussian kernel is used to filter and reduce noise in the grayscale image; then, the image is subjected to vertical projection analysis, and the grayscale mean of each column is calculated. The left and right boundaries of the label are determined based on the grayscale difference between the label region and the bottle background; at the same time, the horizontal gradient magnitude of the image is calculated, the gradient energy of each row is calculated, and the upper and lower boundaries of the label are determined based on the gradient difference between the label text lines and the blank areas of the bottle; finally, a binary mask image is generated based on the determined boundaries, with 1 (valid text region) inside the mask area and 0 (background region) outside the mask area.

[0049] To further improve the accuracy of boundary positioning, this embodiment adopts the following precise boundary positioning algorithm: Within a range of ±10 pixels from the initially determined boundary position, the gray-level gradient magnitude G of the local region is calculated, and the calculation formula is as follows: G(x,y)=sqrt((I(x+1,y)-I(x-1,y))^2+(I(x,y+1)-I(x,y-1))^2); Here, I(x,y) represents the pixel grayscale value at coordinates (x,y) in the unfolded image, and sqrt represents the square root operation. The location of the maximum gradient magnitude is used as the label boundary after precise localization, achieving sub-pixel level accuracy. This method fully utilizes the characteristics of cylindrical unfolded images—no surface distortion and uniform illumination—resulting in clear and consistent boundaries between labels and the background, achieving high-precision region localization without the need for complex deep learning methods.

[0050] The OCR recognition module uses an offline deployment version of the PaddleOCR framework, specifically its PP-OCRv3 model architecture. This model includes: a text detection sub-network (based on DBNet) for detecting text line regions; and a text recognition sub-network (based on SVTR) for recognizing text content. The model weight file undergoes pruning and INT8 quantization, achieving an inference speed of less than 50ms per image on an RTX 3070 GPU. The OCR recognition module receives the masked image generated by the label region localization module, crops the image within the masked region, and feeds it into the OCR model for recognition, outputting the text content of each line of text and its bounding box coordinates in the unfolded image.

[0051] After acquiring the original text fragments from the OCR output, the text recognition unit sorts and concatenates them according to the bounding box coordinates of each fragment. The sorting rule is as follows: first, sort by the ordinate of the top-left corner of the bounding box from smallest to largest (from top to bottom); for fragments with similar ordinates (difference less than 10 pixels), then sort by the x-coordinate from smallest to largest (from left to right). After sorting, the text content of each fragment is concatenated in order, with different lines separated by newlines, forming a complete text string that retains all recognized text information on the label, without any confidence-based filtering or discarding operations, to ensure that a complete semantic context without information loss is provided for subsequent large language models.

[0052] The semantic parsing unit is configured to use the pre-trained Qwen 3.5 large language model to perform semantic-level parsing and error correction on the text sequence. In this embodiment, the Qwen 3.5 model uses the 4-bit quantization version (Qwen 3.5-7B-Instruct-GPTQ-Int4), which is deployed on the NVIDIA GeForce RTX 3070 GPU of the upper industrial control computer. The video memory occupancy is about 4.5GB, and the single inference time is about 0.5 to 1.0 seconds. The model is accelerated and deployed through the vLLM inference framework, supports dynamic batching, and can concurrently process the recognition requests of multiple medicine bottles.

[0053] The working process of the semantic parsing unit is as follows: First step, constructing structured prompt words. Combine the full amount of reserved text strings output by the text recognition unit with the preset medicine naming specification template to form structured prompt words. The format of the prompt word template is: "You are an assistant for extracting medicine label information. Please extract the generic name, trade name, and specification information of the medicine from the label text according to the following medicine naming specifications. Naming specifications: (1) The generic name of the medicine should be the standard name included in the Chinese Pharmacopoeia; (2) The trade name is the registered trademark name of the production enterprise; (3) The specification usually appears in the form of a number + unit combination with units such as'ml','mg', 'g', 'tablet', etc. Please output in JSON format: {'generic name': '', 'trade name': '', 'pecification': ''}. The content of the label text is as follows: "+ the original text string.

[0054] Second step, large language model inference. Input the constructed prompt words into the Qwen 3.5 large language model, and the model performs semantic understanding, error correction, and information extraction under the prior constraints of medical professional knowledge. The model has learned a large amount of medical literature, medicine instructions, and pharmacopoeia content during the pre-training stage, and has the capabilities of identifying spelling mistakes in medicine names, confusion of homographs (such as 'na' and 'natrium','mo' and'mu', etc.), and restoring abbreviations of professional terms.

[0055] Third step, output parsing. Parse the JSON string output by the large language model, and extract the field values of 'generic name', 'trade name', and'specification' as the structured result of semantic parsing.

[0056] The verification unit is coupled with the semantic parsing unit and is configured to access the standard medicine name library for rule verification. In this embodiment, the standard medicine name library is an integrated version of the Chinese Pharmacopoeia (2020 edition) and the medicine database approved by the National Medical Products Administration, which is locally stored in the SQLite database and contains about 150,000 records of medicine generic names, supporting offline queries.

[0057] The rule verification process of the verification unit includes: First, fuzzy matching verification. The generic names of drugs output by the semantic parsing unit are fuzzily matched with the standard drug name database, and the similarity is calculated using the Levenshtein edit distance algorithm. The Levenshtein edit distance is defined as the minimum number of single-character editing operations (insertion, deletion, replacement) required to transform one string into another. The similarity S between the edit distance D and the string length L is calculated using the following formula: S = 1 - D / max(L_pred, L_ref); Where D is the edit distance, max(L_pred,L_ref) represents the string length of the longer of the predicted name L_pred and the reference name L_ref, and S is the similarity, with a value ranging from 0 to 1.

[0058] When the similarity S is greater than or equal to 0.85, the fuzzy matching is considered successful, and the semantic parsing result is marked as "automatically passed"; when the similarity S is less than 0.85, the verification is considered to have failed, and the manual review mark is triggered.

[0059] Secondly, a comprehensive confidence assessment is performed. A weighted comprehensive score is applied, combining the text confidence score output by the OCR recognition module with the generation confidence score output by the large language model (calculated using the log probability of the model's output token). The formula for calculating the comprehensive score C is as follows: C=alpha×C_ocr+(1-alpha)×C_llm; Where C_ocr is the average confidence score of each text segment output by the OCR recognition module, ranging from 0 to 1; C_llm is the generation confidence score of the output field of the large language model, ranging from 0 to 1; and alpha is the weight coefficient, with a value of 0.4. When the comprehensive score C is lower than 0.7, the result is marked as pending manual review, regardless of whether the fuzzy matching passes.

[0060] Finally, closed-loop data recording is performed. For results that fail verification or have a comprehensive score below the threshold, the verification unit packages and stores the following information: the original cylindrical unfolded image (lossless compression in PNG format), the original OCR text sequence, the original JSON string output by the large language model, the fuzzy matching similarity score, and the comprehensive confidence score. This data package is stored in a designated directory on the industrial control computer's SSD, with filenames named using the timestamp and the medicine bottle serial number for easy retrieval later. The accumulated data packages will serve as the dataset for fine-tuning and iterative training of the large language model, forming a closed-loop mechanism for continuous optimization of system performance.

[0061] The overall signal flow of the device is as follows Figure 1As shown: The encoder generates a trigger pulse signal to trigger a line scan camera to acquire line-by-line image data. The line scan camera transmits the image data to the image stitching unit. The image stitching unit stitches the images in real time according to the encoder angle signal, generating a cylindrical unfolded image and transmitting it to the host industrial control computer via the PCIe interface. The text recognition unit receives the cylindrical unfolded image, performs text detection and recognition, and outputs a fully preserved text string. The semantic parsing unit receives the text string, performs semantic parsing and error correction, and outputs structured drug name information. The verification unit receives the semantic parsing results and compares them with the standard drug name database, outputting the final result and anomaly markers. Example

[0062] Please see Figure 2 , Figure 2 This is a flowchart illustrating a method for text recognition and drug name extraction on medicine bottle labels based on line scanning imaging, provided in an embodiment of the present invention. This method corresponds to the device described in Embodiment 1, and each step can be executed by a module of the aforementioned device. The method includes the following steps: Step S101: System initialization and parameter configuration.

[0063] Upon system startup, the following initialization operations are performed: Loading the rotating stage control parameters, including a rotation speed mapping table for different medicine bottle sizes; loading the line scan camera configuration parameters, including exposure time, line frequency, and gain; loading the line light source PWM duty cycle parameters; initializing the image stitching unit FPGA firmware and configuring the encoder resolution-image size correspondence; loading the OCR text recognition model weight file into memory; loading the Qwen3.5 large language model weight file into GPU memory; and loading the standard drug name library into a hash table structure in memory for fast lookup. After initialization, the system enters standby mode, waiting for a medicine bottle arrival signal.

[0064] Step S102: Line scan image acquisition.

[0065] Once the vial is conveyed to the rotating stage and positioned, the system triggers the image acquisition process. Specific operations include: controlling the rotating stage to begin rotation, accelerating to a preset speed, and maintaining stability. For example, for a common 20ml ampoule (approximately 22mm in diameter), the rotating stage speed is set to approximately 1 revolution per second, corresponding to a surface linear velocity of approximately 69mm / s; for a 2ml ampoule (approximately 10mm in diameter), the speed is set to approximately 2 revolutions per second. Simultaneously, the line scan camera, synchronized with the encoder's trigger pulses, begins scanning line by line at a constant line frequency. Each encoder pulse triggers the line scan camera to acquire one line of image data. The formula for calculating the line frequency f_line is: f_line=(v_surface×R_encoder) / (2×pi×r_bottle); Where v_surface is the surface linear velocity of the medicine bottle (unit: mm / s), R_encoder is the number of pulses per encoder revolution (value 4096), r_bottle is the radius of the medicine bottle (unit: mm), and pi is pi. This formula ensures that the number of scan lines corresponding to each millimeter of arc length on the medicine bottle surface is consistent, avoiding image stretching or compression caused by fluctuations in rotation speed or changes in the diameter of the medicine bottle. For example, for a 20ml ampoule with a radius of 11mm, when the surface linear velocity is set to 69mm / s, the calculated line frequency is approximately 4.1kHz, corresponding to an object-side sampling resolution of approximately 14μm / pi, which meets the imaging requirements of 300dpi or higher.

[0066] During image acquisition, a line light source illuminates the scanning line position on the medicine bottle with a preset PWM duty cycle. The narrow band of the line light source's emitting surface coincides with the scanning line of the line scan camera on the medicine bottle surface. The angle between the light source's illumination direction and the camera's optical axis is set to 20°. The specular reflection light after reflection from the cylindrical surface of the medicine bottle deviates from the camera lens axis, ensuring that only diffuse reflection light enters the lens, thus achieving active suppression of reflections. By maintaining a constant rotating stage speed (speed fluctuation controlled within ±1%), a constant PWM duty cycle of the line light source, and consistent parameters such as the line scan camera's exposure time and gain in each acquisition, the cylindrical unfolded image output from each acquisition is guaranteed to have repeatable clarity and brightness consistency, meeting the image quality stability requirements for final product verification.

[0067] Step S103: Real-time image stitching and unfolding.

[0068] The image stitching unit receives the line-by-line image data output from the line scan camera and the angle count signal from the encoder, and performs real-time stitching within the FPGA. The specific stitching process is as follows: For each line of input image data, the column position (i.e., X coordinate) of that line in the cylindrical unfolded image is determined based on the current encoder cumulative count value; the line data is written to the corresponding line position in the frame buffer (i.e., the Y coordinate is fixed to the current scan line number); when the encoder cumulative count value reaches a preset value (corresponding to a 360° rotation of the medicine bottle), a complete frame of the cylindrical unfolded image is stitched. The formula for calculating the width W of the stitched unfolded image is: W=(2×pi×r_bottle) / d_pixel; Where r_bottle is the radius of the bottle, d_pixel is the object-space sampling resolution corresponding to a single pixel, with a value of 0.014mm / pixel (corresponding to the object-space resolution of a 7μm pixel size under a 0.5x telecentric lens), and pi is pi (circular diameter). For example, for a 20ml ampoule with a radius of 11mm, the calculated width of the unfolded image is approximately 4939 pixels, and the height is determined by the linear array resolution of the line scan camera, which is 4096 pixels. The stitched cylindrical unfolded image is transmitted to the upper industrial control computer's memory via the PCIe interface.

[0069] Step S104: Label area localization and OCR text detection.

[0070] After receiving the cylindrical unfolded image, the text recognition unit of the host industrial control computer first performs label region localization. Because the cylindrical unfolded image has excellent characteristics such as no surface distortion and uniform illumination, and the boundary between the label and the background is clear and highly consistent, label region localization does not require a complex deep learning object detection model. Instead, it can achieve efficient and accurate localization using the following traditional image processing rules. Specifically, these include: Vertical projection analysis: For the grayscale unfolded image, the average grayscale value of each column is calculated to form a vertical projection curve. The label area is usually printed with black or dark text, and its average grayscale value is significantly lower than that of the bottle background area. By setting a grayscale threshold, the left and right boundaries of the label are located on the vertical projection curve. The grayscale threshold T_gray is automatically calculated using the Otsu method, achieving adaptability to different lighting conditions and different label colors.

[0071] Edge gradient statistics: Calculate the Sobel horizontal gradient for the grayscale image, sum the gradient magnitudes for each line, and form a horizontal gradient energy curve. The label text area has higher gradient energy due to rich character edges; the blank areas on the bottle body have lower gradient energy. By setting a gradient energy threshold, the upper and lower boundaries of the label are located.

[0072] Based on the location boundary information, a binary mask image of the same size as the unfolded image is generated, with a pixel value of 255 (white) in the effective text area of ​​the label and 0 (black) in the background area.

[0073] Subsequently, a bitwise AND operation is performed between the mask image and the original unfolded image to crop out the valid text region of the label, which is then fed into the OCR recognition module. The OCR recognition module, based on the PaddleOCR framework, performs text detection and recognition, outputting the text content of each line of text and the coordinate information of its bounding box in the unfolded image. The coordinate information is recorded in the format of top-left corner coordinates (x_min, y_min) and bottom-right corner coordinates (x_max, y_max).

[0074] Step S105: Text sequence cleaning and regularization.

[0075] The text recognition unit sorts and concatenates the original text fragments output by the OCR. The sorting rule is as follows: the y-coordinate (y_min) of the top-left corner of the bounding box is used as the primary sorting key, and the text is sorted in ascending order; for text fragments in the same line with a y_min difference of less than 10 pixels, the x-coordinate (x_min) is used as the secondary sorting key, and the text is sorted in ascending order. After sorting, the text content of each fragment is concatenated in sequence, with different lines separated by newlines. This process retains all text content recognized by the OCR without performing any confidence-based filtering or discarding operations. This full-preservation sorting and concatenation strategy ensures that the large language model can receive the complete semantic context containing all text information of the labels and their original spatial arrangement relationships, providing a sufficient information foundation for it to perform semantic understanding and error correction capabilities.

[0076] Step S106: Semantic parsing and error correction using the Qwen3.5 large language model.

[0077] The semantic parsing unit combines the full text string obtained in step S105 with a preset drug naming convention template to construct structured prompt words. The prompt word template has been designed and optimized according to the generic name naming rules of the Chinese Pharmacopoeia, common characteristics of trade names, and drug specification expression conventions.

[0078] The system uses the Qwen3.5 language model to perform inference based on input prompts. The model outputs a structured JSON string containing the extracted "generic name," "product name," and "specification" fields. In cases where the model's output JSON parsing is incorrect (non-standard JSON format), the system performs regular expression fallback extraction, matching the corresponding values ​​from the original model output text by field name to ensure system robustness.

[0079] Step S107: Drug name verification and confidence assessment.

[0080] The verification unit performs double verification on the drug name information output in step S106.

[0081] The first layer of verification involves fuzzy matching against the standard drug name database. The extracted generic drug name string is compared with each record in the local standard drug name database using Levenshtein edit distance, and the record with the highest similarity is taken as the matching result. If the highest similarity S_max is greater than or equal to 0.85, the fuzzy match passes; otherwise, it is marked as "name verification failed".

[0082] The second verification step is a comprehensive confidence assessment. A weighted composite score C is calculated by combining the average OCR confidence score C_ocr with the confidence score C_llm generated by the large language model. C_ocr is obtained by taking the arithmetic mean of the confidence scores of each character output by the OCR model, while C_llm is obtained by taking the mean of the logarithmic probabilities of the tokens output by the large language model when generating the target field, after an exponential transformation. When C is less than 0.7, it is marked as "insufficient confidence".

[0083] Step S108: Output and verify the closed loop.

[0084] For results that pass dual verification, the system outputs the extracted generic name, trade name, and specification information of the drug to the finished product verification system in a standardized JSON format. An example format is: {"Generic Name":"Sodium Chloride Injection","Trade Name":"Serious Saline","Specification":"10ml:90mg"}. Simultaneously, the output results are associated with and stored in the results database along with the corresponding original cylinder unfolded image for easy traceability later.

[0085] For results that fail any verification, the system generates a marker for manual review, packaging and storing information such as the original cylinder unfolded image, OCR text sequence, original output of the large language model, and the reason for verification failure in an abnormal result directory. Simultaneously, it triggers a rejection mechanism to divert the corresponding abnormal medicine bottle from the main conveyor line to the manual review station. The manual review workstation periodically reads the abnormal result directory for pharmacists to manually confirm. Confirmed results can then be added to the fine-tuning training dataset of the large language model.

[0086] The system can be configured to automatically pause the production line and trigger an audible and visual alarm when multiple bottles (configurable, default 3 bottles) fail to pass verification, prompting operators to check the imaging system or the batch quality of the medicine bottles to prevent batch errors from flowing into subsequent processes.

[0087] Using the method described in this embodiment, the overall processing time for a single medicine bottle, from image acquisition to outputting structured drug name information, does not exceed 6 seconds. This includes approximately 2 seconds for line scanning imaging and stitching, approximately 1 second for OCR text detection and recognition, approximately 0.1 seconds for text sequence normalization, approximately 1 second for large language model inference, and approximately 0.5 seconds for verification and output. The remaining time is consumed by mechanical actions such as conveying and clamping. This processing efficiency meets the cycle time requirement of a pharmaceutical finished product verification and dispensing machine that can process more than 10 bottles per minute.

[0088] In summary, this invention organically integrates a line-scan imaging hardware architecture with a large language model for semantic parsing, providing a solution for recognizing text and extracting drug names from medicine bottle labels by eliminating distortion and reflection at the imaging source and achieving semantic-level understanding and error correction in the post-processing stage. This solution boasts significant advantages such as high recognition accuracy, strong robustness, high processing efficiency, and flexible deployment, making it suitable for automated finished product verification scenarios involving cylindrical medicine bottles, such as pharmacies, compounding centers, and pharmaceutical warehouses.

[0089] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A device for recognizing text and extracting drug names from medicine bottle labels based on line scanning imaging, characterized in that, include: A rotating stage configured to hold and rotate medicine bottles; A line scan camera, fixedly mounted and with its field of view covering the surface of the medicine bottle on the rotating stage, is configured to scan at a constant line frequency as the medicine bottle rotates to acquire line-by-line image data of the surface of the medicine bottle. The image stitching unit is coupled to the encoder of the line scan camera and the rotating stage, and is configured to stitch the line-by-line image data in real time according to the signal of the encoder to generate the cylindrical unfolded image of the medicine bottle label; The text recognition unit is configured to perform text detection and recognition on the cylindrical unfolded image and output a text sequence; as well as The semantic parsing unit is configured to use a pre-trained large language model to perform semantic-level parsing and error correction on the text sequence in order to extract structured drug name information.

2. The device for recognizing text and extracting drug names from medicine bottle labels based on line scanning imaging according to claim 1, characterized in that, It also includes a light source, which is a line light source, and its emitted light direction is configured with respect to the scanning direction of the line scan camera so that the specular reflected light is away from the receiving light path of the line scan camera, so as to suppress local reflections on the surface of the medicine bottle during the imaging process.

3. The device for recognizing text and extracting drug names from medicine bottle labels based on line scanning imaging according to claim 1, characterized in that, The character recognition unit includes: The label region positioning module is configured to automatically locate the effective text region of the label and generate a mask on the cylindrical unfolded image after eliminating surface distortion, based on the grayscale difference and texture difference between the medicine bottle label and the bottle background in the cylindrical unfolded image, using the region recognition rules of vertical projection analysis and / or edge gradient statistics. The OCR recognition module is configured to perform text recognition on the area defined by the mask and output the original text fragment containing the position coordinates.

4. The device for recognizing text and extracting drug names from medicine bottle labels based on line scanning imaging according to claim 3, characterized in that, The text recognition unit is further configured to sort and concatenate the original text fragments according to the position coordinates, in a top-to-bottom and left-to-right order, to form a complete and fully preserved text string, which serves as the text sequence input to the semantic parsing unit; the full preservation provides the large language model with a complete context without information loss.

5. The device for recognizing text and extracting drug names from medicine bottle labels based on line scanning imaging according to claim 1, characterized in that, The semantic parsing unit is configured to: input the text sequence and the preset drug naming standard into the large language model in a structured manner, and instruct the large language model to output the extracted generic name, trade name and corresponding specification information of the drug.

6. The device for recognizing text and extracting drug names from medicine bottle labels based on line scanning imaging according to claim 5, characterized in that, It also includes a verification unit coupled to the semantic parsing unit, configured to access a standard drug name database, and perform rule verification on the drug name information extracted by the semantic parsing unit based on the standard drug name database. The rule verification includes: performing fuzzy matching between the extracted generic drug name and the standard drug name database, triggering verification failure when the matching degree is lower than a preset threshold; marking the results of verification failure or parsing abnormality as pending manual review, and simultaneously associating and saving the results with the corresponding cylindrical unfolded image for iterative training of the large language model.

7. A method for text recognition and drug name extraction on medicine bottle labels based on line scanning imaging, applicable to the medicine bottle label text recognition and drug name extraction device based on line scanning imaging as described in any one of claims 1-6, characterized in that, include: The rotating stage is controlled to rotate the medicine bottle, and at the same time, a fixedly installed line scan camera scans the surface of the medicine bottle at a constant line frequency to acquire line-by-line image data. Based on the encoder signal of the rotating stage, the line-by-line image data is stitched together in real time to generate a cylindrical unfolded image of the medicine bottle label; The cylindrical unfolded image is subjected to text detection and recognition to obtain a text sequence; The text sequence is semantically parsed and corrected using a pre-trained large language model, and structured drug name information is extracted and output.

8. The method for text recognition and drug name extraction on medicine bottle labels based on line scan imaging according to claim 7, characterized in that, The step of acquiring line-by-line image data also includes: configuring a line light source and setting the illumination direction of the line light source and the scanning direction of the line scan camera to guide the specular reflection light from the surface of the medicine bottle outside the receiving field of view of the line scan camera.

9. A method for recognizing text and extracting drug names from medicine bottle labels based on line scanning imaging according to claim 7, characterized in that, The specific steps for text detection and recognition in the cylindrical unfolded image include: Based on the grayscale and texture differences between the medicine bottle label and the bottle background in the cylindrical unfolded image, and taking advantage of the characteristics of the cylindrical unfolded image being free of surface distortion and having uniform illumination, the effective text area of ​​the label is automatically located and a mask is generated by using the region recognition rules of vertical projection analysis and / or edge gradient statistics. The OCR text recognition model is invoked to recognize the masked area and output the original text fragment containing the position coordinates. Based on the position coordinates, the original text fragments are sorted and spliced ​​in order from top to bottom and from left to right to form a complete text string that retains all information, which serves as the text sequence and provides the large language model with a complete semantic context without information loss.

10. A method for text recognition and drug name extraction on medicine bottle labels based on line scan imaging according to claim 7, characterized in that, After extracting and outputting structured drug name information, the process also includes: The extracted drug name information is then subjected to fuzzy matching verification with the standard drug name database. For results that fail verification or whose parsing confidence is below a threshold, a label is generated for manual review, and the result is associated with and stored in relation to the corresponding cylindrical unfolded image for iterative optimization training of the large language model.

Images