Intelligent checking method, system and equipment for medicine outer package label information and medium

Abstract

The invention provides a medicine outer package label information intelligent checking method, system and device and a medium, and belongs to the technical field of information. The method comprises the following steps: collecting an original image of a medicine outer package through a handheld PDA terminal; performing label area positioning and geometric correction on the original image to obtain a regular label picture; inputting the regular label picture into a character recognition model, and outputting label text information; processing the label text information based on a large language model, and extracting key field information; verifying the key field information of each batch of medicines to generate standard label field information, constructing a template library containing the standard label field information of each batch of medicines, and binding and storing the standard label field information and the unique index code; and reading to-be-checked drug outer package label information, obtaining to-be-checked index codes and key fields in the to-be-checked drug outer package label information, calling corresponding standard label field information in the template library, comparing the standard label field information with the to-be-checked key fields, and outputting a comparison result.

Description

Technical Field

[0001] This invention belongs to the field of information technology, and more specifically relates to a method, system, device and medium for intelligent verification of information on drug outer packaging labels. Background Technology

[0002] In the raw material and auxiliary material receiving process at pharmaceutical plants, the consistency verification of key fields on the outer packaging labels is a core procedure for implementing the requirements of receiving management standards, and it is directly related to the quality and safety of drug production.

[0003] Currently, the mainstream verification method mainly relies on manual visual inspection. This inspection method has many shortcomings: it not only requires checking dozens or hundreds of packages in a single batch one by one, which is inefficient and difficult to adapt to the needs of large-scale warehousing, but also requires dedicated personnel and professional training, resulting in high labor costs; more importantly, manual operation is prone to fatigue and attention loss, leading to missed inspections and incorrect inspections, and the verification results are mostly unstructured storage, which cannot establish a field-level association of "packaging-batch-traceability", making it difficult to accurately trace the source when quality problems occur in the medicine later.

[0004] To improve this situation, some companies have tried to introduce OCR technology to assist in verification, but they have not yet broken through the core bottlenecks: On the one hand, there are significant differences in the labels of different suppliers, such as mixed text and images and curved surface printing. Existing OCR models need to be trained separately for each type of label, resulting in poor universality and high adaptation costs. On the other hand, the accuracy of model recognition is greatly affected by label wear, uneven ink, and environmental reflection. When such scenarios are not covered, the accuracy drops significantly, which increases the cost of secondary verification. In addition, when adding new suppliers or updating label types, a large number of samples need to be manually labeled and the model needs to be retrained. The single iteration cycle is too long and involves a large amount of manual labeling, sample collection, and computational resources for model training.

[0005] In summary, the existing manual and OCR verification technologies suffer from low efficiency, high cost, poor adaptability, and difficulty in traceability, which have become key factors restricting the intelligent upgrading of the warehousing process and the efficiency of quality traceability. Summary of the Invention

[0006] To address the above problems, the present invention aims to provide a method, system, device, and medium for intelligent verification of drug outer packaging label information. By collecting pre-processed images through a handheld terminal, locating and correcting labels using computer vision technology, realizing text recognition and field extraction using an AI model, constructing a standard label template library with a unique index, and combining it with a dedicated algorithm for comparison, the method significantly improves the efficiency and accuracy of label verification for drug raw materials and excipients entering the warehouse, while also strengthening quality traceability capabilities and reducing operating costs.

[0007] To achieve the above objectives, the present invention employs the following technical solution: In a first aspect, embodiments of this application provide a method for intelligent verification of drug outer packaging label information, including: The original images of the outer packaging of the medicine were captured using a handheld PDA terminal; The original image is subjected to label region localization and geometric correction to obtain a regularized label image; Input the regularized label image into the text recognition model and output the label text information; The tag text information is processed based on a large language model to extract key field information; The key field information of each batch of drugs is verified, standard label field information is generated, and a template library containing the standard label field information of each batch of drugs is constructed. The standard label field information is bound and stored with a unique index code. Read the label information of the outer packaging of the drug to be verified, obtain the index code and key fields to be verified, retrieve the corresponding standard label field information from the template library, compare it with the key fields to be verified, and output the comparison results.

[0008] In an optional implementation, the step of acquiring the original image of the drug packaging via a handheld PDA terminal includes: Activate the camera integrated into the handheld PDA terminal; Adjust the shooting focus and angle to ensure that the outer label of the medicine packaging is completely within the frame; Acquire color digital images that meet the preset resolution requirements; The acquired color digital images are denoised using a 3×3 kernel Gaussian filter algorithm to generate the original image of the drug packaging, which is then stored locally.

[0009] In an optional implementation, the original image is subjected to label region localization and geometric correction to obtain a regularized label image, including: The original image is input into the pre-trained ClipSeg target segmentation model, which outputs a binarized mask image. The edge contours of the binary mask image are extracted using the Canny edge detection algorithm. The label contours are then filtered out by area thresholding, and the set of label contour coordinates is output. The minimum bounding rectangle is calculated based on the set of label outline coordinates, and the coordinates of the four vertices are obtained. Based on the vertex coordinates of the minimum bounding rectangle, a perspective transformation algorithm is used to crop and correct the image from the original image so that the long side is parallel to the horizontal direction, generating a regular label image.

[0010] In an optional implementation, the regularized label image is input into a text recognition model, and the label text information is output, including: The regularized label image is input into the text recognition model, which includes a CNN module, an LSTM module, and a CTC module. The CNN module is used to extract the spatial visual features of the label image, the LSTM module is used to capture the contextual dependencies of the text sequence, and the CTC module is used to achieve non-strict alignment between the input feature sequence and the output character sequence. The text recognition model is trained using sample images of drug labels with different printed fonts, font sizes, and background complexities. The text recognition model first performs grayscale processing on the regular label image, then enhances the contrast between the text and the background using the Sauvola adaptive binarization algorithm, and then scans the text region line by line in the order from left to right and from top to bottom, extracts the feature sequence of each line of text and inputs it into the LSTM module, and obtains the character sequence by decoding through the CTC model. Output a continuous text string containing all the text content on the label based on the character sequence. The continuous text string preserves the original arrangement order of the text on the label and is stored as label text information in TXT format.

[0011] In an optional implementation, the processing of the tag text information based on a large language model to extract key field information includes: Construct prompt words, which include role settings, field extraction rules, and JSON format output examples; the field extraction rules include extraction rules for product name, production batch number, CDE registration number, execution standard, and production date; The input sequence is formed by combining the prompt words with the label text information; Submit the input sequence to the GPT-4 or ERNIE4.0 large language model and send an information extraction command; Receive and parse the response returned by the large language model, and extract the key field information output in JSON format.

[0012] In an optional implementation, the step of verifying the key field information of each batch of drugs, generating standard label field information, constructing a template library containing the standard label field information of each batch of drugs, and binding and storing the standard label field information with a unique index code includes: The API interface of the drug information traceability platform is called to verify the key field information of each batch of drugs. After removing abnormal data, the consistent results are taken as the standard label field information. Extract the production batch number and production date from the standard label field information, and generate a unique index code by combining it with a random character sequence; Standard label field information is stored in a local or cloud template library in JSON file format, and the corresponding unique index code is bound to the JSON file storage path as a unique retrieval identifier.

[0013] In an optional implementation, the step of reading the outer packaging label information of the drug to be verified, obtaining the index code and key fields to be verified therein, retrieving the corresponding standard label field information from the template library, comparing it with the key fields to be verified, and outputting the comparison result includes: Read the batch number and production date from the outer packaging label information of the drug to be verified, match them with the corresponding fragment of the unique index code in the template library, locate and retrieve the corresponding standard label field information; The Levenshtein distance algorithm is used to compare the key fields of the outer packaging label information of the drug to be checked with the standard label fields in the order of product name, production batch number, CDE registration number, implementation standard, and production date. If they match completely, they are considered to be consistent; otherwise, they are marked as inconsistent and the differences are recorded. The output includes a comparison table containing field names, values ​​to be checked, standard values, judgment results, and differences, generating a comparison result report in PDF format.

[0014] Secondly, embodiments of this application also provide an intelligent verification system for drug outer packaging label information, including: The image acquisition module is used to acquire original images of the outer packaging of medicines via a handheld PDA terminal; The label generation module is used to locate and geometrically correct the label region of the original image to obtain a regular label image; The text recognition module is used to input the regularized label image into the text recognition model and output the label text information; The key information extraction module is used to process the tag text information based on a large language model and extract key field information; The template library construction module is used to verify the key field information of each batch of drugs, generate standard label field information, and build a template library containing the standard label field information of each batch of drugs. The standard label field information is bound and stored with a unique index code. The intelligent comparison module is used to read the label information of the outer packaging of the drug to be verified, obtain the index code and key fields to be verified, retrieve the corresponding standard label field information in the template library, compare it with the key fields to be verified, and output the comparison results.

[0015] Thirdly, embodiments of this application also provide an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of the intelligent verification method for drug outer packaging label information as described in any of the above.

[0016] Fourthly, embodiments of this application also provide a storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the intelligent verification method for drug outer packaging label information as described in any of the above embodiments.

[0017] As can be seen from the above technical solutions, the present invention has the following advantages: The intelligent verification method for drug packaging label information provided in this application acquires images using a handheld PDA terminal and performs Gaussian filtering for noise reduction to ensure data source quality. It utilizes the ClipSeg model, Canny edge detection, and perspective transformation to achieve accurate label positioning and correction. A text recognition model with a CNN-LSTM-CTC architecture, trained on multiple scene samples, enhances text recognition adaptability and accuracy. Key fields are accurately extracted according to rules based on a large language model. A standard template library with a unique index is constructed using API verification from a drug information traceability platform. Finally, the Levenshtein algorithm is used for field-by-field comparison to generate a structured report. This method effectively solves the problems of low efficiency, error-proneness, and traceability difficulties associated with manual verification, as well as the poor adaptability and unstable accuracy of traditional OCR technology. It achieves precise field-level traceability and structured storage, significantly reducing labor and management costs in the raw material and excipient warehousing process of pharmaceutical plants, improving verification efficiency and quality traceability, and meeting GMP compliance requirements for the traceability of raw materials and excipients.

[0018] This application utilizes handheld PDAs to automatically acquire images, computer vision technology to quickly locate and correct labels, and AI models to automatically identify and extract fields, replacing manual visual inspection of each item. This significantly shortens the verification time for a single batch of raw and auxiliary materials and meets the high-efficiency processing needs of pharmaceutical factories in large-scale warehousing scenarios.

[0019] This application utilizes the ClipSeg segmentation model to accurately locate label regions, the CNN-LSTM-CTC architecture OCR model to ensure text recognition accuracy, and the large language model to extract key fields according to rules. Combined with the Levenshtein algorithm for field-by-field comparison, it effectively avoids the problems of missed detections and false detections caused by human fatigue and improves the accuracy of field consistency verification.

[0020] This application binds and stores standard label field information with batch information through a unique index code, forming a structured data chain of "packaging label - field information - batch traceability". Combined with a PDF comparison report, it enables rapid field-level traceability when drug quality problems occur, meeting GMP compliance requirements for the traceability of raw materials and excipients.

[0021] This application uses a large language model to process field extraction for different format labels, eliminating the need to retrain the model for supplier-differentiated labels; ClipSeg segmentation and perspective transformation technology adapts to various packaging formats such as curved surface printing and mixed text and graphics, solving the problems of poor format adaptability and high iteration costs of traditional OCR technology.

[0022] This application can effectively reduce the cost of staffing and training dedicated verification personnel and avoid the cost of secondary verification by manual verification; the template library and automated process reduce the technical adaptation cost when adding new suppliers or updating labels, and reduce the overall manpower and management input of pharmaceutical companies in the raw material and excipient warehousing process. Attached Figure Description

[0023] To more clearly illustrate the technical solution of the present invention, the accompanying drawings used in the description will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0024] Figure 1 A flowchart illustrating the intelligent verification method for drug outer packaging label information provided in this application.

[0025] Figure 2 A schematic diagram of the intelligent verification system for drug outer packaging label information provided in this application.

[0026] Figure 3 A schematic diagram of the structure of the electronic device provided in this application. Detailed Implementation

[0027] The various embodiments of this disclosure will be described more fully in the following detailed description of the specific steps of the intelligent verification method for drug outer packaging label information. This disclosure may have various embodiments, and adjustments and changes may be made therein. However, it should be understood that there is no intention to limit the various embodiments of this disclosure to the specific embodiments disclosed herein, but rather this disclosure should be understood to cover all adjustments, equivalents, and / or alternatives falling within the spirit and scope of the various embodiments of this disclosure.

[0028] In the following, the terms “comprising” or “may include”, which may be used in various embodiments of this disclosure, indicate the presence of the disclosed functions, operations, or elements, and do not limit the addition of one or more functions, operations, or elements. Furthermore, as used in various embodiments of this disclosure, the terms “comprising,” “having,” and their cognates are intended only to indicate a particular feature, number, step, operation, element, component, or combination of the foregoing, and should not be construed as primarily excluding the presence of one or more other features, numbers, steps, operations, elements, components, or combinations of the foregoing, or the possibility of adding one or more combinations of the foregoing.

[0029] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0030] Please see Figure 1 The diagram shows a flowchart of a method for intelligent verification of drug outer packaging label information in a specific embodiment. The method includes: S1: Collect original images of the outer packaging of medicines using a handheld PDA terminal.

[0031] In this specific implementation, this step acquires high-quality images of the drug's outer packaging using a handheld PDA terminal, providing a reliable data source for subsequent label processing. The specific implementation process is as follows: First, activate the integrated camera of the handheld PDA terminal and enter the image acquisition mode. The terminal must be equipped with a high-definition camera of 5 megapixels or above, a dimmable LED fill light module, and an operating system version of no less than Android 10.0 to meet the hardware and software requirements for image acquisition.

[0032] Then, point the PDA terminal at the label area of ​​the outer packaging of the medicine (carton or cardboard drum). The built-in program of the terminal will display the view in real time and help adjust the shooting focus and angle to ensure that the label is completely in the view and that the label area accounts for no less than 60% of the total image area.

[0033] When the above framing conditions are met, the terminal automatically triggers the shooting command to capture a color digital image with a resolution of 1920×1080 pixels, which can balance image clarity and storage efficiency.

[0034] At this point, the acquired color image is immediately subjected to a 3×3 kernel Gaussian filter algorithm for grayscale noise reduction. This algorithm removes noise caused by ambient light interference, ensuring that the signal-to-noise ratio of the processed image reaches 30dB or higher, thus guaranteeing image quality.

[0035] Finally, the noise-reduced image is used as the original image, named according to the rule of "acquisition time (format: yyyyMMddHHmmss) + device ID", for example, "20241130153022_PDA001", and stored in the " / Image / Raw / " directory on the local PDA terminal for easy access and management later.

[0036] S2: Perform label region localization and geometric correction on the original image to obtain a regular label image.

[0037] In this specific implementation, this step accurately separates the label region from the original image and eliminates tilt and distortion issues, ultimately outputting a well-formed label image. The specific implementation process is as follows: First, the original image generated in step S1 is input into the pre-trained ClipSeg target segmentation model. This model has been fine-tuned using 100,000 different drug label images. The input size is uniformly scaled to 512×512 pixels. After processing, the model outputs a binary mask image, where the pixel value of the label area is set to 255 and the pixel value of the non-label area is set to 0, thus achieving the initial separation of the label and the background.

[0038] Then, the Canny edge detection algorithm with a low threshold of 50 and a high threshold of 150 is used to process the binarized mask image to extract continuous and complete label edge contours. Simultaneously, a threshold filtering method with a contour area of ​​not less than 10,000 pixels² is applied to remove small-area noise contours, retaining only unique label contours. The final output is a set of contour coordinates {(x1,y1),(x2,y2),…,(x...} n ,y n )}, where x and y both represent the pixel coordinates of the image.

[0039] At this point, the minAreaRect() function of the OpenCV library is called to calculate the minimum bounding rectangle for the above set of label outline coordinates. During the process, the center coordinates (X0,Y0), width w, height h, and tilt angle θ (the tilt angle θ ranges from -90° to 0°) of the rectangle are obtained simultaneously, and the coordinates of the four vertices {(X1,Y1),(X2,Y2),(X3,Y3),(X4,Y5)} are output in clockwise order.

[0040] Finally, the vertex coordinates of the target rectangle (i.e., the regularized label image) are defined as {(0,0),(1280,0),(1280,720),(0,720)}. This size setting is based on the aspect ratio of the original label to ensure information integrity. A perspective transformation matrix is ​​constructed based on the vertices of the original rectangle and the target rectangle. The label area is cropped and tilted using this matrix, and finally a regularized label image with its long side parallel to the horizontal direction is output.

[0041] S3: Input the regularized label image into the text recognition model and output the label text information.

[0042] In a specific implementation, this step converts the visual text information in the label image into processable text information. This is achieved primarily through a dedicated text recognition model, and the specific implementation process is as follows: First, the regularized label image generated in step S2 is input into the text recognition model. This model adopts a "convolutional neural network (CNN) + long short-term memory network (LSTM) + connection-temporal classification (CTC)" architecture. The CNN module is responsible for extracting the spatial visual features of the label image, the LSTM module can capture the contextual dependencies of the text sequence, and the CTC module is used to solve the problem of non-strict alignment between the input feature sequence and the output character sequence to ensure recognition continuity.

[0043] It should be noted that the text recognition model has been trained on at least 100,000 sample images of drug labels. The training samples cover more than 8 common printing fonts, including Song, Hei, and Kai, with font sizes ranging from 6 to 24. They also include more than 5 background scenarios, such as black text on a white background, black text on a yellow background, and mixed text and graphics on a light background. In addition, actual database scenario samples such as slightly worn labels, uneven ink, and local reflections are included to adapt to complex application environments. After training, the model achieved a character recognition accuracy of 99% or higher on the test set.

[0044] Then, the model processes the input regularized label images according to a fixed process: first, the color image is converted into a single-channel grayscale image to simplify the calculation; then, the contrast between the text and the background is enhanced by the Sauvola adaptive binarization algorithm; then, the text region is scanned line by line in the order from left to right and from top to bottom, the feature sequence of each line of text is extracted and input into the LSTM module; finally, the feature sequence is converted into a recognizable character sequence by the CTC decoding algorithm.

[0045] Finally, the model outputs a continuous text string containing all the text content on the label, including Chinese characters, letters, numbers, punctuation marks, etc., and strictly preserves the original arrangement order of the text on the label. This text string is stored in TXT format, with the filename being consistent with the corresponding regular label image except for the suffix, and serves as the label text information for subsequent field extraction.

[0046] S4: Process the tag text information based on the large language model and extract key field information.

[0047] In this specific implementation, this step relies on the semantic understanding capabilities of a large language model to accurately extract structured key fields from the tag text information. The specific implementation process is as follows: First, construct the prompt words, which contain three core parts: First, role setting, clarifying that the model is a "drug label information processing expert" and needs to extract fields according to professional standards; second, field extraction rules, including extraction rules for product name, production batch number, CDE registration number, implementation standard, and production date. Among them, the product name must be the complete chemical name or generic name; the production batch number must retain the original character format (e.g., "VX4144P") and the content after "batch number:" must be completely extracted; the CDE registration number must conform to the 11-digit format of Y / B + 4-digit year + 7-digit number; the execution standard must be the first consecutive string without spaces after "execution standard" (e.g., "GB / T2023"); the production date must be converted to the format "YYYY-MM-DD"; missing fields should be output as empty strings "" and should not be fabricated; thirdly, the output example is specified, the output format is JSON, the example is {"product name":"hydroxyurea","CDE registration number":"Y20190001536","production date":"2024-11-30","production batch number":"VX4144P","execution standard":"CP2020"}.

[0048] Then, the prompt words constructed above, the label text information output in step S3, and the "start extracting information" instruction are combined into a complete input sequence to ensure that the model clearly understands the task objective and processing rules.

[0049] At this point, the input sequence is submitted to the GPT-4 or ERNIE4.0 large language model. The model's semantic analysis and information extraction capabilities are used to complete the field processing. After processing, the model returns response content containing key fields.

[0050] Finally, the response content returned by the model is parsed, and the key field information output in JSON format is extracted. The fields include product name, production batch number, CDE registration number, implementation standard, and production date, which lays the foundation for subsequent standard verification and comparison.

[0051] S5: Verify the key field information of each batch of drugs, generate standard label field information, and build a template library containing the standard label field information of each batch of drugs. Bind and store the standard label field information with a unique index code.

[0052] In this specific implementation, this step establishes standardized label information for each batch of drugs and builds a template library with a unique index to achieve traceability management. The specific implementation process is as follows: First, from the same batch of pharmaceutical raw materials and excipients, at least three packaging samples with clear, undamaged, and unobstructed labels are manually selected to ensure the integrity and representativeness of the sample information. Then, the complete process of steps S1 to S4 is performed on each selected sample to obtain the key field information of each sample, providing multiple sets of reference data for subsequent standard verification.

[0053] At this point, the key field information is verified in two ways: first, by calling the API interface of the drug information traceability platform of the relevant agency of the State Drug Administration and verifying the accuracy of the fields with the help of the official database; second, by manually comparing the drug production batch records, eliminating abnormal data after double verification, and taking the consistent results of multiple sets of samples as the standard label field information for the batch.

[0054] Furthermore, a unique index code is generated based on the rule of "batch number + production date + 6-digit random sequence", such as "B20241130-20241130-7A3F9D". This code serves as the unique identifier of the standard information of the batch, ensuring the uniqueness of the retrieval.

[0055] Finally, the standard label field information is stored in JSON file format. The storage location can be a local server or a cloud database, which can be collectively referred to as the template library. At the same time, the unique index code is bound to the storage path of the JSON file, making the index code the core retrieval identifier of the template library, which facilitates quick retrieval later.

[0056] S6: Read the outer packaging label information of the drug to be verified, obtain the index code and key fields to be verified, retrieve the corresponding standard label field information from the template library, compare it with the key fields to be verified, and output the comparison result.

[0057] In this specific implementation, this step achieves automated and accurate comparison between the label to be verified and the standard label, and generates a structured report. The specific implementation process is as follows: For the outer packaging of the medicine to be checked, complete the image acquisition, label correction, text recognition and field extraction according to the process of steps S1 to S4 to obtain the key field information to be checked.

[0058] Then, input the batch number and production date of the medicine to be verified. After parsing the input information, match it with the "batch number + production date" fragment in the template library to locate the standard label field information of the corresponding batch and complete the retrieval.

[0059] The Levenshtein distance algorithm is used to perform field comparisons. The comparison order is fixed as "product name, production batch number, CDE registration number, implementation standard, and production date". The comparison standards for each field are clear: the product name must be completely identical in characters; the production batch number must match the original format including spaces and special symbols; the CDE registration number must meet the requirements in both format and characters; the implementation standard must be completely identical without spaces; and the production date must be identical in both format and characters of "YYYY-MM-DD". If the standards are fully met, it is judged as "identical"; otherwise, it is marked as "inconsistent" and the difference between the value to be checked and the standard value is recorded.

[0060] After the comparison is completed, two types of results are output: one is a comparison table, which includes columns for "field name, value to be checked, standard value, judgment result, and difference content", which intuitively displays the comparison status of each field; the other is a PDF report, which integrates the batch information of the drug to be checked, the unique index code, the comparison time and the comparison table, and supports local storage or uploading to the pharmaceutical factory's back-end management system for easy subsequent traceability and management.

[0061] In this embodiment, images are collected and preprocessed using a handheld PDA terminal to ensure data source quality. ClipSeg target segmentation, Canny edge detection, and perspective transformation are used to achieve accurate label positioning and correction. A CNN-LSTM-CTC architecture text recognition model is used to adapt to diverse label scenarios and improve recognition accuracy. Key fields are accurately extracted according to rules based on a large language model. A standard label template library with a unique index is constructed using API verification from the drug information traceability platform. Then, the Levenshtein distance algorithm is used to complete precise field-level comparison and generate a structured report. This effectively solves the problems of low efficiency, error-proneness, and high labor costs associated with traditional manual verification, as well as the poor adaptability and insufficient recognition stability of traditional OCR technology for different label formats. Simultaneously, a structured association of "label information, batch data, and traceability chain" is constructed, significantly improving the efficiency and accuracy of label verification for raw materials and excipients entering the pharmaceutical factory's warehouse. This strengthens field-level traceability capabilities when drug quality problems occur, reduces operational management costs, and meets the GMP compliance requirements for the traceability of raw materials and excipients in the pharmaceutical industry.

[0062] like Figure 2 As shown, the following are embodiments of the intelligent verification system for drug outer packaging label information provided in this disclosure. This system and the intelligent verification method for drug outer packaging label information in the above embodiments belong to the same inventive concept. For details not described in detail in the embodiments of the intelligent verification system for drug outer packaging label information, please refer to the embodiments of the intelligent verification method for drug outer packaging label information described above.

[0063] A smart verification system for drug outer packaging label information includes: The image acquisition module is used to acquire original images of the outer packaging of medicines via a handheld PDA terminal; The label generation module is used to locate and geometrically correct the label region of the original image to obtain a regular label image.

[0064] The text recognition module is used to input the regularized label image into the text recognition model and output the label text information.

[0065] The key information extraction module is used to process the tag text information based on a large language model and extract key field information.

[0066] The template library construction module is used to verify the key field information of each batch of drugs, generate standard label field information, and build a template library containing the standard label field information of each batch of drugs. The standard label field information is then bound and stored with a unique index code.

[0067] The intelligent comparison module is used to read the label information of the outer packaging of the drug to be verified, obtain the index code and key fields to be verified, retrieve the corresponding standard label field information in the template library, compare it with the key fields to be verified, and output the comparison results.

[0068] The intelligent verification system for drug packaging label information provided in this embodiment collects and preprocesses images through a handheld PDA terminal, combines computer vision technology to achieve accurate label positioning and correction, uses an AI model to complete text recognition and key field extraction, relies on the drug information traceability platform to verify and build a standard label template library with a unique index, and finally completes field comparison through a dedicated algorithm. It effectively solves the problems of low efficiency, high cost, error-proneness and traceability difficulties of manual verification, and overcomes the shortcomings of traditional OCR technology such as poor adaptability to label format, unstable recognition accuracy and high iteration cost. It significantly improves the efficiency and accuracy of drug raw material and excipient label verification, strengthens quality traceability capabilities, reduces operating costs, and meets the relevant compliance requirements of the pharmaceutical industry.

[0069] Figure 3 A schematic diagram of the hardware structure of an electronic device for implementing various embodiments of the present invention.

[0070] The intelligent verification method for drug packaging label information provided in this application can be applied to electronic devices. Those skilled in the art will understand that the electronic device structure involved in the embodiments of this invention does not constitute a limitation on the electronic device. An electronic device may include more or fewer components than illustrated, or combine certain components, or have different component arrangements. In the embodiments of this invention, the electronic device includes, but is not limited to, laptop computers, desktop computers, workbenches, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the embodiments of this application described and / or claimed herein.

[0071] Electronic devices may include processors, external memory interfaces, internal memory, universal serial bus (USB) interfaces, charging management modules, power management modules, batteries, wireless communication modules, audio modules, speakers, microphones, sensor modules, buttons, cameras, displays, and SIM card interfaces, etc.

[0072] A processor may include one or more processing units, such as: a central processing unit (CPU), an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, memory, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural network processing unit (NPU). Different processing units may be independent devices or integrated into one or more processors.

[0073] The processor can serve as the nerve center and command center of an electronic device. The controller can generate operation control signals based on the instruction opcode and timing signals to control the fetching and execution of instructions.

[0074] The processor may also include memory for storing instructions and data. In some embodiments, the memory in the processor is a cache memory. This memory can store instructions or data that the processor has just used or that are used repeatedly. If the processor needs to use the instruction or data again, it can retrieve it directly from this memory. This avoids repeated accesses, reduces processor latency, and thus improves system efficiency.

[0075] An external storage interface (ESI) can be used to connect external memory cards, such as microSD cards, to expand the storage capacity of electronic devices. The external memory card communicates with the processor through the ESI to perform data storage functions, such as saving music and video files on the external memory card.

[0076] Internal memory can be used to store computer executable program code, which includes instructions. The processor executes various functional applications and data processing of electronic devices by running the instructions stored in internal memory. Internal memory can include a program storage area and a data storage area. Internal memory can include high-speed random access memory, and can also include non-volatile memory, such as at least one disk storage device, flash memory device, universal flash storage (UFS), etc.

[0077] Wireless communication functionality in electronic devices can be achieved through antennas, wireless communication modules, modem processors, and baseband processors.

[0078] Wireless communication modules can provide solutions for wireless communication applications in electronic devices, including wireless local area networks (WLANs) (such as wireless fidelity (Wi-Fi) networks), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), and infrared (IR) technologies.

[0079] Electronic devices can implement audio functions through audio modules, speakers, receivers, microphones, headphone jacks, and application processors.

[0080] Electronic devices can achieve shooting functions through ISPs, cameras, video codecs, GPUs, displays, and application processors.

[0081] Electronic devices can achieve display functions through GPUs, displays, and application processors.

[0082] A GPU is a microprocessor for image processing, connected to the display screen and application processor. GPUs are used to perform mathematical and geometric calculations for graphics rendering. A processor may include one or more GPUs, which execute program instructions to generate or modify display information.

[0083] A display screen is used to display images, videos, etc. A display screen includes a display panel.

[0084] The aforementioned electronic device realizes the intelligent verification method for drug outer packaging label information of this application by acquiring and preprocessing images through a handheld PDA, locating and correcting labels by combining computer vision technology, recognizing text and extracting key fields using an AI model, verifying and constructing a standard template library with a unique index based on the drug information traceability platform, and then comparing them through a dedicated algorithm. This effectively solves the problems of low efficiency and error-proneness of manual verification and poor format adaptation of traditional OCR technology, and achieves the beneficial effects of improving the efficiency and accuracy of drug raw material and auxiliary material warehousing label verification, strengthening quality traceability capabilities, reducing operating costs, and meeting compliance requirements.

[0085] The storage medium provided in this application stores a program product capable of implementing an intelligent verification method for drug outer packaging label information.

[0086] The intelligent verification methods for drug outer packaging label information include: The original images of the outer packaging of the medicine were captured using a handheld PDA terminal; The original image is subjected to label region localization and geometric correction to obtain a regularized label image; Input the regularized label image into the text recognition model and output the label text information; The tag text information is processed based on a large language model to extract key field information; The key field information of each batch of drugs is verified, standard label field information is generated, and a template library containing the standard label field information of each batch of drugs is constructed. The standard label field information is bound and stored with a unique index code. Read the label information of the outer packaging of the drug to be verified, obtain the index code and key fields to be verified, retrieve the corresponding standard label field information from the template library, compare it with the key fields to be verified, and output the comparison results.

[0087] In some possible implementations, the intelligent verification method for drug outer packaging label information disclosed herein can be implemented as a program product, which includes program code. When the program product is run on a terminal device, the program code is used to cause the terminal device to perform the steps described in the "Exemplary Methods" section above according to various exemplary embodiments of this disclosure.

[0088] The storage medium disclosed herein may be any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples (a non-exhaustive list) of readable storage media include: an electrical connection having one or more wires, a portable disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof.

[0089] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for intelligent verification of drug outer packaging label information, characterized in that, include: The original images of the outer packaging of the medicine were captured using a handheld PDA terminal; The original image is subjected to label region localization and geometric correction to obtain a regularized label image; Input the regularized label image into the text recognition model and output the label text information; The tag text information is processed based on a large language model to extract key field information; The key field information of each batch of drugs is verified, standard label field information is generated, and a template library containing the standard label field information of each batch of drugs is constructed. The standard label field information is bound and stored with a unique index code. Read the label information of the outer packaging of the drug to be verified, obtain the index code and key fields to be verified, retrieve the corresponding standard label field information from the template library, compare it with the key fields to be verified, and output the comparison results.

2. The intelligent verification method for drug outer packaging label information according to claim 1, characterized in that, The process of acquiring original images of the drug's outer packaging via a handheld PDA terminal includes: Activate the camera integrated into the handheld PDA terminal; Adjust the shooting focus and angle to ensure that the outer label of the medicine packaging is completely within the frame; Acquire color digital images that meet the preset resolution requirements; The acquired color digital images are denoised using a 3×3 kernel Gaussian filter algorithm to generate the original image of the drug packaging, which is then stored locally.

3. The intelligent verification method for drug outer packaging label information according to claim 2, characterized in that, The original image is subjected to label region localization and geometric correction to obtain a regularized label image, including: The original image is input into the pre-trained ClipSeg target segmentation model, which outputs a binarized mask image. The edge contours of the binary mask image are extracted using the Canny edge detection algorithm. The label contours are then filtered out by area thresholding, and the set of label contour coordinates is output. The minimum bounding rectangle is calculated based on the set of label outline coordinates, and the coordinates of the four vertices are obtained. Based on the vertex coordinates of the minimum bounding rectangle, a perspective transformation algorithm is used to crop and correct the image from the original image so that the long side is parallel to the horizontal direction, generating a regular label image.

4. The intelligent verification method for drug outer packaging label information according to claim 3, characterized in that, The regularized label image is input into the text recognition model, and the label text information is output, including: The regularized label image is input into the text recognition model, which includes a CNN module, an LSTM module, and a CTC module. The CNN module is used to extract the spatial visual features of the label image, the LSTM module is used to capture the contextual dependencies of the text sequence, and the CTC module is used to achieve non-strict alignment between the input feature sequence and the output character sequence. The text recognition model is trained using sample images of drug labels with different printed fonts, font sizes, and background complexities. The text recognition model first performs grayscale processing on the regular label image, then enhances the contrast between the text and the background using the Sauvola adaptive binarization algorithm, and then scans the text region line by line in the order from left to right and from top to bottom, extracts the feature sequence of each line of text and inputs it into the LSTM module, and obtains the character sequence by decoding through the CTC model. Output a continuous text string containing all the text content on the label based on the character sequence. The continuous text string preserves the original arrangement order of the text on the label and is stored as label text information in TXT format.

5. The intelligent verification method for drug outer packaging label information according to claim 4, characterized in that, The process of processing the tag text information based on the large language model to extract key field information includes: Construct prompt words, which include role settings, field extraction rules, and JSON format output examples; the field extraction rules include extraction rules for product name, production batch number, CDE registration number, execution standard, and production date; The input sequence is formed by combining the prompt words with the label text information; Submit the input sequence to the GPT-4 or ERNIE4.0 large language model and send an information extraction command; Receive and parse the response returned by the large language model, and extract the key field information output in JSON format.

6. The intelligent verification method for drug outer packaging label information according to claim 5, characterized in that, The process of validating key field information for each batch of drugs, generating standard label field information, constructing a template library containing standard label field information for each batch of drugs, and binding and storing the standard label field information with a unique index code includes: The API interface of the drug information traceability platform is called to verify the key field information of each batch of drugs. After removing abnormal data, the consistent results are taken as the standard label field information. Extract the production batch number and production date from the standard label field information, and generate a unique index code by combining it with a random character sequence; Standard label field information is stored in a local or cloud template library in JSON file format, and the corresponding unique index code is bound to the JSON file storage path as a unique retrieval identifier.

7. The intelligent verification method for drug outer packaging label information according to claim 6, characterized in that, The process involves reading the outer packaging label information of the drug to be verified, obtaining the index code and key fields to be verified, retrieving the corresponding standard label field information from the template library, comparing it with the key fields to be verified, and outputting the comparison results, including: Read the batch number and production date from the outer packaging label information of the drug to be verified, match them with the corresponding fragment of the unique index code in the template library, locate and retrieve the corresponding standard label field information; The Levenshtein distance algorithm is used to compare the key fields of the outer packaging label information of the drug to be checked with the standard label fields in the order of product name, production batch number, CDE registration number, implementation standard, and production date. If they match completely, they are considered to be consistent; otherwise, they are marked as inconsistent and the differences are recorded. The output includes a comparison table containing field names, values ​​to be checked, standard values, judgment results, and differences, generating a comparison result report in PDF format.

8. A smart verification system for drug outer packaging label information, characterized in that, The system employs the intelligent verification method for drug outer packaging label information as described in any one of claims 1 to 7; The system includes: The image acquisition module is used to acquire original images of the outer packaging of medicines via a handheld PDA terminal; The label generation module is used to locate and geometrically correct the label region of the original image to obtain a regular label image; The text recognition module is used to input the regularized label image into the text recognition model and output the label text information; The key information extraction module is used to process the tag text information based on a large language model and extract key field information; The template library construction module is used to verify the key field information of each batch of drugs, generate standard label field information, and build a template library containing the standard label field information of each batch of drugs. The standard label field information is bound and stored with a unique index code. The intelligent comparison module is used to read the label information of the outer packaging of the drug to be verified, obtain the index code and key fields to be verified, retrieve the corresponding standard label field information in the template library, compare it with the key fields to be verified, and output the comparison results.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps of the intelligent verification method for drug outer packaging label information as described in any one of claims 1 to 7.

10. A storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps of the intelligent verification method for drug outer packaging label information as described in any one of claims 1 to 7.

Images