Drug image recognition method and system based on deep learning

Abstract

The application discloses a medicine image recognition method and system based on deep learning, and relates to the technical field of image recognition. The method comprises the following steps: obtaining a high-resolution image of a medicine tablet to be recognized, positioning a boundary area of a character engraved on the surface of the medicine tablet by using a pre-trained character area detection model, and extracting a gray image sequence; performing character-by-character segmentation and recognition on the gray image sequence to generate an initial character sequence, and meanwhile, extracting a global physical feature set of the medicine tablet; constructing a sequence error correction model, taking the global physical feature set as input, correcting the initial character sequence, and outputting a corrected character sequence; matching the corrected character sequence with a medicine tablet feature database, determining the type of the medicine tablet according to a matching similarity, and outputting a final recognition result, wherein a similarity threshold is dynamically adjusted according to a historical review rate of the current batch of medicine tablets. The application effectively improves the accuracy of medicine image recognition.

Description

Technical Field

[0001] This invention relates to the field of image recognition technology, and more specifically to a method and system for drug image recognition based on deep learning. Background Technology

[0002] With the development of medical information and automation technologies, hospital pharmacies have gradually introduced equipment such as automated repackaging machines for the routine management of tablet medications. Current technologies primarily rely on machine vision to perform overall image verification of the repackaged medication bags, or on manual comparison to process loose tablets detached from their original packaging for identification and sorting.

[0003] However, existing technologies have significant shortcomings: on the one hand, image verification methods based on fixed rules have a high false positive rate when dealing with issues such as worn characters on the tablet surface or uneven lighting, requiring frequent manual intervention; on the other hand, when medication orders need to be adjusted and specific medications need to be removed from the repackaged bags, the tablets have completely separated from the original packaging information, and the identification work relies entirely on the manual identification and picking by pharmacists or nurses, which is not only inefficient and increases the burden on personnel, but also poses a safety hazard of misidentification, resulting in insufficient accuracy in drug identification and directly affecting the patient's medication safety. Summary of the Invention

[0004] This invention provides a method and system for drug image recognition based on deep learning, aiming to solve the technical problem of insufficient accuracy in drug image recognition in the prior art.

[0005] In view of the above problems, the present invention provides a drug image recognition method and system based on deep learning.

[0006] In a first aspect, the present invention provides a drug image recognition method based on deep learning, comprising: Acquire high-resolution images of the tablets to be identified, locate the boundary regions of the characters engraved on the tablet surface using a pre-trained character region detection model, and extract grayscale image sequences. The grayscale image sequence is segmented and recognized character by character to generate an initial character sequence, while the global physical feature set of the tablet is extracted. Construct a sequence error correction model, and use the global physical feature set as input to correct the initial character sequence and output the corrected character sequence; The corrected character sequence is matched with the tablet feature database, the tablet category is determined based on the matching similarity, and the final recognition result is output. The similarity threshold is dynamically adjusted based on the historical verification rate of the current batch of tablets.

[0007] Secondly, the present invention provides a drug image recognition system based on deep learning, comprising: The image localization and recognition module is used to acquire high-resolution images of the tablets to be identified, locate the boundary regions of the characters engraved on the surface of the tablets through a pre-trained character region detection model, and extract grayscale image sequences. The character segmentation and recognition module is used to segment and recognize the grayscale image sequence character by character, generate an initial character sequence, and extract the global physical feature set of the tablet. The character sequence correction module is used to construct a sequence error correction model, and uses the global physical feature set as input to correct the initial character sequence and output the corrected character sequence. The feature matching and judgment module is used to match the corrected character sequence with the tablet feature database, determine the tablet category based on the matching similarity, and output the final recognition result. The similarity threshold is dynamically adjusted based on the historical verification rate of the current batch of tablets.

[0008] One or more technical solutions provided in this invention have at least the following technical effects or advantages: This invention provides a drug image recognition method and system based on deep learning. It extracts grayscale image sequences through image acquisition and character localization, accurately locates the imprinted character regions on drug tablets using a pre-trained model, and extracts ordered grayscale image sequences, providing a high-quality data foundation for subsequent recognition. Character segmentation and global feature extraction are performed simultaneously, generating an initial character sequence while acquiring global physical features such as shape and color, taking into account both character information and the overall attributes of the drug tablet. Sequence error correction based on global features uses an attention mechanism to accurately associate global attributes with the character sequence, effectively correcting initial recognition deviations and improving character sequence accuracy. Dynamic threshold matching, combined with historical review rates to adjust similarity standards, and an enhanced review mechanism, ensure the reliability of matching results and reduce false positive rates. This replaces traditional manual operations, effectively improving drug image recognition efficiency, reducing the burden on medical staff, and providing technical support for the precision and safety of clinical drug management. Attached Figure Description

[0009] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0010] Figure 1 A flowchart illustrating the deep learning-based drug image recognition method provided in an embodiment of the present invention; Figure 2 A schematic diagram of the structure of a deep learning-based drug image recognition system provided in an embodiment of the present invention; The components represented by each number in the attached diagram are explained below: Image localization and recognition module 11, character segmentation and recognition module 12, character sequence correction module 13, feature matching and judgment module 14. Detailed Implementation

[0011] This invention provides a method and system for drug image recognition based on deep learning, which is used to address the technical problem of insufficient accuracy in drug image recognition in the prior art.

[0012] Example 1, as Figure 1 As shown, this invention provides a drug image recognition method based on deep learning, the method comprising: S100: Acquire a high-resolution image of the tablet to be identified, locate the boundary region of the characters engraved on the tablet surface using a pre-trained character region detection model, and extract the grayscale image sequence.

[0013] In this embodiment of the invention, a high-resolution image of the tablet to be identified is acquired. A pre-trained character region detection model is used to locate the boundary regions of the characters engraved on the tablet surface, and a grayscale image sequence is extracted. The engraved characters on the tablet surface are accurate identifiers for the medicine, but actual tablet images suffer from numerous interference problems: background noise, such as reflections from the shooting platform or stains on the tablet surface, can obscure character details; scratches and spots on the tablet are easily misidentified as fake characters; and the characters are not fixed in position and are arranged randomly in the image. Directly performing character recognition on the entire image would lead to high recognition errors and low efficiency. Therefore, it is necessary to first accurately locate the real character regions and filter out invalid interference using a pre-trained model, and then standardize the character image to provide clear and consistent input for subsequent character recognition and error correction modules, ensuring the overall accuracy of drug identification.

[0014] Step S100 in the method provided in this embodiment of the invention includes: Specifically, a pre-trained character region detection model is used to locate the boundary regions of the characters engraved on the surface of the tablet and extract grayscale image sequences, including: The high-resolution image is input into a pre-trained character region detection model, and the output obtains the coordinates and confidence scores of multiple candidate boundary regions. The pre-training steps of the character region detection model include: Based on the drug image sample library, sample tablet images containing clearly engraved characters are collected to form a sample image set; For each sample tablet image in the sample image set, the coordinates of the boundary region of the engraved characters are marked and a confidence label is generated for each boundary region, forming a boundary region coordinate label set and a corresponding confidence label set; A network architecture for a character region detection model is constructed based on machine learning, wherein the output layer of the network architecture is configured with both coordinate output channels and confidence output channels. The character region detection model is trained under supervision using the sample image set, boundary region coordinate label set, and confidence label set. During the training process, the coordinate regression loss and confidence prediction loss are optimized simultaneously until the error between the predicted boundary region coordinates output by the model and the boundary region coordinate label set converges to a preset convergence threshold, and the matching degree between the confidence output and the confidence label set reaches a preset standard, thus completing the pre-training of the model. The candidate boundary regions are filtered according to a preset confidence threshold, and the candidate boundary regions with a confidence level greater than or equal to the confidence threshold are retained as valid boundary regions. Based on the coordinates of the effective boundary region, the corresponding image sub-blocks are extracted from the high-resolution image; Each of the image sub-blocks is converted to grayscale and sorted from left to right according to its spatial position in the original image to generate a grayscale image sequence.

[0015] First, acquire a high-resolution image of the tablet to be identified. A high-resolution image refers to an image with a pixel density ≥ 5472×3648, which can clearly present details such as character strokes and outlines, avoiding blurry characters due to insufficient pixels. Using a 20-megapixel industrial CMOS camera, under fixed conditions, such as a light source brightness of 500 lux, a shooting distance of 30cm, and no obstructions, capture a frontal image of the tablet to be identified, ensuring that the engraved characters are fully visible in the image, and finally output a standard image of 5472×3648 pixels. For example, using a 20-megapixel industrial camera to capture the front of an aspirin tablet, obtain a 5472×3648 pixel image, ensuring that the edges of each stroke of the character "ASPIRIN" in the image are sharp, without reflections or stains, and that the character details are fully discernible.

[0016] Secondly, the pre-training steps of the character region detection model include: First, based on a drug image sample library, sample tablet images containing clearly engraved characters are collected to form a sample image set. The sample image set is a collection of tablet images containing a large number of clearly engraved characters and corresponding drug information, used by the model to learn the relationship between the appearance and position of drug characters. All images are uniformly 5472×3648 pixels. For example, 1000 common tablets are selected from the drug image sample library, with 10 images of each type taken, including scenes with normal lighting and a slight tilt of 5°, ensuring that the characters are not blurred or obstructed, ultimately forming a sample set of 10,000 5472×3648 pixel images.

[0017] Secondly, for each sample tablet image in the sample image set, the coordinates of the boundary regions of the engraved characters are labeled, and a confidence label is generated for each boundary region, forming a boundary region coordinate label set and a corresponding confidence label set. The boundary region coordinates refer to the character position represented by a rectangle (x1, y1, x2, y2), where (x1, y1) are the top-left pixel coordinates, and (x2, y2) are the bottom-right pixel coordinates, based on the pixel dimensions of a 5472×3648 pixel image. The confidence label is a value between 0 and 1, representing the probability that the boundary region is a real character; the closer to 1, the higher the confidence.

[0018] For example, a professional pharmacist uses the LabelImg tool to label each sample image: a rectangle is used to completely select the character group. For instance, for an image of an aspirin tablet engraved with "ASPIRIN", the coordinates of the boundary regions of each of the seven individual characters are labeled. If the characters are clear and unobstructed, the labels are 0.9-1.0, forming a set of coordinate labels and a set of confidence labels. For example, A: (1200,800,1350,950), S: (1360,800,1510,950), P: (1520,800,1670,950), I: (1680,800,1830,950), R: (1840,800,1990,950), I: (2000,800,2150,950), N: (2160,800,2310,950). The confidence label for each individual character region is set to 1. Similarly, the coordinates of the boundary regions corresponding to the engraved characters and the confidence labels are labeled for the remaining sample tablet images, ultimately resulting in a set of coordinate labels and a set of confidence labels for 10,000 sample tablet images.

[0019] Furthermore, a network architecture for a character region detection model is constructed based on machine learning. The output layer of this network architecture is configured with both a coordinate output channel and a confidence output channel. The network architecture comprises an input layer, a hidden layer, and an output layer. The input layer receives a 5472×3648 pixel image of a pill, the hidden layer extracts image features, and the output layer includes a coordinate output channel and a confidence output channel. The coordinate output channel is a module within the output layer specifically designed to predict the coordinates of character regions, with the coordinate results adapted to the 5472×3648 pixel image dimension. The confidence output channel is a module within the output layer specifically designed to predict the confidence level of the regions.

[0020] For example, a character region detection model is built based on the YOLOv5 lightweight network architecture. The specific network structure design is as follows: The input layer is set to an adaptation size of 640×640 pixels. For the original 5472×3648 pixel pill image, the model will first scale it proportionally to 640×640 pixels before feature extraction, and the subsequent output results can be back-mapped to the pixel dimension of the original 5472×3648 pixel image; 12 convolutional layers are deployed in the hidden layer to extract the core features such as the edges and contours of the characters in the pill image layer by layer; The output layer is configured with two types of output channels: the coordinate output channel is used to output pixel coordinate values, which accurately correspond to the position of the character region in the original 5472×3648 pixel image, and the confidence output channel is used to output a confidence value in the range of 0-1, which represents the probability that the region is a real character region.

[0021] Furthermore, the character region detection model is trained under supervised supervision using the sample image set, boundary region coordinate label set, and confidence label set. During training, both coordinate regression loss and confidence prediction loss are optimized simultaneously until the error between the predicted boundary region coordinates output by the model and the boundary region coordinate label set converges to a preset convergence threshold, and the matching degree between the confidence output and the confidence label set reaches a preset standard, thus completing the model's pre-training. Supervised training refers to using labeled samples to allow the model to learn the mapping relationship between input and output. The coordinate regression loss measures the error between the predicted coordinates and the labeled coordinates; the smaller the error, the more accurate the coordinates. The confidence prediction loss measures the error between the predicted confidence and the labeled confidence; the smaller the error, the better the match. The convergence threshold is a preset upper limit of error, such as 0.001; below this value, the model is considered stable.

[0022] For example, the 10,000 sample sets are divided into 8:2 sets of 8,000 training sets and 2,000 validation sets to supervise the training of the character region detection model: a sample tablet image of 5472×3648 pixels is input, the character region detection model automatically scales and outputs the prediction result; two types of loss are calculated; the model parameters are adjusted in reverse to reduce the loss; training is completed when the coordinate regression loss is ≤0.001 and the confidence prediction loss is ≤0.05. For example, during initial training, the character region detection model predicted the coordinates of the aspirin tablet "A" as (1150, 780, 2450, 1020) with a confidence score of 0.83. The coordinate regression loss and confidence prediction loss were 0.01 and 0.12, respectively. After 30 rounds of training, the predicted coordinates were (1198, 802, 2397, 998) with a confidence score of 0.95. The coordinate regression loss was 0.0008 ≤ 0.001, and the confidence prediction loss was 0.04 ≤ 0.05. Both types of losses met the requirements, and the training was completed.

[0023] Based on this, the high-resolution image is input into a pre-trained character region detection model, which outputs the coordinates and confidence scores of multiple candidate boundary regions. The high-resolution image of the pill to be identified is input into the trained character region detection model; the model first scales the image proportionally to 640×640 pixels, then traverses all regions of the image using an 80×80 grid and 6 anchor boxes, extracting character features and calculating similarity; finally, it outputs 10 candidate regions that satisfy a feature similarity > 0.5. Each region contains coordinates adapted to the original 5472×3648 pixel image, and a confidence value in the 0-1 range. The number of candidate regions is determined by the actual number of suspected character feature regions in the image, rather than being a fixed setting.

[0024] For example, after inputting an aspirin tablet image, the character region detection model outputs 10 candidate regions, of which 7 are single-character regions of “A, S, P, I, R, I, N” with a confidence level of 0.95-0.98, and 3 are interference regions, such as tablet scratches: (3000, 1500, 3200, 1600) with a confidence level of 0.72; background stains: (800, 2000, 950, 2100) with a confidence level of 0.65; and reflective points: (4000, 2500, 4150, 2600) with a confidence level of 0.68.

[0025] Subsequently, the multiple candidate boundary regions are filtered according to a preset confidence threshold, retaining those with a confidence level greater than or equal to the threshold as valid boundary regions. The confidence threshold is used to eliminate pseudo-character regions with low confidence and retain genuine character regions. A preset confidence threshold of 0.8 is used to iterate through all candidate regions, retaining those with a confidence level ≥ 0.8 and eliminating those with a confidence level < 0.8. For example, when filtering 10 candidate regions one by one: 7 single-character candidate regions are retained as valid regions, and 3 interfering regions are eliminated.

[0026] Then, based on the coordinates of the effective boundary region, corresponding image sub-blocks are extracted from the high-resolution image. An image sub-block is a local image extracted from the original image, containing only the effective character region. It focuses on character details, reduces background interference, and maintains the same pixel density as the original image. Using OpenCV, the corresponding local region is accurately extracted from the original image based on the coordinates of the effective boundary region, resulting in independent image sub-blocks. For example, using coordinates such as A: (1200, 800, 1350, 950) and S: (1360, 800, 1510, 950), seven independent sub-blocks are extracted from the original image. Each sub-block contains only one letter, with the character filling the entire sub-block and having complete edges.

[0027] Finally, each image sub-block is converted to grayscale and sorted from left to right according to its spatial position in the original image to generate a grayscale image sequence. Grayscale conversion refers to converting an RGB color image into a black-and-white gradient image with pixel values ​​from 0 to 255, reducing color interference while preserving the contrast between the characters and the background. The grayscale image sequence is a set of grayscale sub-blocks sorted by the left-right position of the characters in the original image, ensuring that the character order matches the actual imprint. The character sub-blocks are grayscaled with a grayscale value of 0.299×R + 0.587×G + 0.114×B. If multiple valid sub-blocks exist, they are sorted from left to right in the original image. Individual sub-blocks are directly used as sequence elements and combined to form the grayscale image sequence.

[0028] For example, seven single-character color sub-blocks are converted into grayscale images, with the characters being dark gray and having a pixel value of approximately 50, and the background being pure white and having a pixel value of approximately 255. They are then sorted horizontally from left to right according to the sequence A, S, P, I, R, I, N, generating a grayscale image sequence: [grayscale block A, grayscale block S, grayscale block P, grayscale block I, grayscale block R, grayscale block I, grayscale block N].

[0029] In this embodiment of the invention, false characters and background noise are eliminated through confidence screening, and the advantages of high-resolution images are combined to effectively improve the accuracy of character region localization; grayscale processing reduces color interference, orderly arrangement ensures consistent character order, and high-resolution sub-blocks preserve character details, providing high-quality data in a unified format for subsequent recognition and error correction; the constructed character region detection model supports differences in character size and position of different tablets and is compatible with high-definition input, laying the foundation for high accuracy and real-time performance of drug recognition.

[0030] S200: Perform character-by-character segmentation and recognition on the grayscale image sequence to generate an initial character sequence, and simultaneously extract the global physical feature set of the tablet.

[0031] In this embodiment of the invention, the grayscale image sequence is segmented and recognized character by character to generate an initial character sequence, while simultaneously extracting the global physical feature set of the tablet. Although the grayscale image sequence obtained after S100 has focused on single-character regions, some grayscale sub-blocks may have blurred boundaries due to character adhesion and ink dot diffusion, making direct recognition prone to errors. Relying solely on character information can easily lead to misjudgments of tablets with the same characters but different characters; for example, aspirin from different manufacturers may both use the character "ASPIRIN," but their shapes, colors, and surface textures may differ. Therefore, adaptive threshold binarization of the grayscale sub-blocks is needed to address the character blurring caused by uneven illumination. Vertical projection segmentation is used to separate adhered characters, and then character-by-character recognition is performed to generate the initial sequence. Simultaneously, the shape, color, texture, and other global physical features of the tablet are extracted, providing multi-dimensional auxiliary information for subsequent character sequence error correction and final tablet identification, thus improving the uniqueness and accuracy of the recognition.

[0032] Step S200 in the method provided in this embodiment of the invention includes: The process of segmenting and recognizing characters one by one in the grayscale image sequence to generate an initial character sequence includes: For each grayscale image sub-block in the grayscale image sequence, adaptive threshold binarization processing is performed to obtain a binarized image sub-block; Vertical projection analysis is performed on each binarized image sub-block, and the character segmentation boundary is determined based on the valley position of the projection histogram, thus segmenting each binarized image sub-block into multiple individual character images; Each individual character image is input into the character recognition model, and the output is the character category probability distribution of each individual character image; Based on the character category probability distribution of each individual character image, the character category corresponding to the maximum probability is selected as the recognition result of the corresponding individual character image; The recognition results are arranged according to the sorting of grayscale image sub-blocks and the left-to-right order of character segmentation boundaries within each grayscale image sub-block to generate an initial character sequence.

[0033] First, adaptive threshold binarization is performed on each grayscale image sub-block in the grayscale image sequence to obtain a binarized image sub-block.

[0034] Specifically, for each grayscale image sub-block in the grayscale image sequence, adaptive threshold binarization processing is performed to obtain a binarized image sub-block, including: For each grayscale image sub-block, a local window traversal method is used to calculate the grayscale mean and standard deviation in the neighborhood of each pixel. Based on the grayscale mean and standard deviation, and combined with the preset contrast adjustment coefficient, the local binarization threshold of each pixel is dynamically calculated. The gray value of each pixel is compared with the corresponding local binarization threshold. If the gray value is greater than or equal to the local binarization threshold, the pixel is marked as foreground; otherwise, it is marked as background. All foreground pixels are assigned a first brightness value, and background pixels are assigned a second brightness value to generate the binarized image sub-block.

[0035] First, for each grayscale image sub-block, a local window traversal method is used to calculate the grayscale mean and standard deviation within the neighborhood of each pixel. A local window is a small rectangular region centered on the target pixel, such as a 3×3 pixel area, used to statistically analyze the grayscale distribution around that pixel. The grayscale mean m(x,y) is the average grayscale value of all pixels within the local window, reflecting the overall brightness of the region. The grayscale standard deviation s(x,y) is the dispersion of grayscale values ​​among all pixels within the local window, reflecting the contrast of the region. For each grayscale image sub-block, a local window of a preset size is used to traverse the entire sub-block pixel by pixel. That is, each pixel within the sub-block is treated as the target pixel, and a 3×3 local window is drawn centered on it. The grayscale mean and standard deviation within the neighborhood of each target pixel are calculated, ensuring that all pixels in the sub-block are covered without omission.

[0036] For example, take pixel (10,10) within the grayscale sub-block A. Its 3×3 neighborhood grayscale values ​​are [120,125,122,124,123,121,126,124,122]: Grayscale mean m(10,10)=(120+125+...+122) / 9≈123; Grayscale standard deviation s(10,10)=[(120-123)] 2 +...+(122-123) 2 ] / 9≈1.8. For each pixel in the sub-block, a 3×3 window is taken with itself as the center, and the corresponding gray mean and standard deviation are calculated respectively.

[0037] Secondly, based on the grayscale mean and standard deviation, and combined with a preset contrast adjustment coefficient, the local binarization threshold for each pixel is dynamically calculated. The local binarization threshold T(x,y) is a dynamic threshold for a single pixel, used to distinguish characters from the background. T(x,y)=m(x,y)×(1+k×(s(x,y) / R-1)), where T(x,y) is the local binarization threshold of pixel (x,y), m(x,y) is the grayscale value of all pixels in the neighborhood window of this pixel, s(x,y) is the grayscale standard deviation of all pixels in the neighborhood window, k is a preset contrast adjustment coefficient, usually taken as 0.2, used to control the influence of local contrast on the threshold, and R is the normalization constant of the grayscale standard deviation, usually taken as 128. For each pixel within the grayscale sub-block, substitute the corresponding m(x,y) and s(x,y) values ​​into the formula to dynamically calculate the local binarization threshold specific to each pixel, ensuring that the threshold of each pixel is adapted to its local grayscale features.

[0038] For example, for a grayscale sub-block A: each pixel within the sub-block needs to have its own local binarization threshold calculated, and there is no uniform fixed value; taking pixel (10,10) as an example, given k=0.2 and R=128, the local binarization threshold T(10,10)=123×(1+0.2×(1.8 / 128-1))≈99; as another example, for pixel (50,50), where m(50,50)=118 and s(50,50)=2.1, then T(50,50)=118×(1+0.2×(2.1 / 128-1))≈95. The threshold for different pixels varies due to different local features.

[0039] Next, the grayscale value of each pixel is compared with the corresponding local binarization threshold. If the grayscale value is greater than or equal to the local binarization threshold, the pixel is marked as foreground; otherwise, it is marked as background. Foreground refers to the region representing a character after binarization, usually the character strokes. Background refers to the region representing a non-character after binarization, usually the background color of a pill. For each pixel within a grayscale sub-block, its original grayscale value is compared with the corresponding local binarization threshold: if the grayscale value is ≥ the local binarization threshold, it is marked as foreground; otherwise, it is marked as background, completing the classification of all pixels. For example, taking grayscale sub-block A as an example: traverse all pixels in the sub-block and complete the classification judgment one by one; among them, the grayscale value of pixel (10,10) is 123 ≥ 99, so it is marked as foreground; the grayscale value of pixel (80,80) is 85 < 92, so it is marked as background.

[0040] Further, all foreground pixels are assigned a first brightness value, and background pixels are assigned a second brightness value, generating the binarized image sub-block. The first brightness value refers to the value assigned to the foreground pixels, usually 255, corresponding to white. The second brightness value refers to the value assigned to the background pixels, usually 0, corresponding to black. For all pixels within the grayscale sub-block, pixels marked as foreground pixels are uniformly assigned the first brightness value of 255, and background pixels are uniformly assigned the second brightness value of 0, ultimately generating a binarized image sub-block composed of black and white. For example, after processing, the grayscale sub-block A yields: all character stroke pixels marked as foreground, such as (10,10), are assigned a value of 255; all pill background pixels marked as background, such as (80,80), are assigned a value of 0; finally, a black and white "A" binarized sub-block with no gradient and clear boundaries is obtained, with all pixels within the sub-block being assigned values ​​without omission.

[0041] Secondly, vertical projection analysis is performed on each binarized image sub-block. The character segmentation boundaries are determined based on the trough positions of the projection histogram, thus segmenting each binarized image sub-block into multiple individual character images. The vertical projection histogram is a column-pixel distribution curve obtained by counting the number of foreground pixels in each column of the binarized sub-block. The trough positions are the columns with extremely low pixel counts in the vertical projection histogram, corresponding to the gaps between characters and serving as segmentation boundaries. The vertical projection histogram of the binarized sub-block is calculated, and the pixel counts of all columns in the histogram are traversed to locate the trough positions as segmentation boundaries, splitting the sub-block into individual character images.

[0042] For example, if a grayscale sub-block contains the "IR" character, its binarized sub-block has 100 columns: count the number of foreground pixels in each column to obtain the corresponding values ​​for the 100 columns, i.e., the vertical projection histogram; traversing the histogram, it is found that the number of foreground pixels in the 50th column is only 2, which is much lower than the number of foreground pixels in the surrounding columns, and is determined to be a trough position; using the 50th column as the dividing boundary, the binarized sub-block is split into two independent single character images, "I" on the left and "R" on the right.

[0043] Furthermore, each individual character image is input into the character recognition model, which outputs the character category probability distribution for each individual character image. The character recognition model refers to a pre-trained CNN classification model capable of recognizing characters such as letters and numbers. The character category probability distribution refers to the probability value corresponding to each character category output by the character recognition model, reflecting the recognition confidence level. Each segmented individual character image is input one by one into the pre-trained character recognition model, and the model outputs a complete character category probability distribution for each character image. For example, if segmented individual character images such as "A", "S", and "P" are input sequentially, each image outputs its own probability distribution; when the character image "A" is input, the character recognition model outputs the probability distribution: A (0.98), B (0.01), and the probabilities of other characters are all <0.005; when the character image "I" is input, the character recognition model outputs the probability distribution: I (0.99), L (0.008), and the probabilities of other characters are all <0.002.

[0044] Based on this, according to the character category probability distribution of each individual character image, the character category corresponding to the maximum probability is selected as the recognition result for that individual character image. For each individual character image, the character corresponding to the maximum probability is selected as the recognition result. For example, in the output probability distribution of the character image "A", the probability corresponding to A is the maximum, which is 0.98, so A is selected as the recognition result.

[0045] Finally, the recognition results are arranged according to the sorting of the grayscale image sub-blocks and the left-to-right order of the character segmentation boundaries within each grayscale image sub-block to generate an initial character sequence. All recognition results are arranged sequentially according to the sorting of the grayscale sub-blocks in S100 and the left-to-right order of the characters within each sub-block to generate the initial character sequence. For example, the segmented individual character images are arranged in the order A, S, P, I, R, I, N, and the recognition results for each character image are A, S, P, I, R, I, N respectively; all results are combined in this order to generate the initial character sequence: ASPIRIN.

[0046] In addition, a global physical feature set of the tablet is extracted simultaneously. This global physical feature set includes at least shape contour parameters, color histogram distribution, and surface texture complexity index.

[0047] First, the shape and contour parameters of the tablet are extracted. These parameters describe the tablet's shape, including the aspect ratio of the bounding rectangle and its roundness. The aspect ratio refers to the width / length of the smallest rectangle enclosing the tablet's contour, reflecting its degree of roundness. A roundness value closer to 1 indicates a more rounded shape. Image segmentation techniques are used, and the Otsu algorithm is employed to separate the tablet region from the original image, obtaining the complete contour of the entire tablet. Based on this contour, the aspect ratio and roundness of the bounding rectangle are calculated to ensure coverage of the overall tablet's shape features. For example, this aspirin tablet is round. After extracting the tablet's contour, the smallest bounding rectangle enclosing the contour is fitted, with a width of 500 pixels and a length of 505 pixels. The aspect ratio is approximately 500 / 505 ≈ 1. The area of ​​the contour is approximately 196250 pixels, and the perimeter is approximately 3140 pixels. The roundness is calculated as 4 × π × 196250 / (3140). 2 ≈0.98, close to 1, which conforms to the characteristics of a circle.

[0048] Secondly, the color histogram distribution of the tablet is extracted. The color histogram distribution is an indicator describing color distribution, including the saturation channel mean and variance. The saturation channel mean refers to the average saturation value in the HSV color space, reflecting the vibrancy of the color. The saturation channel variance refers to the dispersion of saturation, reflecting color uniformity. The entire original tablet image is converted from RGB color space to HSV color space, and all pixel values ​​of the saturation channel are extracted. Their mean and variance are calculated to reflect the overall color characteristics of the tablet. For example, aspirin is white. The processing procedure is as follows: After conversion, all 20 million pixels of the saturation channel are extracted. The saturation mean of all pixels is approximately 0.1, indicating a less vibrant color. The saturation variance of all pixels is approximately 0.02, indicating a uniform color distribution with no obvious color distortion.

[0049] Finally, the surface texture complexity index of the tablet is extracted. The surface texture complexity index describes the smoothness of the surface, including the LBP histogram energy value and entropy value. LBP (Local Binary Pattern) generates codes by comparing pixel grayscale values ​​with their neighbors, reflecting local texture. The LBP histogram energy value is the sum of squares of the encoded histogram; a higher value indicates a simpler texture and a smoother surface. The LBP histogram entropy value measures the dispersion of the histogram; a lower value indicates a simpler texture. The LBP algorithm is applied to all pixels in the entire tablet area to generate an LBP encoded histogram. Based on the histogram, the energy and entropy values ​​are calculated to reflect the overall surface texture characteristics of the tablet. For example, for aspirin with a smooth surface, the process is as follows: the LBP algorithm is applied to all pixels in the tablet area, generating an encoded histogram containing 256 intervals. The histogram energy value is approximately 0.85, indicating few texture types and a simple surface; the histogram entropy value is approximately 0.3, indicating concentrated texture distribution and a smooth surface.

[0050] In this embodiment of the invention, by calculating the local binarization threshold pixel by pixel, it is possible to accurately adapt to areas with uneven lighting. Combined with vertical projection segmentation, it effectively separates the character adhesion problem, improves the accuracy of single character recognition, and makes the generated initial character sequence more reliable. Based on the global physical features such as shape, color, and texture extracted from the key pixels and regions of the entire tablet, a unique tablet identifier is added in addition to the characters, which can effectively avoid misjudgment of the same characters but different tablets. The final initial character sequence and the combination of global physical features provide complete and structured multi-dimensional data support for subsequent character error correction and tablet matching.

[0051] S300: Construct a sequence error correction model, and use the global physical feature set as input to correct the initial character sequence and output the corrected character sequence.

[0052] In this embodiment of the invention, a sequence error correction model is constructed, and the initial character sequence is corrected using the global physical feature set as input, outputting a corrected character sequence. The initial character sequence generated by S200 may have recognition errors due to issues such as character wear, lighting interference, and adhesion segmentation deviations, such as misclassifying 'I' as '1', 'O' as '0', and 'S' as '5'. Furthermore, relying solely on visual character features cannot verify errors; different tablets may have the same character sequence, but global physical features are unique identifiers for each type of tablet. For example, aspirin is round, white, and has a smooth surface, while a certain antibiotic tablet is oval, yellow, and has a textured surface. Therefore, a sequence error correction model needs to be constructed to fuse global physical features with the initial character sequence, establish the association between the tablet's global features and character positions through an attention mechanism, accurately correct recognition errors, and ensure that the character sequence matches the tablet's identity.

[0053] Step S300 in the method provided in this embodiment of the invention includes: The shape contour parameters, color histogram distribution, and surface texture complexity index in the global physical feature set are numerically concatenated to form a global feature vector; Each character in the initial character sequence is converted into a corresponding character embedding vector, and then arranged in sequence order to form a character sequence embedding matrix; The global feature vector and the character sequence embedding matrix are fused together and then input into the sequence error correction model. The sequence error correction model calculates the association weights between global features and the position of each character in the sequence based on an attention mechanism, and performs weighted adjustments on the character sequence embedding matrix. Based on the adjusted character sequence embedding matrix, the corrected character at each position is decoded and output sequentially by a classifier; The corrected characters at each position are combined in their original order to form the corrected character sequence.

[0054] First, the shape contour parameters, color histogram distribution, and surface texture complexity index from the global physical feature set are numerically concatenated to form a global feature vector. The global feature vector is a one-dimensional vector formed by concatenating the numerical parameters of the three global physical features—shape, color, and texture—in a fixed order; it is a numerical representation of the global features of the tablet. Numerical concatenation refers to integrating discrete feature parameters into a continuous vector according to a preset order of shape, color, and texture, without loss of dimension. All numerical parameters in the global physical feature set are concatenated in a fixed order—shape contour parameters, color histogram distribution, and surface texture complexity index—to form a one-dimensional global feature vector. For example, the global physical feature parameters of this aspirin tablet are: aspect ratio of the bounding rectangle = 1.0, roundness = 0.98, saturation mean = 0.1, saturation variance = 0.02, LBP energy value = 0.85, LBP entropy value = 0.3; after being concatenated in sequence, a 6-dimensional global feature vector is generated: [1.0, 0.98, 0.1, 0.02, 0.85, 0.3].

[0055] Next, each character in the initial character sequence is converted into a corresponding character embedding vector, and these vectors are arranged in sequence to form a character sequence embedding matrix. A character embedding vector is a dense vector of fixed dimensions that represents the semantic features of a single character. The character sequence embedding matrix is ​​a two-dimensional matrix formed by arranging the embedding vectors of each character in the initial character sequence in order, where the number of rows equals the number of characters, and the number of columns equals the dimension of the embedding vectors. A pre-trained character embedding model, such as Word2Vec or a CNN embedding layer, is used to convert each character in the initial character sequence into a fixed-dimensional character embedding vector; all character embedding vectors are then arranged vertically according to the order of the initial character sequence to generate the character sequence embedding matrix.

[0056] For example, the initial erroneous character sequence is "ASP1RIN", which consists of 7 characters. Each character is converted into a 128-dimensional embedding vector: A[0.12,0.35,...,0.21], S[0.23,0.18,...,0.42], P[0.19,0.41,...,0.33], I[0.56,0.29,...,0.17], R[0.31,0.22,...,0.45], I[0.28,0.37,...,0.19], N[0.15,0.43,...,0.27]. The 7 128-dimensional vectors are arranged in order to generate a 7-row × 128-column character sequence embedding matrix.

[0057] Furthermore, the global feature vector and the character sequence embedding matrix are fused and input into the sequence correction model. The global feature matrix is ​​a two-dimensional matrix obtained by copying and expanding the global feature vector, with the same number of rows as the character sequence embedding matrix, and each row containing the same global feature vector. The fused feature matrix is ​​a matrix formed by concatenating the global feature matrix and the character sequence embedding matrix along the feature dimension, containing both character semantic features and tablet global features.

[0058] For example, the 6-dimensional global feature vector is copied 7 times and matched with the 7 rows of the character sequence embedding matrix to generate a 7-row × 6-column global feature matrix. The 7-row × 6-column global feature matrix is ​​then concatenated with the 7-row × 128-column character sequence embedding matrix along the column dimension to generate a 7-row × 134-column fusion feature matrix. For instance, the 6-dimensional global feature vector [1.0, 0.98, 0.1, 0.02, 0.85, 0.3] of aspirin is copied 7 times to obtain a 7-row × 6-column global feature matrix, where each row is the 6-dimensional vector. After concatenating with the 7-row × 128-column character embedding matrix, a 7-row × 134-column fusion feature matrix is ​​obtained, where each row corresponds to the character embedding feature and global physical feature of a character position.

[0059] Subsequently, the sequence error correction model calculates the association weights between global features and the position of each character in the sequence based on the attention mechanism, and performs weighted adjustments on the character sequence embedding matrix.

[0060] The sequence error correction model calculates the association weights between global features and each character position in the sequence based on an attention mechanism, and performs weighted adjustments on the character sequence embedding matrix, including: Use the global feature vector as the query vector; The character sequence embedding matrix is ​​used as the key vector matrix and the value vector matrix; Calculate the dot product between the query vector and each position vector in the key vector matrix to obtain the initial attention score; The initial attention scores are normalized to generate the association weights; The value vector matrix is ​​weighted and summed using the associated weights to obtain a weighted and adjusted character sequence embedding matrix.

[0061] First, the global feature vector is used as the query vector. The query vector is a vector based on the global feature vector used to query which character positions need correction. The global feature vector is mapped to the same dimension as the character embedding vector to serve as the query vector. For example, the 6-dimensional global feature vector [1.0, 0.98, 0.1, 0.02, 0.85, 0.3] of aspirin is mapped to a 128-dimensional query vector [0.08, 0.15, ..., 0.22] through a fully connected layer.

[0062] Secondly, the character sequence embedding matrix is ​​used as the key vector matrix and value vector matrix. The key vector matrix is ​​the matrix used for matching the query vector, based on the character sequence embedding matrix. The value vector matrix is ​​the matrix used for final weighted adjustment, also based on the character sequence embedding matrix. The character sequence embedding matrix is ​​directly used as the key vector matrix and value vector matrix. For example, both the key vector matrix and value vector matrix are initially 7 rows × 128 columns of character sequence embedding matrix.

[0063] Further, the dot product of the query vector and each position vector in the key vector matrix is ​​calculated to obtain an initial attention score. The initial attention score refers to the dot product of the query vector and each character position vector in the key vector matrix, representing the initial correlation between the global feature and that character position. The query vector is then multiplied by the dot product of each character position vector in the key vector matrix to obtain multiple initial attention scores, each corresponding to a character position. For example, the dot product of the query vector and character vector A is 15.2, the dot product of the query vector and character vector S is 14.8, the dot product of the query vector and character vector P is 15.1, the dot product of the query vector and character vector 1 is 8.3, the dot product of the query vector and character vector R is 14.9, the dot product of the query vector and character vector I is 15.0, and the dot product of the query vector and character vector N is 14.7, generating an initial attention score list: [15.2, 14.8, 15.1, 8.3, 14.9, 15.0, 14.7].

[0064] Then, the initial attention scores are normalized to generate the association weights. Normalization involves using the Softmax function to convert the initial attention scores into weight values ​​in the 0-1 range, with the sum of all weights being 1, highlighting the weights of highly relevant character positions. The association weights are the normalized attention scores, representing the final degree of association between the global features and each character position. Softmax normalization is performed on all initial attention scores, calculating the proportion of each score to the total score, generating multiple association weights. For example, the initial attention score list: [15.2,14.8,15.1,8.3,14.9,15.0,14.7], after Softmax normalization, generates association weights: [0.2,0.13,0.2,0.02,0.15,0.18,0.12], where the weight of the erroneous character 1 is only 0.02, far lower than other characters.

[0065] Then, the value vector matrix is ​​weighted and summed using the association weights to obtain a weighted adjusted character sequence embedding matrix. Weighted summation refers to multiplying the association weights by the vectors at the corresponding character positions in the value vector matrix and then summing the results, weakening the features of erroneous characters and strengthening the features of correct characters. The association weight at each character position is multiplied by the vector in the corresponding row of the value vector matrix, and then all the weighted vectors are arranged by row to generate the adjusted character sequence embedding matrix. For example, the association weight of the erroneous character 1 is 0.02, and its embedding vector is significantly weakened; the weights of the other characters are approximately 0.17-0.18, and their features are preserved. The adjusted embedding vector at position 1 approaches the embedding vector of 1, ultimately generating a 7-row × 128-column adjusted character sequence embedding matrix.

[0066] Based on this, according to the adjusted character sequence embedding matrix, the classifier sequentially decodes and outputs the corrected character at each position. The classifier is the output layer of the sequence correction model, mapping the adjusted character embedding vectors to character category probability distributions. The corrected character refers to the character corresponding to the maximum probability output by the classifier, i.e., the corrected character. Each row of the adjusted character sequence embedding matrix is ​​input into the classifier, which outputs the category probability distribution for each character position. The character corresponding to the maximum probability is selected as the corrected character at that position. For example, inputting the adjusted embedding vector at position 1 into the classifier outputs the probability distribution: I (0.99), 1 (0.01), and the remaining characters < 0.001. Therefore, the corrected character at this position is I; the maximum probability for the remaining character positions is the original character, with no correction.

[0067] Finally, the corrected characters at each position are combined in their original order to form the corrected character sequence. For example, the corrected characters in the order A, S, P, I, R, I, N are combined to generate the corrected character sequence: ASPIRIN.

[0068] In this embodiment of the invention, a sequence error correction model that integrates the global physical features of a tablet is constructed. By using an attention mechanism, a dynamic association between global features and character positions is accurately established. This not only weakens the feature weight of erroneous characters in the initial character sequence but also strengthens the character features that match the tablet's identity. Ultimately, the erroneous initial character sequence is corrected into an accurate corrected character sequence, effectively solving the error problem caused by interference in simple character recognition. At the same time, the unique global physical features of the tablet are used to avoid misjudgments of identical characters but different tablets, improving the matching accuracy between the character sequence and the tablet's identity and providing accurate character basis for subsequent tablet identity confirmation.

[0069] S400: Match the corrected character sequence with the tablet feature database, determine the tablet category based on the matching similarity, and output the final recognition result. The similarity threshold is dynamically adjusted based on the historical verification rate of the current batch of tablets.

[0070] In this embodiment of the invention, the corrected character sequence is matched with a tablet feature database. The tablet category is determined based on the matching similarity, and the final recognition result is output. The similarity threshold is dynamically adjusted based on the historical verification rate of the current batch of tablets. Although the corrected character sequence after S300 correction reduces recognition errors, it may still lead to matching ambiguity due to insufficient uniqueness of tablet characters and a large number of database entries. Furthermore, a fixed similarity threshold cannot accommodate the quality differences between different batches of tablets; for example, some batches of tablets have high character wear rates and low historical verification rates. Therefore, it is necessary to accurately match the corrected character sequence with the tablet feature database, quantify the similarity through edit distance, and dynamically adjust the threshold based on the historical verification rate of the current batch. This avoids both excessively high thresholds leading to missed judgments and excessively low thresholds leading to misjudgments. Simultaneously, a verification mechanism is designed for matching failures, and the recognition process is iteratively optimized through image enhancement to ensure the accuracy and adaptability of the final recognition result.

[0071] Step S400 in the method provided in this embodiment of the invention includes: The process includes matching the modified character sequence with a tablet feature database and determining the tablet category based on the matching similarity, including: Using the modified character sequence as the query sequence, retrieve all character sequence entries of known tablets from the tablet feature database; Calculate the edit distance between the corrected character sequence and each character sequence entry, and convert it into a similarity score as the initial matching similarity. Based on the historical verification rate of the current batch of tablets, the similarity threshold is adjusted according to a preset linear mapping relationship, wherein the similarity threshold is adjusted upwards when the historical verification rate increases. The initial matching similarity is compared with the dynamically adjusted similarity threshold; When one or more preliminary matching similarities are greater than or equal to the similarity threshold, the corresponding tablet category with the highest similarity is selected as the judgment result output; When all preliminary matching similarities are less than the similarity threshold, the process is switched to a review mechanism.

[0072] First, using the modified character sequence as the query sequence, all character sequence entries for known tablets are retrieved from the tablet feature database. The tablet feature database is a structured database storing the character sequences, global physical features, and category information of all known tablets, supporting fast retrieval. The query sequence, i.e., the modified character sequence output by S300, serves as the core basis for matching. Character sequence entries refer to the standard character sequence corresponding to each known tablet in the database, such as ASPIRIN and AMOXICILLIN. Using the modified character sequence as the query keyword, a combination of string prefix matching and global fuzzy search is used to filter all character sequence entries containing or highly similar to the query sequence from the tablet feature database, forming a candidate matching set. For example, using ASPIRIN as the query sequence, three candidate entries are retrieved from the database: ASPIRIN (aspirin), ASPIRIN-500 (aspirin 500mg), and ASPIRIN-LA (aspirin extended-release tablets), forming a candidate matching set.

[0073] Next, the edit distance between the corrected character sequence and each character sequence entry is calculated and converted into a similarity score, which serves as the preliminary matching similarity. Edit distance is a metric that measures the difference between two strings; it refers to the minimum number of insertion, deletion, and replacement operations required to transform one string into another. The fewer the operations, the higher the similarity. The similarity score is a value normalized to the edit distance in the range of 0-1, where 1 represents a perfect match and 0 represents a complete mismatch. The formula is: Similarity Score = 1 - Edit Distance / Length of the Longer String. The edit distance between the corrected character sequence and each character sequence entry in the candidate matching set is calculated and then converted into a similarity score using a normalization formula, serving as the preliminary matching similarity.

[0074] For example, the calculation of the corrected character sequence ASPIRIN with 3 candidate entries is as follows: ASPIRIN: edit distance = 0, longer string length = 7, similarity score = 1 - 0 / 7 = 1.0. ASPIRIN-500: "-500" needs to be inserted, edit distance = 4, longer string length = 11, similarity score = 1 - 4 / 11 ≈ 0.636. ASPIRIN-LA: "-LA" needs to be inserted, edit distance = 3, longer string length = 11, similarity score = 1 - 3 / 10 = 0.7. A preliminary matching similarity list is generated: [1.0, 0.636, 0.7].

[0075] Furthermore, based on the historical verification rate of the current batch of tablets, the similarity threshold is adjusted according to a preset linear mapping relationship. The similarity threshold is adjusted upwards when the historical verification rate increases. The historical verification rate refers to the proportion of tablets in the current batch that are correctly identified after manual verification, reflecting the overall quality and identification difficulty of the batch. The linear mapping relationship refers to the preset correspondence rule between the historical verification rate and the similarity threshold. For example, when the verification rate ∈ [95%, 100%], the similarity threshold = 0.8 + (verification rate - 95%) × 0.02, ensuring that the threshold dynamically adapts to the verification rate. The dynamic threshold is the similarity judgment standard adjusted according to the historical verification rate; the higher the verification rate, the higher the threshold. The historical verification rate of the current batch of tablets is extracted, substituted into the preset linear mapping relationship, and the dynamically adjusted similarity threshold is calculated. For example, if the historical review rate of the current batch is 98%, the preset linear mapping relationship is: similarity threshold = 0.8 + (review rate - 95%) × 0.02; dynamic threshold = 0.8 + (98% - 95%) × 0.02 = 0.86.

[0076] Based on this, the initial matching similarity is compared with the dynamically adjusted similarity threshold.

[0077] First, when one or more preliminary matching similarities are greater than or equal to the similarity threshold, the corresponding tablet category with the highest similarity is selected as the judgment result output. If one or more similarities are greater than or equal to the threshold, the tablet category corresponding to the highest similarity is selected as the output. For example, comparing the preliminary matching similarity list [1.0, 0.636, 0.7] with the dynamic threshold 0.86: only the similarity 1.0 corresponding to ASPIRIN is greater than or equal to 0.86, and it is the highest value, so the judgment result output is ASPIRIN.

[0078] Otherwise, if all preliminary matching similarities are less than the aforementioned similarity threshold, the process will proceed to the review mechanism. For example, if the corrected character sequence in the example is ASPRIN, the edit distance between ASPRIN and ASPRIN is 1, and the similarity score is 1 - 1 / 7 ≈ 0.857 < 0.86, then all similarities will fail to meet the threshold, and the process will proceed to the review mechanism.

[0079] When all preliminary matching similarities are less than the similarity threshold, a review mechanism is initiated, including: The high-resolution image is subjected to morphological erosion, morphological dilation, and local contrast enhancement operations in sequence to obtain the enhanced image. Based on the enhanced image, the process from locating the boundary region of the imprinted character to generating the corrected character sequence is repeated, and a new matching similarity is calculated. Determine whether the new matching similarity is greater than or equal to the similarity threshold. If so, output the corresponding pill category as the final recognition result. If not, continue alternating between morphological operations and local feature enhancement, and calculate the corresponding matching similarity until it is greater than or equal to the similarity threshold or the preset maximum number of iterations is reached; When the maximum number of iterations is reached, the result with the highest matching similarity in each iteration is selected as the final recognition result, and the recognition data is recorded in the historical review record database.

[0080] First, morphological erosion, morphological dilation, and local contrast enhancement are sequentially performed on the high-resolution image to obtain the enhanced image. Morphological erosion involves traversing the image using a preset structuring element, retaining foreground pixels completely covered by the structuring element, removing small noise points, and refining character edges. Morphological dilation involves traversing the image using the same structuring element, filling gaps between character strokes, and restoring the integrity of character outlines; combined with erosion, it achieves noise reduction and shape preservation. Local contrast enhancement uses the CLAHE (Contrast-Limited Adaptive Histogram Equalization) algorithm to increase the grayscale difference between characters and the background, making character details clearer. The following operations are performed sequentially on the original high-resolution image: morphological erosion using a 3×3 rectangular structuring element, iterated once; morphological dilation using a 3×3 rectangular structuring element, iterated once; local contrast enhancement using the CLAHE algorithm, with a block size of 8×8 and a contrast limit of 2.0, to obtain the enhanced image.

[0081] For example, for the original image corresponding to the "ASPIRN" correction sequence with one missing "I" character: the tiny scratches on the tablet surface are removed by erosion, making the character edges cleaner; the minor breaks in the character strokes are filled by dilation, restoring the missing "I" stroke outline in ASPIRN; after local contrast enhancement, the grayscale difference between the character and the background increases from 15 to 30, and the "I" stroke is clearly distinguishable, resulting in the enhanced image.

[0082] Secondly, based on the enhanced image, the process from locating the boundary region of the imprinted characters to generating the corrected character sequence is repeated, and a new matching similarity is calculated. Based on the enhanced image, the processes S100 image localization and character region extraction, S200 character-by-character segmentation and global feature extraction, and S300 sequence error correction are repeated to generate a new corrected character sequence; then, a new matching similarity is calculated using the above method. For example, after repeating the process, the corrected character sequence of the enhanced image is corrected from ASPIRN to ASPIRN, and the new matching similarity is 1.0.

[0083] Further, it is determined whether the new matching similarity is greater than or equal to the similarity threshold. If so, the corresponding pill category is output as the final recognition result. It is also determined whether the new matching similarity is greater than or equal to the dynamic threshold: if so, the corresponding pill category is output. For example, if the new matching similarity after the first enhancement is 1.0 (≥ 0.86), "Aspirin" is directly output as the final recognition result.

[0084] If not, the alternating processing of morphological operations and local feature enhancement continues, and the corresponding matching similarity is calculated until it is greater than or equal to the similarity threshold or the preset maximum number of iterations is reached. The maximum number of iterations refers to the preset number of times image enhancement and recognition are repeated to avoid infinite iteration. For example, the preset maximum number of iterations is 3. If the new matching similarity is less than the similarity threshold, the alternating processing of erosion, dilation, and local contrast enhancement is repeated, and the matching similarity is calculated again. For example, if the similarity after the first enhancement is 0.82 < 0.86, a second enhancement is performed, and erosion, dilation, and contrast enhancement are repeated. The new similarity is 0.88 ≥ 0.86, and the result is output.

[0085] Finally, when the maximum number of iterations is reached, the result with the highest matching similarity in all iterations is selected as the final recognition result, and the recognition data is recorded in the historical review record database. The highest similarity in all iterations refers to the maximum value among all matching similarities during multiple enhancement recognition processes. When the maximum number of iterations is reached, the result corresponding to the highest similarity in all iterations is selected for output, and the data is recorded in the historical review record database. For example, if after 3 iterations the highest similarity = 0.85 < 0.86, the result corresponding to this highest similarity is "Aspirin," and the data is recorded in the historical review record database.

[0086] In this embodiment of the invention, by correcting the edit distance matching between the character sequence and the tablet feature database, and dynamically adjusting the similarity threshold in conjunction with the historical review rate of the current batch, adaptive optimization of the matching standard is achieved. This ensures the recognition efficiency of batches with high review rates while reducing the risk of misjudgment for low-quality batches. For cases where the matching does not meet the standard, alternating iterations of morphological operations and local contrast enhancement effectively repair character detail damage and improve the accuracy of secondary recognition. Finally, through the dual mechanism of direct matching and iterative review, the reliability and robustness of the final recognition result are ensured, while minimizing the workload of manual review and the recognition error rate, making it suitable for practical application needs in various scenarios such as hospital pharmacies and automated packaging machines.

[0087] Through the specific implementation methods described above, the embodiments of the present invention achieve the following technical effects: This invention provides a deep learning-based method and system for drug image recognition. First, precise single-character localization and adaptive threshold binarization effectively separate characters from complex backgrounds and lighting interference. Then, vertical projection segmentation addresses character adhesion issues, simultaneously extracting global physical features such as tablet shape, color, and texture, providing multi-dimensional data for recognition. Next, a sequence error correction model is constructed using feature fusion and attention mechanisms, leveraging the dynamic correlation between global physical features and character sequences to correct character recognition errors. Finally, an alternating verification mechanism combining edit distance matching, dynamic threshold adjustment based on historical verification rates, and morphological operations with local enhancements adapts to quality differences between batches of tablets and handles low-similarity matching scenarios. Ultimately, this effectively mitigates various practical problems such as character wear, background interference, and batch differences, improving the accuracy and adaptability of tablet recognition, reducing the risk of misjudgment and missed judgment, and meeting the practical application needs in multiple scenarios.

[0088] Example 2, as Figure 2 As shown, the present invention provides a drug image recognition system based on deep learning, the system comprising: The image localization and recognition module 11 is used to acquire a high-resolution image of the tablet to be identified, locate the boundary region of the characters engraved on the surface of the tablet through a pre-trained character region detection model, and extract the grayscale image sequence. The character segmentation and recognition module 12 is used to segment and recognize the grayscale image sequence character by character to generate an initial character sequence, and at the same time extract the global physical feature set of the tablet. The character sequence correction module 13 is used to construct a sequence error correction model, and to correct the initial character sequence using the global physical feature set as input, and output the corrected character sequence. The feature matching and judgment module 14 is used to match the modified character sequence with the tablet feature database, determine the tablet category based on the matching similarity, and output the final recognition result. The similarity threshold is dynamically adjusted based on the historical verification rate of the current batch of tablets.

[0089] In one embodiment, the image positioning and recognition module 11 is further configured to: Specifically, a pre-trained character region detection model is used to locate the boundary regions of the characters engraved on the surface of the tablet and extract grayscale image sequences, including: The high-resolution image is input into a pre-trained character region detection model, and the output obtains the coordinates and confidence scores of multiple candidate boundary regions. The pre-training steps of the character region detection model include: Based on the drug image sample library, sample tablet images containing clearly engraved characters are collected to form a sample image set; For each sample tablet image in the sample image set, the coordinates of the boundary region of the engraved characters are marked and a confidence label is generated for each boundary region, forming a boundary region coordinate label set and a corresponding confidence label set; A network architecture for a character region detection model is constructed based on machine learning, wherein the output layer of the network architecture is configured with both coordinate output channels and confidence output channels. The character region detection model is trained under supervision using the sample image set, boundary region coordinate label set, and confidence label set. During the training process, the coordinate regression loss and confidence prediction loss are optimized simultaneously until the error between the predicted boundary region coordinates output by the model and the boundary region coordinate label set converges to a preset convergence threshold, and the matching degree between the confidence output and the confidence label set reaches a preset standard, thus completing the pre-training of the model. The candidate boundary regions are filtered according to a preset confidence threshold, and the candidate boundary regions with a confidence level greater than or equal to the confidence threshold are retained as valid boundary regions. Based on the coordinates of the effective boundary region, the corresponding image sub-blocks are extracted from the high-resolution image; Each of the image sub-blocks is converted to grayscale and sorted from left to right according to its spatial position in the original image to generate a grayscale image sequence.

[0090] In one embodiment, the character segmentation and recognition module 12 is further configured to: The process of segmenting and recognizing characters one by one in the grayscale image sequence to generate an initial character sequence includes: For each grayscale image sub-block in the grayscale image sequence, adaptive threshold binarization processing is performed to obtain a binarized image sub-block; Vertical projection analysis is performed on each binarized image sub-block, and the character segmentation boundary is determined based on the valley position of the projection histogram, thus segmenting each binarized image sub-block into multiple individual character images; Each individual character image is input into the character recognition model, and the output is the character category probability distribution of each individual character image; Based on the character category probability distribution of each individual character image, the character category corresponding to the maximum probability is selected as the recognition result of the corresponding individual character image; The recognition results are arranged according to the sorting of grayscale image sub-blocks and the left-to-right order of character segmentation boundaries within each grayscale image sub-block to generate an initial character sequence.

[0091] Specifically, for each grayscale image sub-block in the grayscale image sequence, adaptive threshold binarization processing is performed to obtain a binarized image sub-block, including: For each grayscale image sub-block, a local window traversal method is used to calculate the grayscale mean and standard deviation in the neighborhood of each pixel. Based on the grayscale mean and standard deviation, and combined with the preset contrast adjustment coefficient, the local binarization threshold of each pixel is dynamically calculated. The gray value of each pixel is compared with the corresponding local binarization threshold. If the gray value is greater than or equal to the local binarization threshold, the pixel is marked as foreground; otherwise, it is marked as background. All foreground pixels are assigned a first brightness value, and background pixels are assigned a second brightness value to generate the binarized image sub-block.

[0092] The global physical feature set includes at least shape contour parameters, color histogram distribution, and surface texture complexity index.

[0093] In one embodiment, the character sequence correction module 13 is further configured to: The shape contour parameters, color histogram distribution, and surface texture complexity index in the global physical feature set are numerically concatenated to form a global feature vector; Each character in the initial character sequence is converted into a corresponding character embedding vector, and then arranged in sequence order to form a character sequence embedding matrix; The global feature vector and the character sequence embedding matrix are fused together and then input into the sequence error correction model. The sequence error correction model calculates the association weights between global features and the position of each character in the sequence based on an attention mechanism, and performs weighted adjustments on the character sequence embedding matrix. Based on the adjusted character sequence embedding matrix, the corrected character at each position is decoded and output sequentially by a classifier; The corrected characters at each position are combined in their original order to form the corrected character sequence.

[0094] The sequence error correction model calculates the association weights between global features and each character position in the sequence based on an attention mechanism, and performs weighted adjustments on the character sequence embedding matrix, including: Use the global feature vector as the query vector; The character sequence embedding matrix is ​​used as the key vector matrix and the value vector matrix; Calculate the dot product between the query vector and each position vector in the key vector matrix to obtain the initial attention score; The initial attention scores are normalized to generate the association weights; The value vector matrix is ​​weighted and summed using the associated weights to obtain a weighted and adjusted character sequence embedding matrix.

[0095] In one embodiment, the feature matching determination module 14 is further configured to: The process includes matching the modified character sequence with a pill feature database and determining the pill category based on the matching similarity, including: Using the modified character sequence as the query sequence, retrieve all character sequence entries of known tablets from the tablet feature database; Calculate the edit distance between the corrected character sequence and each character sequence entry, and convert it into a similarity score as the initial matching similarity. Based on the historical verification rate of the current batch of tablets, the similarity threshold is adjusted according to a preset linear mapping relationship, wherein the similarity threshold is adjusted upwards when the historical verification rate increases. The initial matching similarity is compared with the dynamically adjusted similarity threshold; When one or more preliminary matching similarities are greater than or equal to the similarity threshold, the corresponding tablet category with the highest similarity is selected as the judgment result output; When all preliminary matching similarities are less than the similarity threshold, the process is switched to a review mechanism.

[0096] When all preliminary matching similarities are less than the similarity threshold, a review mechanism is initiated, including: The high-resolution image is subjected to morphological erosion, morphological dilation, and local contrast enhancement operations in sequence to obtain the enhanced image; Based on the enhanced image, the process from locating the boundary region of the imprinted character to generating the corrected character sequence is repeated, and a new matching similarity is calculated. Determine whether the new matching similarity is greater than or equal to the similarity threshold. If so, output the corresponding pill category as the final recognition result. If not, continue alternating between morphological operations and local feature enhancement, and calculate the corresponding matching similarity until it is greater than or equal to the similarity threshold or the preset maximum number of iterations is reached; When the maximum number of iterations is reached, the result with the highest matching similarity in each iteration is selected as the final recognition result, and the recognition data is recorded in the historical review record database.

[0097] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A drug image recognition method based on deep learning, characterized in that, The method includes: Acquire high-resolution images of the tablets to be identified, locate the boundary regions of the characters engraved on the tablet surface using a pre-trained character region detection model, and extract grayscale image sequences. The grayscale image sequence is segmented and recognized character by character to generate an initial character sequence, while the global physical feature set of the tablet is extracted. Construct a sequence error correction model, and use the global physical feature set as input to correct the initial character sequence and output the corrected character sequence; The corrected character sequence is matched with the tablet feature database, the tablet category is determined based on the matching similarity, and the final recognition result is output. The similarity threshold is dynamically adjusted based on the historical verification rate of the current batch of tablets. The pre-trained character region detection model is used to locate the boundary regions of the characters engraved on the surface of the tablet, and a grayscale image sequence is extracted, including: The high-resolution image is input into a pre-trained character region detection model, and the output obtains the coordinates and confidence scores of multiple candidate boundary regions. The pre-training steps of the character region detection model include: Based on the drug image sample library, sample tablet images containing clearly engraved characters are collected to form a sample image set; For each sample tablet image in the sample image set, the coordinates of the boundary region of the engraved characters are marked and a confidence label is generated for each boundary region, forming a boundary region coordinate label set and a corresponding confidence label set; A network architecture for a character region detection model is constructed based on machine learning, wherein the output layer of the network architecture is configured with both coordinate output channels and confidence output channels. The character region detection model is trained under supervision using the sample image set, boundary region coordinate label set, and confidence label set. During the training process, the coordinate regression loss and confidence prediction loss are optimized simultaneously until the error between the predicted boundary region coordinates output by the model and the boundary region coordinate label set converges to a preset convergence threshold, and the matching degree between the confidence output and the confidence label set reaches a preset standard, thus completing the pre-training of the model. The candidate boundary regions are filtered according to a preset confidence threshold, and the candidate boundary regions with a confidence level greater than or equal to the confidence threshold are retained as valid boundary regions. Based on the coordinates of the effective boundary region, the corresponding image sub-blocks are extracted from the high-resolution image; Each of the image sub-blocks is converted to grayscale and sorted from left to right according to their spatial position in the original image to generate a grayscale image sequence. Construct a sequence error correction model, and use the global physical feature set as input to correct the initial character sequence, outputting a corrected character sequence, including: The shape contour parameters, color histogram distribution, and surface texture complexity index in the global physical feature set are numerically concatenated to form a global feature vector; Each character in the initial character sequence is converted into a corresponding character embedding vector, and then arranged in sequence order to form a character sequence embedding matrix; The global feature vector and the character sequence embedding matrix are fused together and then input into the sequence error correction model. The sequence error correction model calculates the association weights between global features and the position of each character in the sequence based on an attention mechanism, and performs weighted adjustments on the character sequence embedding matrix. Based on the adjusted character sequence embedding matrix, the corrected character at each position is sequentially decoded and output by the classifier; The corrected characters at each position are combined in their original order to form the corrected character sequence.

2. The drug image recognition method based on deep learning according to claim 1, characterized in that, The grayscale image sequence is segmented and recognized character by character to generate an initial character sequence, including: For each grayscale image sub-block in the grayscale image sequence, adaptive threshold binarization processing is performed to obtain a binarized image sub-block; Vertical projection analysis is performed on each binarized image sub-block, and the character segmentation boundary is determined based on the valley position of the projection histogram, thus segmenting each binarized image sub-block into multiple individual character images; Each individual character image is input into the character recognition model, and the output is the character category probability distribution of each individual character image; Based on the character category probability distribution of each individual character image, the character category corresponding to the maximum probability is selected as the recognition result of the corresponding individual character image; The recognition results are arranged according to the sorting of grayscale image sub-blocks and the left-to-right order of character segmentation boundaries within each grayscale image sub-block to generate an initial character sequence.

3. The drug image recognition method based on deep learning according to claim 2, characterized in that, For each grayscale image sub-block in the grayscale image sequence, adaptive threshold binarization processing is performed to obtain a binarized image sub-block, including: For each grayscale image sub-block, a local window traversal method is used to calculate the grayscale mean and standard deviation in the neighborhood of each pixel. Based on the grayscale mean and standard deviation, and combined with the preset contrast adjustment coefficient, the local binarization threshold of each pixel is dynamically calculated. The gray value of each pixel is compared with the corresponding local binarization threshold. If the gray value is greater than or equal to the local binarization threshold, the pixel is marked as foreground; otherwise, it is marked as background. All foreground pixels are assigned a first brightness value, and background pixels are assigned a second brightness value to generate the binarized image sub-block.

4. The drug image recognition method based on deep learning according to claim 1, characterized in that, The sequence error correction model calculates the association weights between global features and the position of each character in the sequence based on an attention mechanism, and performs weighted adjustments on the character sequence embedding matrix, including: Use the global feature vector as the query vector; The character sequence embedding matrix is ​​used as the key vector matrix and the value vector matrix; Calculate the dot product between the query vector and each position vector in the key vector matrix to obtain the initial attention score; The initial attention scores are normalized to generate the association weights; The value vector matrix is ​​weighted and summed using the associated weights to obtain a weighted and adjusted character sequence embedding matrix.

5. The drug image recognition method based on deep learning according to claim 1, characterized in that, The modified character sequence is matched with a pill feature database, and the pill category is determined based on the matching similarity, including: Using the modified character sequence as the query sequence, retrieve all character sequence entries of known tablets from the tablet feature database; Calculate the edit distance between the corrected character sequence and each character sequence entry, and convert it into a similarity score as the initial matching similarity. Based on the historical verification rate of the current batch of tablets, the similarity threshold is adjusted according to a preset linear mapping relationship, wherein the similarity threshold is adjusted upwards when the historical verification rate increases. The initial matching similarity is compared with the dynamically adjusted similarity threshold; When one or more preliminary matching similarities are greater than or equal to the similarity threshold, the corresponding tablet category with the highest similarity is selected as the judgment result output; When all preliminary matching similarities are less than the similarity threshold, the process is switched to a review mechanism.

6. The deep learning-based drug image recognition method according to claim 5, characterized in that, When all preliminary match similarities are less than the aforementioned similarity threshold, a review mechanism is initiated, including: The high-resolution image is subjected to morphological erosion, morphological dilation, and local contrast enhancement operations in sequence to obtain the enhanced image; Based on the enhanced image, the process from locating the boundary region of the imprinted character to generating the corrected character sequence is repeated, and a new matching similarity is calculated. Determine whether the new matching similarity is greater than or equal to the similarity threshold. If so, output the corresponding pill category as the final recognition result. If not, continue alternating between morphological operations and local feature enhancement, and calculate the corresponding matching similarity until it is greater than or equal to the similarity threshold or the preset maximum number of iterations is reached; When the maximum number of iterations is reached, the result with the highest matching similarity in each iteration is selected as the final recognition result, and the recognition data is recorded in the historical review record database.

7. A drug image recognition system based on deep learning, characterized in that, The system is used to implement the deep learning-based drug image recognition method according to any one of claims 1-6, the system comprising: The image localization and recognition module is used to acquire high-resolution images of the tablets to be identified, locate the boundary regions of the characters engraved on the surface of the tablets through a pre-trained character region detection model, and extract grayscale image sequences. The character segmentation and recognition module is used to segment and recognize the grayscale image sequence character by character, generate an initial character sequence, and extract the global physical feature set of the tablet. The character sequence correction module is used to construct a sequence error correction model, and uses the global physical feature set as input to correct the initial character sequence and output the corrected character sequence. The feature matching and judgment module is used to match the corrected character sequence with the tablet feature database, determine the tablet category based on the matching similarity, and output the final recognition result. The similarity threshold is dynamically adjusted based on the historical verification rate of the current batch of tablets.

Images