Ocr-based medical document intelligent recognition method and system

Abstract

The application discloses an OCR-based medical document intelligent identification method and system, relates to the technical field of document identification, and comprises the following steps: collecting a medical document image through a mobile terminal, generating a binary image based on the collected image through a U-Net combined with multi-scale feature fusion and an attention mechanism, and performing clipping; based on the clipped binary image, extracting text information through an OCR model, and extracting structured fields according to the layout rules and geometric distribution of the medical document; and performing deterministic rule judgment and risk assessment on the structured data. The application reduces the image calculation complexity through standard gray scale processing, improves the text region feature extraction accuracy through the U-Net structure combined with the CBAM attention mechanism, realizes effective fusion of multi-scale features and noise suppression in combination with the Attention Gate, realizes text direction correction in combination with the Hough transform, and improves the text detection robustness and recognition accuracy.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of document recognition, in particular to an intelligent recognition method and system for medical documents based on OCR. BACKGROUND

[0002] In recent years, with the acceleration of medical informatization, intelligent recognition technology for medical documents has gradually become a research hotspot. As an important part of this field, OCR (Optical Character Recognition) technology aims to realize the automatic recognition and digital processing of text information in paper or electronic medical documents. Early OCR technology mainly relies on traditional machine learning methods, such as using feature extraction combined with a classifier to recognize characters. However, these methods perform poorly when faced with complex and variable medical documents, especially when dealing with handwritten prescriptions, small fonts on drug packaging, and test report forms containing a large number of professional terms and abbreviations. In recent years, the development of deep learning, especially convolutional neural networks (CNN), has brought revolutionary progress to OCR technology. By training large-scale data sets, CNN-based OCR models can more accurately locate and recognize various types of text information.

[0003] Despite the above progress, there are still some deficiencies in existing technologies, especially in the intelligent recognition of medical documents. First, general OCR systems usually directly process raw RGB images, lacking pre-processing mechanisms for the imaging characteristics of medical documents (such as indoor non-uniform lighting, high light reflection, paper creases, or reflective materials on drug packaging), making it difficult to effectively separate text regions from backgrounds. Second, although mainstream text detection models (such as EAST, DB) can locate regular text, their segmentation accuracy significantly decreases when faced with low contrast, blurred characters, or skewed layouts commonly found in medical documents. Therefore, existing intelligent recognition technology for medical documents still has problems such as fuzzy text recognition, unclear text region segmentation, and incomplete structure field extraction. SUMMARY

[0004] In view of the above existing problems, the present application is proposed.

[0005] Therefore, the present application provides an intelligent recognition method and system for medical documents based on OCR, which solves the problems of fuzzy text recognition, unclear text region segmentation, and incomplete structure field extraction.

[0006] To solve the above technical problems, the present application provides the following technical solutions:

[0007] In a first aspect, the present application provides an intelligent recognition method for medical documents based on OCR, which includes,

[0008] Medical document images are collected via mobile terminals. Based on the collected images, binary images are generated using U-Net combined with multi-scale feature fusion and attention mechanisms, and then cropped.

[0009] Based on cropped binary images, text information is extracted using an OCR model, and structured fields are extracted according to the layout rules and geometric distribution of medical documents.

[0010] Perform deterministic rule judgments and risk assessments on structured data, and generate graphic reports based on the risk assessment results.

[0011] As a preferred embodiment of the OCR-based intelligent medical document recognition method of the present invention, the method comprises: generating a binary image based on the acquired image by combining U-Net with multi-scale feature fusion and attention mechanism, and cropping it; converting the RGB color image into a single-channel grayscale image using the ITU-R BT.601 standard grayscale conversion formula; inputting the grayscale image into the U-Net structure; using ResNet50 as the encoder backbone network; after the ResNet50 encoder outputs the feature map, performing channel and spatial attention enhancement on the feature map through the CBAM module to obtain the enhanced feature map; using the enhanced feature map as the encoder output and passing it to the decoder branch of the corresponding resolution for upsampling to obtain the upsampled feature map; at the skip connection of the network, mapping the enhanced feature map and the upsampled feature map to the same channel dimension through 1×1 convolution, and using the Attention Gate mechanism to calculate the spatial attention mask.

[0012] Spatial attention mask Generate a fused feature map, and then use the Sigmoid activation function to fuse the feature map. The loss function is obtained by converting the data into a binary probability map and using a weighted combination of Dice loss and BCE loss. The chain rule is used to calculate the gradient of the loss with respect to each weight, and all learnable parameters in the network are adjusted according to the gradient to reduce the value of the loss function. This process is repeated iteratively until the maximum number of iterations is reached, and the final binary probability map is obtained based on the trained network. Set threshold The probability value of pixel (x,y) Greater than or equal to the threshold If the output is white, it is black otherwise. A binary image is generated. The Hough linear transform is performed on the binary image to obtain a rotated and corrected binary image. Contour detection is then performed to obtain a cropped binary image.

[0013] As a preferred scheme of the medical document intelligent recognition method based on OCR, wherein: the step of extracting text information based on the cropped binary image through the OCR model comprises the following steps: using a single-scale Retinex algorithm to perform contrast enhancement on the cropped binary image, sequentially performing adaptive histogram equalization, morphological closing operation, median filtering, Sobel edge enhancement and background suppression on the preliminarily enhanced image to obtain a final enhanced image, inputting the final enhanced image into a PP-OCRv3 model, and the model extracts multi-scale features through a convolutional backbone network, and the extracted features are input into two parallel output branches to obtain a text score map S and a four-point offset regression map respectively, and a text detection loss function is calculated, the probability of each pixel point belonging to a text region is calculated by using a differentiable binarization function on the text score map S;

[0014] A continuous probability map P is obtained through the text probability of each pixel point, the text region mask is obtained by thresholding the continuous probability map P, and the text region mask is analyzed by using an 8-connected domain analysis method to find all the connected regions, i.e., text instance regions, and the four-point offset regression map corresponding to the pixels in each text instance region is decoded to obtain the four-point coordinates of the text instance region, and the height and width of the minimum bounding rectangle are calculated, and a threshold value is set When the height of the minimum bounding rectangle is less than the threshold value , it is considered that the region is too small and is not an effective text block, and a perspective transformation is performed on each effective text block to generate a standardized rectangular image block Each standardized image block is input into an SVTR recognition network to output the character sequence of each text block, a text recognition output loss function is calculated, a total loss function is calculated in combination with the text detection loss function, gradients are calculated by using a back propagation algorithm to minimize the total loss function, all learnable parameters are updated, and the final character sequence of each text block is output, the average confidence of each text block is calculated, and a threshold value is set When the average confidence is greater than or equal to the threshold value , it is considered that the recognition result is reliable, otherwise, it is recorded as low-confidence text, and the detection and recognition results are combined in a spatial order to form a complete structured text sequence. As a preferred scheme of the medical document intelligent recognition method based on OCR, wherein: the step of extracting structured fields according to the layout rule and geometric distribution of the medical document comprises the following steps: arranging the structured text sequence in an ascending order according to the upper boundaries of each text block

[0015] , and calculating the vertical spacing between adjacent text blocks , setting a threshold value , and when the vertical spacing is greater than the threshold value , it is considered that the text block is a structured field, otherwise, it is considered that the text block is not a structured field.​​ less than a threshold value If so, they are considered to belong to the same logical line, and for the text blocks within the same logical line, the horizontal spacing of adjacent boxes is calculated , a threshold value is set When the horizontal spacing is less than the threshold value , the contents of adjacent text blocks are spliced into a complete text, and the final logical line set is obtained. The left boundary coordinates and the right boundary coordinates of all text blocks in each logical line are extracted, and one-dimensional Gaussian smoothing is performed on all left boundary coordinates and right boundary coordinates to obtain left and right smoothing curves f(x) and g(x).

[0016] The local maximum of the left and right smoothing curves f(x) and g(x) is detected, and the maximum peak position is taken as the "item column center" , "result column center" , according to the geometric prior of the three-column symmetry distribution of the medical test sheet, the reference range column center is calculated , the distance between the left boundary of the text block in each logical line and the three column centers is calculated, the text block is assigned to the column with the smallest distance to determine the field type, the item column text is matched with the medical index dictionary, the index naming is unified, the numerical value and unit are extracted from the result column text, the interval in the reference range column text is converted to a numerical interval, and each row of information is combined into a standardized structure.

[0017] As a preferred scheme of the medical document intelligent recognition method based on OCR, wherein: the determination rule judgment and risk assessment on the structured data are based on the standardized structure combined by each row of information and the basic health profile of the user. For each index item in the structured data, quantitative abnormality judgment is performed. In the determined abnormal index, the deterministic IF-THEN rule matching is performed according to the medical rule library. According to the risk level obtained by matching, the corresponding medical explanation text is automatically mapped to form a readable result.

[0018] As a preferred scheme of the medical document intelligent recognition method based on OCR, wherein: the generation of the picture-text report according to the risk assessment result refers to a pre-set natural language report template. For each test item, the readable result is replaced according to the template placeholder to generate a complete natural language sentence. According to the risk level of all test items, the health suggestion text is mapped. All natural language sentences and health suggestions are combined to form a complete report structure. The report is displayed in the form of a card on the user interface of the mobile terminal.

[0019] As a preferred scheme of the medical document intelligent recognition method based on OCR, wherein: the medical document image is collected by a mobile terminal, which refers to arranging uniform illumination light sources in an indoor environment and collecting color images through the rear main camera of the mobile terminal.

[0020] In a second aspect, the present application provides an OCR-based intelligent medical document recognition system, comprising,

[0021] a document image acquisition module, configured to place a document under uniform illumination, align and shoot to avoid reflection, and acquire a high-quality RGB image;

[0022] a document segmentation enhancement module, configured to generate a binary image based on a U-Net and an attention mechanism, and reserve a main body region through rotation and cropping;

[0023] a text enhancement recognition module, configured to fuse SSR, CLAHE and edge enhancement to improve image quality, and output a text sequence with confidence through a detection-recognition model;

[0024] a structured field extraction module, configured to distribute text to item, result and reference range fields according to a medical document layout rule;

[0025] a risk assessment report module, configured to judge abnormalities in combination with a health portrait and medical rules, and generate a graphic risk report with explanations and suggestions.

[0026] In a third aspect, the present application provides a computer device comprising a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, any step of the OCR-based intelligent medical document recognition method according to the first aspect of the present application is implemented.

[0027] In a fourth aspect, the present application provides a computer readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, any step of the OCR-based intelligent medical document recognition method according to the first aspect of the present application is implemented.

[0028] The present application has the following beneficial effects: through standard gray scaling and Retinex image enhancement technology, the present application effectively improves the recognizability of images under complex illumination, combines a U-Net structure based on ResNet50 and CBAM attention mechanism, realizes high-precision extraction of a text region and suppression of background interference, completes arrangement correction of multi-page, inclined and long documents through Hough transformation and perspective transformation, extracts "item-result-reference range" logical fields through Gaussian fitting, and realizes standardized naming and structured output in cooperation with a medical dictionary, thereby improving text recognition accuracy, structure restoration capability and subsequent data availability. BRIEF DESCRIPTION OF DRAWINGS

[0029] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings needed to be used in the embodiment description will be briefly introduced as follows. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative effort on the basis of these drawings.

[0030] Fig. 1 Flow chart of the OCR-based medical document intelligent recognition method in embodiment 1.

[0031] Fig. 2 Structure diagram of the OCR-based medical document intelligent recognition system in embodiment 1.

[0032] Fig. 3 Flow chart of the text recognition and structured extraction in embodiment 1. DETAILED DESCRIPTION

[0033] In order to make the above-mentioned purposes, features and advantages of the present application more apparent and easy to understand, the specific embodiments of the present application will be described in detail below with reference to the drawings in the specification.

[0034] In the following description, many specific details are set forth in order to provide a thorough understanding of the present application. However, the present application can be practiced in other ways not described herein, and the skilled in the art can make similar generalizations without departing from the scope of the present application, so the present application is not limited to the specific embodiments disclosed below.

[0035] Secondly, the "one embodiment" or "embodiment" referred to herein can include specific features, structures or characteristics contained in at least one implementation of the present application. In this specification, "in one embodiment" appearing in different places does not mean the same embodiment, nor is it an independent or alternative embodiment that excludes other embodiments.

[0036] Embodiment 1, refer to Figs. 1-3 , the first embodiment of the present application, the embodiment provides an OCR-based medical document intelligent recognition method, comprising the following steps:

[0037] S1, collecting medical document images through a mobile terminal, based on the collected images, generating a binary image through U-Net combined with multi-scale feature fusion and attention mechanism, and performing clipping;

[0038] Specifically, capturing medical document images via mobile terminal involves setting up a uniform lighting source in an indoor environment, avoiding direct light and strong reflections. The subject (flat documents such as drug packaging, medical test reports, or prescriptions) is placed flat on a dark, non-reflective, non-glaring surface, ensuring the surface is smooth, without creases or curls. The center of the document is aligned with the center of the mobile terminal's rear main camera's viewfinder, ensuring the long side of the document is parallel to the long side of the camera's field of view. If the document has multiple pages or exceeds the length of a single frame (e.g., a long prescription), it is photographed page by page in sequence, ensuring each page is complete, to obtain RGB color images. .

[0039] By implementing measures such as providing uniform indoor lighting, using a dark, non-reflective desktop, ensuring documents are flat and aligned, and taking standardized photos of multi-page documents, reflections, shadows, and distortions are effectively eliminated, significantly improving image clarity and text readability.

[0040] Furthermore, based on the acquired images, a binary image is generated using U-Net combined with multi-scale feature fusion and attention mechanisms, and then cropped. The RGB color image is then converted using the ITU-R BT.601 standard grayscale conversion formula. Convert to a single-channel grayscale image to reduce subsequent computational complexity and accurately represent brightness information:

[0041] ;

[0042] in, It is the grayscale value of the pixel (x, y). This is a single-channel grayscale image, where R(x,y), G(x,y), and B(x,y) represent the intensity of pixel (x,y) in the red, green, and blue channels, respectively. , , It is a coefficient set based on the human eye's sensitivity to different colors;

[0043] grayscale image The input is fed into a U-Net-based architecture, using a ResNet50 pre-trained on ImageNet as the encoder backbone. CBAM (Convolutional Block Attention) is incorporated into the identity block outputs of stages 3, 4, and 5 of the ResNet50 to automatically focus on key information (such as text regions) and suppress irrelevant regions (such as highlights and reflections). After the ResNet50 encoder outputs feature map E, the CBAM module performs channel and spatial attention enhancement on feature map E. The channel attention weights are calculated using the following formula:

[0044] ;

[0045] in, is the channel attention weight, AvgPool(E) and MaxPool(E) are the average pooling and max pooling operations on the feature map E, respectively, to obtain information at different levels, and MLP is a multi-layer perceptron, is a sigmoid activation function.

[0046] based on the channel attention weight element-wise multiplication on the feature map E to obtain an intermediate feature map:

[0047] ;

[0048] wherein, is the intermediate feature map, element-wise multiplication, which enhances the response of key channels in the feature map;

[0049] based on the intermediate feature map the spatial attention weight is calculated:

[0050] ;

[0051] wherein, is the spatial attention weight, is a 7x7 convolution operation for further extracting spatial features, is a sigmoid activation function.

[0052] through the spatial attention weight element-wise multiplication on the intermediate feature map to obtain an enhanced feature map:

[0053] ;

[0054] wherein, is the enhanced feature map;

[0055] the enhanced feature map is taken as the encoder output and passed to the corresponding resolution of the decoder branch for upsampling to obtain an upsampled feature map. At the network’s skip connection, the enhanced feature map and the upsampled feature map are respectively mapped to the same channel dimension through a 1x1 convolution, and a spatial attention mask is calculated using an Attention Gate (AG) mechanism:

[0056] ;

[0057] wherein, is the spatial attention mask, is the enhanced feature map the encoder feature map mapped to the same channel dimension, is the decoder up-sampled feature map after mapping to the same channel dimension, , is the learnable 1x1 convolution weight for aligning the channel dimension of the encoder and decoder, [·;·] is the channel concatenation operation, and ReLU(·) is the ReLU activation function, is the sigmoid activation function.

[0058] The spatial attention mask is applied to the encoder feature map and fused with the decoder up-sampled feature map to generate the fused feature map , which dynamically masks irrelevant background noise (such as reflections, textures, etc.):

[0059] ;

[0060] wherein, is the fused feature map, is the element-wise multiplication.

[0061] The fused feature map is converted into a binary probability map through the sigmoid activation function, which represents the binary probability of the text region, and a weighted combination of Dice loss and BCE loss is adopted to ensure high precision of the segmentation result, and the loss function is:

[0062] ;

[0063] wherein, is the loss function, is the balance coefficient, which is set through fuzzy logic, is the Dice loss function, is the BCE loss function.

[0064] Dice loss function The calculation formula is:

[0065] ;

[0066] wherein, O is the ideal binary map annotated by human, is a small constant to prevent division by zero error;

[0067] BCE loss function The calculation formula is:

[0068] ;

[0069] wherein, Q is the total number of pixels of the image.

[0070] based on the calculated loss function , the chain rule (i.e. backpropagation algorithm) is used to calculate the gradient of the loss with respect to each weight, and all learnable parameters in the network are adjusted according to the gradient to make the loss function value decrease, and repeated iterations are performed until the maximum number of iterations is reached, and the final binary probability map is obtained based on the trained network , the threshold is set through statistical analysis , when the probability value of the pixel point (x, y) is greater than or equal to the threshold , output white, otherwise black, generate a binary image , to correct the slight tilt when shooting, the binary image is subjected to Hough line transformation, which is used to detect the main direction of all text blocks in the image, and the most voted angle is selected as the global text main direction angle through the cumulative voting method , according to the main direction angle , the image is rotated and corrected, and the text block is aligned to the horizontal (or vertical):

[0071] ;

[0072] wherein, is the binary image after rotation correction, is the image rotation angle;

[0073] to remove background interference and only keep the document main body area, the binary image after rotation correction is subjected to contour detection to extract all closed contours, which are sorted according to their area size, and the largest one is selected as the document main body contour. Based on the contour's bounding box, the document main body area range is determined, and a fixed pixel margin is expanded around it to completely retain the document edge information, and the cropped binary image is obtained .

[0074] The ITU-R BT.601 standard is used to convert the RGB image to a grayscale image, which reduces the computational complexity while accurately retaining the brightness information. A pre-trained ResNet50 combined with a CBAM module is used to automatically focus on the text area and suppress irrelevant background noise, improving the accuracy of feature extraction. The channel and spatial attention mechanisms are further enhanced to optimize the feature expression of the text area and enhance the model's ability to resist interference. The Attention Gate mechanism and the combination of Dice loss and BCE loss are used to ensure high-precision text segmentation results. The Hough transform is used to correct the tilted text blocks and the contour detection is used to remove background interference, only keeping the document main body area, ensuring that the output binary image has good visual effects and high information fidelity, improving the text recognition efficiency and greatly enhancing the accuracy of information extraction.

[0075] S2. Based on the cropped binary image, extract text information through the OCR model, and extract structured fields according to the layout rules and geometric distribution of medical documents;

[0076] Specifically, based on cropped binary images, text information is extracted using an OCR model targeting the cropped binary images. Contrast enhancement is performed using the single-scale Retinex (SSR) algorithm to correct for uneven illumination and improve local contrast. The calculation expression is as follows:

[0077] ;

[0078] in, These are the pixel (x, y) points of the initially enhanced image. This is the image after initial enhancement. It represents the (x, y) pixel value of the cropped region. The standard deviation is Gaussian kernel, These are the weight coefficients, set experimentally, and * represents the convolution operation.

[0079] The image after initial enhancement Adaptive histogram equalization (CLAHE), morphological closing operation, median filtering, Sobel edge enhancement and background suppression are performed sequentially.

[0080] The adaptive histogram equalization (CLAHE) is applied to the initially enhanced image. The CLAHE algorithm is used to enhance the local contrast of the image, ensuring uniform brightness distribution in different areas.

[0081] The morphological closing operation described above is applied to the image obtained after CLAHE processing. It uses a morphological closing operation (dilation followed by erosion) to repair character breaks caused by noise or blurring, while maintaining character integrity.

[0082] ;

[0083] ;

[0084] Here, D is the structuring element, a small matrix that defines the shape and size of the neighborhood, typically a square or circle. This is an image obtained after Claesh processing. It is an expansion operation. It is an erosion operation, and (s,l) is the coordinate offset in the structuring element D;

[0085] The median filtering is a smoothing process on the image after morphological closing operation, which removes noise by using a 3x3 window median filter while preserving edge information and protecting character features as much as possible;

[0086] The Sobel edge enhancement is based on the smoothing result, calculating the gradients in horizontal and vertical directions respectively, and calculating the gradient magnitude based on the gradients in horizontal and vertical directions for enhancing the character edges:

[0087] ;

[0088] where T is the gradient magnitude, , are the gradients in horizontal and vertical directions respectively, obtained by using Sobel operator convolution;

[0089] Based on the gradient magnitude T and the smoothed image, a weighted fusion is performed to obtain an edge-enhanced image:

[0090] ;

[0091] where, is the edge-enhanced image, is the fusion weight coefficient, which is set by grid search, is the image smoothed by median filtering;

[0092] The background suppression is to calculate the binarization threshold using Otsu method to generate a binary mask:

[0093] ;

[0094] where, is the binary mask, represents the background, i.e. non-text area, represents the foreground, i.e. the area containing character or text structure, is the pixel point (x, y) of the edge-enhanced image;

[0095] Based on the binary mask the final enhanced image is obtained :

[0096] ;

[0097] where, is the final enhanced image, is the pixel point (x, y) of the final enhanced image;

[0098] The final enhanced image Input to the PP-OCRv3 model, the model extracts multi-scale features through a convolutional backbone network (using a lightweight MobileNetV3 structure and pre-trained on ImageNet), and the backbone network is fine-tuned end-to-end on a dataset containing real medical images (covering drug packaging, test report sheets, handwritten prescriptions, etc.) to enhance robustness to low-contrast, blurred, tilted, and handwritten text. During the training phase, the extracted features are input to two parallel output branches (text detection branch, geometric regression branch), which together constitute the detection head of the PP-OCRv3 Chinese medical version. The two parallel branches obtain a text score map S and a four-point offset regression map , which provides basic data for text region extraction and positioning, and synchronously calculates a joint loss function to drive end-to-end optimization:

[0099] ;

[0100] wherein, is the text detection loss function, , is a balance coefficient (optimized through a validation set), is a weighted Dice Loss loss function, is a Smooth L1 Loss loss function, is a real text score map, is a real four-point offset regression map;

[0101] The calculation formula is:

[0102] ;

[0103] wherein S(x, y) is the original confidence of pixel point (x, y) belonging to a text region, is the real confidence of pixel point (x, y) belonging to a text region;

[0104] The calculation formula is:

[0105] ;

[0106] wherein, is the difference between the predicted offset and the real offset, is a smoothing constant;

[0107] The probability of each pixel point belonging to a text region is calculated using a differentiable binarization function on the text score map S:

[0108] ;

[0109] where P(x, y) is the text probability of pixel point (x, y), S(x, y) is the original confidence of pixel point (x, y) belonging to the text region, is the differentiable threshold learned in the training process, k is a scaling factor, which is set through experiments;

[0110] A continuous probability map P is obtained through the text probability of each pixel point, the text region mask is obtained by thresholding the continuous probability map P, and the text region mask is analyzed by an 8-connected domain analysis method to find all connected regions, each connected region represents a single text block, and is regarded as an independent text instance region. The four-point offset regression map corresponding to the pixels in each text instance region , that is, the offset of the pixel point, is weighted and averaged to decode the four-point coordinates of the text instance region:

[0111] ;

[0112] wherein, is the four-point coordinate, is the circumscribed quadrilateral of the text block, represents the boundary of the text region, is the 4th vertex coordinate of the text block i;

[0113] The calculation formula is:

[0114] ;

[0115] ;

[0116] wherein, j is the four vertices of the quadrilateral, , are the offsets of the jth vertex in the x and y directions, respectively;

[0117] For each four-point coordinate , the height and width of the minimum bounding rectangle are calculated:

[0118] ;

[0119] ;

[0120] wherein, is the height of the minimum bounding rectangle, is the minimum bounding rectangle width;

[0121] Set the threshold ( determined by the effective text height distribution statistics on the medical image validation set), when the height of the circumscribed rectangle is less than the threshold If the area is too small, it is considered that the region is not an effective text block and is discarded, and a perspective transformation (for converting an arbitrary quadrilateral region in an image into a rectangular region) is performed on each effective text block:

[0122] ;

[0123] where H is a homography matrix calculated by four vertex coordinates of the source quadrilateral and four vertex coordinates of the target rectangle, is a homogeneous coordinate;

[0124] The final augmented image is subjected to a perspective transformation using the homography matrix H to generate a normalized rectangular image block, and the calculation formula is:

[0125] ;

[0126] wherein, is the normalized rectangular image block, is the size of the target rectangle, i.e., the size of the minimum circumscribed rectangle;

[0127] Each normalized image block is input into the SVTR recognition network (composed of a convolutional layer and a visual Transformer encoder), the local features of the normalized image block are extracted by the convolutional layer, and the calculation formula of the convolutional layer is:

[0128] ;

[0129] wherein, is the feature map output by the convolutional layer, is a pixel (x, y) of the feature map, K is a convolution kernel, and * is a convolution operation;

[0130] The feature map output by the convolutional layer is input into the visual Transformer encoder, which effectively models the long-range dependency relationship between characters through a self-attention mechanism, can better understand and recognize the context information in the character sequence, and the SVTR network outputs the character sequence of each text block and the single-character confidence of each character, wherein, is the predicted character sequence of the text block i, is the number of characters of the text block i, is the u-th predicted character, and the corresponding loss function is calculated:

[0131] ;

[0132] wherein, is the text recognition output loss function,​ is a real character sequence, is the u-th real character, CE is cross-entropy, is a CTC loss function, , is a balance coefficient;

[0133] The calculation formula is:

[0134] ;

[0135] wherein, is a mapping function for removing duplicate characters and white spaces, is an alignment path composed of characters or white spaces, is the length of the time step of the SVTR network output, is the probability of the model predicting the character at the time step , is a set of all legal paths that can be mapped to the character sequence;

[0136] The total loss function is calculated by weighted summation of the text detection loss function and the text recognition output loss function , and the gradient is calculated by the back propagation algorithm to minimize the total loss function to update all learnable parameters (including the backbone network, the detection head, and the text recognition), and finally the PP-OCRv3 model optimized by training outputs the final character sequence of each text block and the single character confidence of each character, wherein, is the predicted character sequence of the final text block i, is the number of characters of the final text block i, is the final u-th predicted character, and to evaluate the reliability of each line of text recognition, the average confidence of each text block is calculated:

[0137] ;

[0138] wherein, is the output single character confidence, is the average confidence of the text block i;

[0139] By fuzzy logic, the threshold is set, and when the average confidence is greater than or equal to the threshold If the result is positive, the recognition is considered reliable; otherwise, it is recorded as low-confidence text for subsequent verification. The detection and recognition results are combined in spatial order (from top to bottom, from left to right) to form a complete structured text sequence. , where N is the total number of detected text blocks.

[0140] This method employs single-scale Retinex (SSR) to correct uneven illumination, combined with CLAHE, morphological closing operations, median filtering, and Sobel edge enhancement to effectively repair character breaks, suppress noise, and strengthen edges, ensuring the integrity and clarity of the text structure. It utilizes a medically fine-tuned PP-OCRv3 model to achieve high-precision text detection and performs geometric normalization on text blocks in any direction through perspective transformation. By fusing local features with Transformer long-range context modeling capabilities through an SVTR recognition network, it accurately identifies complex medical terms and implements result reliability grading based on confidence thresholds. This effectively addresses practical challenges such as uneven illumination, character blurring, and skew distortion, significantly improving the automated structuring capabilities of medical documents such as drug packaging, test reports, and handwritten prescriptions.

[0141] Furthermore, based on the layout patterns and geometric distribution of medical documents, structured fields are extracted, which refers to the structured text sequence. According to the upper boundary of each text block Arrange the text in ascending order to obtain a top-to-bottom text sequence, assuming the vertical spacing between adjacent text blocks is:

[0142] ;

[0143] in, It is the vertical spacing;

[0144] Setting thresholds through Bayesian optimization When vertical spacing Less than the threshold If the two are in the same logical line, then the horizontal spacing between adjacent boxes is calculated for text blocks within the same logical line.

[0145] ;

[0146] in, It refers to the horizontal spacing;

[0147] Thresholds are set through statistical analysis. When the horizontal spacing Less than the threshold When the text blocks are concatenated into a single complete text, a logical line set is obtained. ,in, It is a logical line. It is a logical row index, where M is the total number of logical rows. is the character sequence of the i-th logical line text block, is the four-point coordinate of the i-th logical line text block, extracts the left boundary coordinate and the right boundary coordinate of all text blocks from each logical line , respectively, one-dimensional Gaussian smoothing is performed on all left boundary coordinates and right boundary coordinates ; ;

[0148] ;

[0149] ;

[0150] where f(x) is the smoothing curve of all left boundary coordinates, g(x) is the smoothing curve of all right boundary coordinates, is the smoothing parameter;

[0151] detects the local maximum of the left and right smoothing curves f(x) and g(x), and takes the maximum peak position as the "item column center" , "result column center" , according to the geometric prior of the three-column symmetry distribution of the medical test sheet, assuming that the image width is W, the reference range column center is calculated:

[0152] ;

[0153] where is the reference range column center;

[0154] For each text block in the logical line, the distance between the left boundary and the three column centers is calculated:

[0155] ;

[0156] where is the distance, is the "item column", is the "result column", is the "reference range column";

[0157] ​Assigning text blocks to the column with the minimum distance, determining the field type, matching item column text with a medical indicator dictionary (e.g., "ALT", "AST", "white blood cell count", etc.), unifying indicator naming, extracting numerical values and units from result column text, converting intervals in reference range column text to numerical intervals, and combining each row of information into a standardized structure Record={type, items:[{name, value, unit, ref}]} where name is the test item name, value is the test result numerical value, unit is the unit, and ref is the lower and upper limits of the item reference interval.

[0158] By logical line reconstruction, adaptive column center positioning, and spatial field classification, multi-column medical test reports can be accurately parsed without templates. Combined with medical dictionary standardization and numerical structuring, the problems of layout differences, text fragmentation, and inconsistent naming are effectively solved, significantly improving the accuracy of field recognition and the quality of structured output.

[0159] S3. Perform deterministic rule judgment and risk assessment on structured data, and generate a graphic-text report based on the risk assessment results.

[0160] Specifically, deterministic rule judgment and risk assessment on structured data are based on the standardized structure Record={type, items:[{name, value, unit, ref}]} combined from each row of information and the user's basic health profile U={age, sex, allergy}, where age is age, sex is gender, and allergy is basic information such as allergy history, which is entered by the user in advance. For each indicator item (name, value, ref) in the structured data, quantitative abnormality determination is performed:

[0161] ;

[0162] where flag is the abnormality identification result, value is the actual test value, ref[0] and ref[1] are the lower and upper limits of the reference range, respectively;

[0163] In the determination of abnormal indicators, deterministic IF-THEN rule matching is performed according to a medical rule library (based on the experience of medical experts, typical risk patterns are predefined):

[0164] ;

[0165] where Rl is the lth rule, is the logical condition of the rule, is the risk level output;

[0166] ​According to the matched risk level, the corresponding medical explanation text is automatically mapped to form a readable result.

[0167] By combining user health portraits with test indicators for individualized abnormality judgment and performing risk inference based on a medical rule base, precise mapping from numerical abnormalities to clinical risks is realized, and readable medical explanations are automatically matched, which not only improves the clinical relevance and accuracy of the judgment, but also enhances the explainability of the results and the user trust. It effectively supports primary diagnosis and treatment and patient self-health management.

[0168] Further, according to the risk assessment result, a picture-text report is generated by using a preset natural language report template, which is used to map the structured fields of each test item to a textual description. For each test item, the readable result is replaced according to the template placeholder to generate a complete natural language sentence. According to the risk level of all test items, a health suggestion text is mapped (normal → "Your indicators are all within the normal range. Please maintain good living habits.", mild risk → "Some indicators are mildly abnormal. It is recommended to maintain regular work and rest, avoid high-fat diet, and review in a week.", obvious risk → "The test result is obviously abnormal. It is recommended to seek medical treatment for further examination as soon as possible."), all natural language sentences and health suggestions are combined to form a complete report structure. The report is displayed in the form of a card on the user interface of the mobile terminal, and each test item corresponds to a result card (including item name, value and reference range, abnormality identification color, medical explanation and suggestion). The "Identified incorrectly? Click to feedback" entry is provided at the bottom of each card, and the user can manually annotate and correct the identification error.

[0169] By generating a natural language report through a template and matching personalized health suggestions according to the risk level, the structured test results are converted into a text that is easy to understand and clinically reasonable, which significantly improves patient understanding. At the same time, through the user feedback entry, not only the interaction trust is enhanced, but also a data closed loop for model continuous optimization is built.

[0170] The embodiment also provides an OCR-based medical document intelligent identification system, which comprises:

[0171] A document image acquisition module is used to place the document under uniform illumination, align the shooting and avoid reflection to obtain a high-quality RGB image;

[0172] A document segmentation enhancement module is used to generate a binary image based on U-Net and attention mechanism, and the main body area is retained through rotation and cropping;

[0173] A text enhancement recognition module is used to fuse SSR, CLAHE and edge enhancement to improve image quality, and output a text sequence with confidence through a detection-recognition model;

[0174] a structured field extraction module configured to distribute the text to the item, result and reference range fields according to the medical document layout rule;

[0175] a risk assessment report module configured to combine the health portrait and medical rules to determine the abnormality and generate a graphic risk report with explanation and suggestion.

[0176] The embodiment further provides a computer device suitable for the case of the medical document intelligent recognition method based on OCR, including a memory and a processor; the memory is configured to store computer executable instructions, and the processor is configured to execute the computer executable instructions to implement the medical document intelligent recognition method based on OCR proposed in the above embodiment.

[0177] The computer device can be a terminal, and the computer device includes a processor, a memory, a communication interface, a display screen and an input device connected through a system bus. The processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operating system and the computer program in the non-volatile storage medium to run. The communication interface of the computer device is configured to perform wired or wireless communication with an external terminal. The wireless communication can be achieved through WIFI, a carrier network, NFC (Near Field Communication) or other technologies. The display screen of the computer device can be a liquid crystal display screen or an electronic ink display screen. The input device of the computer device can be a touch layer overlaid on the display screen, or a key, trackball or touchpad arranged on the shell of the computer device, or an external keyboard, touchpad or mouse, etc.

[0178] The embodiment further provides a storage medium having a computer program stored thereon, the program being executed by a processor to implement the medical document intelligent recognition method based on OCR proposed in the above embodiment. The storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as a static random access memory (SRAM), an electrically erasable programmable read-only memory (EEPROM), an erasable programmable read-only memory (EPROM), a programmable read-only memory (PROM), a read-only memory (ROM), a magnetic storage, a flash memory, a magnetic disk or an optical disk.

Claims

1. An OCR-based intelligent recognition method for medical documents, characterized in that: Comprising, Collect medical document images through a mobile terminal, generate a binary image based on the collected images through a U-Net combined with multi-scale feature fusion and attention mechanism, and perform cropping; Based on the cropped binary image, extract text information through an OCR model, and extract structured fields according to the layout rules and geometric distribution of medical documents; Perform deterministic rule judgment and risk assessment on structured data, and generate a graphic report according to the risk assessment results; Convert an RGB color image into a single-channel grayscale image using the ITU-R BT.601 standard grayscale conversion formula, input the grayscale image into a U-Net structure, and use ResNet50 as the encoder backbone network. After the ResNet50 encoder outputs the feature map, the CBAM module is used to enhance the channel and spatial attention of the feature map respectively, and the enhanced feature map is obtained. The enhanced feature map is used as the encoder output and is transmitted to the corresponding resolution decoder branch for upsampling to obtain the upsampled feature map. At the jump connection of the network, the enhanced feature map and the upsampled feature map are respectively mapped to the same channel dimension through 1x1 convolution, and the Attention Gate mechanism is used to calculate the spatial attention mask. Through spatial attention mask Generate fusion feature map, pass the fusion feature map through a sigmoid activation function Convert to a binary probability map, and obtain a loss function using a weighted combination of Dice loss and BCE loss , calculate the gradient of the loss with respect to each weight using the chain rule, and adjust all learnable parameters in the network based on the gradient to reduce the loss function value, repeat iteration until the maximum number of iterations is reached, and obtain the final binary probability map based on the trained network , set a threshold When the probability value of the pixel point (x, y) is greater than or equal to the threshold , output white, otherwise black, generate a binary image, perform Hough line transformation on the binary image to obtain a rotation-corrected binary image, and perform contour detection to obtain a cropped binary image; The single-scale Retinex algorithm is used for contrast enhancement on the cropped binary image, and the preliminarily enhanced image is sequentially subjected to adaptive histogram equalization, morphological closing operation, median filtering, Sobel edge enhancement and background suppression to obtain a final enhanced image to the PP-OCRv3 model, and the model extracts multi-scale features through a convolutional backbone network, and the extracted features are input into two parallel output branches to obtain a text score map S and a four-point offset regression map , and a text detection loss function is calculated, and a differentiable binarization function is used to calculate the probability of each pixel point belonging to a text region for the text score map S. A continuous probability map P is obtained by the text probability of each pixel point, the text region mask is obtained by thresholding the continuous probability map P, and the text region mask is analyzed by an 8-connected domain analysis method to find all connected regions, i.e., text instance regions, and a four-point offset regression map corresponding to the pixels in each text instance region is obtained , the four-point coordinates of the text instance region are obtained by decoding, and the height and width of the minimum bounding rectangle are calculated, and a threshold is set When the height of the minimum bounding rectangle is less than the threshold , it is considered that the region is too small and is not an effective text block, and the perspective transformation is performed on each effective text block to generate a standardized rectangular image block Each standardized image block is input into the SVTR recognition network to output the character sequence of each text block, and a text recognition output loss function is calculated, a total loss function is calculated in combination with a text detection loss function, a gradient is calculated by a back propagation algorithm to minimize the total loss function, all learnable parameters are updated, and the final character sequence of each text block is output, the average confidence of each text block is calculated, and a threshold is set When the average confidence is greater than or equal to the threshold , it is considered that the recognition result is reliable, otherwise it is recorded as low-confidence text, and the detection and recognition results are combined in spatial order to form a complete structured text sequence; The structured text sequence is arranged in ascending order according to the upper boundary of each text block, and the vertical spacing between adjacent text blocks is calculated The threshold is set The vertical spacing between adjacent text blocks is calculated When the vertical spacing is less than the threshold , it is considered that they belong to the same logical line The horizontal spacing between adjacent frames in the same logical line is calculated The threshold is set When the horizontal spacing is less than the threshold , the contents of adjacent text blocks are spliced into a complete text, and finally a logical line set is obtained The left boundary coordinates and the right boundary coordinates of all text blocks in each logical line are extracted, and one-dimensional Gaussian smoothing is performed on all left boundary coordinates and right boundary coordinates to obtain left and right smooth curves 、 ; Detect local maxima of left and right smooth curves , Take the maximum peak position as the "item column center" , "result column center" Calculate the reference range column center according to the geometric prior of the three-column symmetry distribution of medical test sheets Calculate the distance between the left boundary of each logical row text block and the three column centers, assign the text block to the column with the smallest distance to determine the field type, match the item column text with the medical indicator dictionary, unify the indicator naming, extract the numerical value and unit from the result column text, convert the interval in the reference range column text to a numerical interval, and combine each row of information into a standardized structure.

2. The OCR-based medical document intelligent recognition method of claim 1, wherein: The deterministic rule judgment and risk assessment on structured data are based on the standardized structure of each row of information combined with the user's basic health profile. For each indicator item in the structured data, quantitative anomaly judgment is performed. In the determined abnormal indicators, deterministic IF-THEN rule matching is performed according to the medical rule base. According to the risk level obtained by matching, the corresponding medical explanation text is automatically mapped to form a readable result. 3.The OCR-based medical document intelligent identification method of claim 2, wherein: The generation of a graphic report according to the risk assessment results refers to a pre-set natural language report template. For each test item, the readable result is replaced according to the template placeholder to generate a complete natural language sentence. According to the risk level mapping health suggestion text of all test items, all natural language sentences and health suggestions are combined to form a complete report structure. The report is displayed in the form of a card on the user interface of the mobile terminal.

4. The OCR-based medical document intelligent recognition method of claim 3, wherein: The medical document image collection through the mobile terminal refers to arranging uniform illumination light sources in an indoor environment and collecting color images through the rear main camera of the mobile terminal.

5. The system for intelligent recognition of medical documents based on OCR, according to any one of claims 1 to 4, characterized in that: Comprising, A document image collection module for placing the document under uniform illumination, aligning the shooting and avoiding reflection to obtain high-quality RGB images; A document segmentation enhancement module for generating a binary image based on U-Net and attention mechanism, and retaining the main body area after rotation and cropping; A text enhancement and recognition module for fusing SSR, CLAHE and edge enhancement to improve image quality, and outputting a text sequence with confidence through a detection-recognition model; A structured field extraction module for distributing text to item, result and reference range fields according to the layout rules of medical documents; A risk assessment report module for combining health profile and medical rule judgment to generate a graphic risk report with explanation and suggestion. 6.A computer device, comprising a memory and a processor, wherein the memory stores a computer program, and the computer device is characterized in that: The computer program is executed by the processor to implement the steps of the method for intelligent recognition of medical documents based on OCR according to any one of claims 1-4.

7. A computer readable storage medium having stored thereon a computer program, characterized in that: The computer program is executed by the processor to implement the steps of the method for intelligent recognition of medical documents based on OCR according to any one of claims 1-4.

Images