Text detection and recognition method for medical bill image

Abstract

The invention relates to the technical field of text detection and recognition of images, in particular to a text detection and recognition method for medical bill images. The method comprises the following steps: S1, performing feature enhancement on a medical bill image through a feature enhancement module, and reconstructing a table structure of the medical bill image; s2, after the table structure is reconstructed, feature fusion of the medical bill image is further carried out, and a pre-detection table image is formed; s3, integrating a pre-detection table image and a visual language pre-training model MDETR to enhance feature expression; s4, recognizing a text region from the pre-detection table image by using a connection time sequence classification CTC method improved through an Attention mechanism; and S5, further performing deep analysis on the detected text region through a fusion model of the SVTR network and the PP-HGNet. According to the invention, through feature enhancement, table structure reconstruction, feature fusion and integration with a visual language pre-training model MDETR, a text region can be identified in a complex medical bill image.

Description

Technical Field

[0001] The present invention relates to the technical field of text detection and recognition of images, and in particular to a text detection and recognition method for medical bill images. Background Art

[0002] The main ways to obtain image data of medical bills are scanned copies in actual scenes and photos taken by mobile devices such as mobile phones. Given that the resolution of mobile device cameras is generally limited, the captured images of medical bills are often mixed with various interference factors, such as stains, scratches, handwritten notes, and seal patterns. These interference factors are not only of various types, but may also obscure or cover important text information, posing a serious challenge to the accurate recognition of text detection algorithms, resulting in reduced algorithm accuracy and increased error rates. In addition, some images undergo a compression process when transmitted over the network, which will cause a decrease in image quality, further affecting the accuracy of subsequent image processing steps and text recognition; medical bills are key medical records, and their text layout is compact and diverse, with text content occupying more than 80% of the page, covering a wide range of information categories and integrating multiple languages ​​and symbol systems. Specifically, Chinese is used to record patient details, medical service items, etc.; English may appear in the names of medical institutions, doctor information, etc.; Greek letters are more common in medical terms and symbols; and mathematical formulas may appear in examination results or professional medical reports. The diversity and complexity of this text form increases the difficulty of identifying and processing medical bills, requiring targeted parsing and recognition strategies for various types of text. In addition, in specific application scenarios, such as instant settlement, online insurance review, etc., the image text recognition of medical bills needs to have a high degree of timeliness. In summary, a text detection and recognition method for medical bill images is provided. Summary of the invention

[0003] The purpose of the present invention is to provide a text detection and recognition method for medical bill images to solve the problems of complex medical bill images, poor image quality and diverse text forms raised in the above background technology.

[0004] To achieve the above object, the present invention provides a method for text detection and recognition of medical bill images, comprising the following steps:

[0005] S1. Perform feature enhancement on the medical bill image through a feature enhancement module, and reconstruct the table structure of the medical bill image;

[0006] S2, after reconstructing the table structure, further perform feature fusion of the medical bill image to form a pre-detection table image;

[0007] S3, integrating the pre-detection table image with the visual language pre-training model MDETR to enhance feature expression;

[0008] S4, using the connection temporal classification (CTC) method improved by the attention mechanism to identify the text area from the pre-detection table image;

[0009] S5. The detected text area is further deeply analyzed through the fusion model of SVTR network and PP-HGNet.

[0010] As a further improvement of the technical solution, in S1, the feature enhancement module is used to enhance the features of the medical bill image, including the following steps:

[0011] S1.1, converting a color medical bill image into a grayscale medical bill image, and adjusting a histogram of the medical bill image;

[0012] S1.2, smoothing the medical bill image by using a Gaussian filtering algorithm to remove high-frequency noise, preliminarily extracting the area containing the table structure from the processed grayscale medical bill image, and dividing the extracted area into multiple smaller area blocks;

[0013] S1.3, enhance the contrast of medical bill images by histogram stretching method, and use Laplacian operator to sharpen the image edges;

[0014] S1.4, use dilation operation to enhance horizontal line segments in the image;

[0015] S1.5, dynamically adjust the threshold according to the local characteristics of the medical bill image and perform adaptive binarization processing;

[0016] S1.6, applying Hough transform in each area block to detect and reconstruct horizontal line segments, performing cluster analysis on the detected line spacing using K-means clustering algorithm, and calculating the mean line spacing;

[0017] S1.7. Set a threshold u based on the mean line spacing, use the dilation operation to further extract horizontal line segments, set a threshold h based on the mean line spacing, detect vertical line segments, merge the detected horizontal line segments with the vertical line segments, and reconstruct the table structure of the medical bill image.

[0018] As a further improvement of the technical solution, in S1.2, the Gaussian filtering algorithm is specifically:

[0019]

[0020] Among them, p represents the offset of the pixel on the x-axis; r represents the offset of the pixel on the y-axis; σ represents the standard deviation of the Gaussian distribution; G(p, r, σ) represents the weight of each pixel and its neighborhood in the medical bill image when performing Gaussian filtering.

[0021] As a further improvement of the technical solution, in S1.3, the Laplace operator is specifically:

[0022]

[0023] Wherein, f(x,y) represents the gray value of the medical bill image at the coordinate (x,y); x represents the x-axis in the two-dimensional space; y represents the y-axis in the two-dimensional space; f represents the function in the two-dimensional space; represents partial derivative.

[0024] As a further improvement of the technical solution, in S1.5, the threshold is dynamically adjusted according to the local characteristics of the medical bill image to perform adaptive binarization processing, including the following steps:

[0025] S1.51, select a window size of 5×5, and for each pixel in the medical bill image, use the selected window to calculate the local average brightness around the pixel;

[0026] S1.52, calculating an offset value, and taking the local average brightness plus the offset value as the threshold b of the local area;

[0027] S1.53, compare the gray value of each pixel with the calculated threshold of the local area, if it is greater than the threshold b, set it to white, otherwise set it to black;

[0028] S1.54. Repeat steps S1.51 to S1.53 for the entire image until all pixels have been processed.

[0029] As a further improvement of the technical solution, in S1.6, the application of Hough transform to detect straight lines in the image includes the following steps:

[0030] S1.61, use standard Hough transform to detect straight lines;

[0031] S1.62, traverse each edge point in the medical bill image, and accumulate votes for the line representing the edge point in the parameter space;

[0032] S1.63. For each edge point, calculate all line parameter combinations and increase the count in the corresponding parameter space;

[0033] S1.64, find local maximum points in parameter space;

[0034] S1.65. Draw a straight line in the original medical bill image based on the detected local maximum point.

[0035] As a further improvement of the technical solution, in S2, after reconstructing the table structure, further feature fusion is performed, including the following steps:

[0036] S2.1, further process the image using a custom nuclear mask;

[0037] S2.2, applying the dilation operation to the pre-extracted area block again to extract horizontal line segments;

[0038] S2.3, applying the standard Hough transform again in the pre-extracted area block to reconstruct the horizontal line segment;

[0039] S2.4, perform cluster analysis again on the reconstructed horizontal line segments to calculate the mean value of the line distance;

[0040] S2.5, set a threshold based on the mean value of line spacing, extract vertical line segments, and further improve the table structure;

[0041] S2.6, divide the region into blocks again and determine the region threshold m;

[0042] S2.7, the features of each block area are fused, and the contours are pre-extracted to form a pre-detection table image.

[0043] As a further improvement of the technical solution, in S3, the pre-detection table image is integrated with the visual language pre-training model MDETR to enhance feature expression, including the following steps:

[0044] S3.1, using the formed pre-detection table image as input, performing standardization processing on the pre-detection table image;

[0045] S3.2, inputting the pre-processed pre-detection table image into the MDETR model;

[0046] S3.3, interacting the image features and the text features, and encoding the image features;

[0047] S3.4, decode the encoded and interacted features and output the detection results;

[0048] S3.5, post-processing the target detection results output by the model MDETR through the non-maximum suppression NMS method;

[0049] S3.6. Use the detection results provided by the model MDETR to further confirm and refine the boundaries of the table and the location of the cells.

[0050] As a further improvement of the technical solution, in S4, the text area is identified from the pre-detection table image using a connection temporal classification CTC method improved by the Attention mechanism, including the following steps:

[0051] S4.1, input pre-detection table image;

[0052] S4.2, use convolutional neural network (CNN) to extract features from images;

[0053] S4.3. Input the feature map into the recurrent neural network RNN, and add the Attention mechanism based on the neural network RNN;

[0054] S4.4, generate an output sequence and calculate the CTC loss between the output sequence and the true label;

[0055] S4.5. Use a greedy strategy to decode the output sequence, determine the text area in the image based on the decoding results, and identify it.

[0056] As a further improvement of the technical solution, in S5, the detected text area is further deeply analyzed through the fusion model of the SVTR network and PP-HGNet, including the following steps:

[0057] S5.1, inputting the detected text region image, and preprocessing each text region image;

[0058] S5.2, concatenating the feature map generated by the SVTR network with the feature map generated by the PP-HGNet to form a new feature map;

[0059] S5.3, use the attention mechanism to further process the concatenated feature map to generate the final feature representation;

[0060] S5.4. Use the CTC decoder to convert the final feature representation into a text sequence, and post-process the decoded text sequence.

[0061] Compared with the prior art, the present invention has the following beneficial effects:

[0062] 1. In this text detection and recognition method for medical bill images, through feature enhancement, table structure reconstruction, feature fusion, and integration with the visual language pre-training model MDETR, this method can accurately identify text areas in complex medical bill images. Histogram adjustment, Gaussian filtering, Laplace sharpening and other technologies in the feature enhancement step help to improve image quality and make the text clearer and more legible. In addition, through adaptive binarization and Hough transform, the table structure can be effectively detected, thereby improving the accuracy of text area detection. The improved CTC method combined with the Attention mechanism further improves the accuracy of text recognition, especially when dealing with blurred or handwritten text.

[0063] 2. In this text detection and recognition method for medical bill images, unnecessary computational load is reduced through noise reduction and smoothing operations in the preprocessing stage, and the robustness of the system is enhanced through the application of feature fusion and multimodal models. Using a 5x5 window for adaptive binarization, the threshold can be adjusted according to the local environment to better adapt to images under different lighting conditions. In addition, deep analysis through the fusion model of the SVTR network and PP-HGNet can more effectively handle diverse medical bill formats and maintain a high recognition rate under different writing styles. These steps not only improve the processing speed, but also enable the system to maintain stable performance when faced with a variety of different inputs. BRIEF DESCRIPTION OF THE DRAWINGS

[0064] Figure 1 The figure is a flow chart of the overall method of the present invention. DETAILED DESCRIPTION

[0065] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0066] Example: See Figure 1 As shown, this embodiment provides a text detection and recognition method for medical bill images, comprising the following steps:

[0067] S1. Perform feature enhancement on the medical bill image through a feature enhancement module, and reconstruct the table structure of the medical bill image;

[0068] Among them, the feature enhancement module EFEM is a U-shaped cascaded module. It is equipped with four-input and four-output channels, and is designed for enhancing and fusing multiple feature maps. In the text detection solution, EFEM is the core force, especially in multi-scale feature enhancement. To achieve top performance, EFEM is carefully built in three stages: the first is upper scale enhancement, which broadens the feature field of view and makes the details fuller; followed by the pyramid squeeze attention module (PSA), which uses the attention mechanism to highlight key features and filter out noise; the last is lower scale enhancement, which refines and integrates features and strengthens the discrimination ability;

[0069] In this embodiment, the feature enhancement module is used to enhance the features of the medical bill image, including the following steps:

[0070] S1.1, converting the color medical bill image into a grayscale medical bill image to reduce the computational complexity, and adjusting the histogram of the medical bill image to enhance the contrast so that the text in the image is clearer;

[0071] S1.2, smoothing the medical bill image by using a Gaussian filtering algorithm to remove high-frequency noise, preliminarily extracting the area containing the table structure from the processed grayscale medical bill image, and dividing the extracted area into multiple smaller area blocks so that each part can be processed more finely;

[0072] Among them, Gaussian filtering can improve the overall visual effect of the image, making the image look clearer and more natural. This is very helpful for subsequent image processing steps (such as edge detection, feature extraction, etc.); Gaussian filtering is applicable to a variety of image processing scenarios, whether it is processing natural images or document images, it can provide good smoothing effects, especially when processing medical bill images with more text, it can effectively improve the readability and recognition rate of the text; images processed by Gaussian filtering can better perform subsequent processing steps such as edge detection and feature extraction, because at this time the important features in the image (such as text, table lines, etc.) will be more obvious, and interference factors will be weakened; The specific Gaussian filtering algorithm is:

[0073]

[0074] Where p represents the offset of the pixel on the x-axis; r represents the offset of the pixel on the y-axis; σ represents the standard deviation of the Gaussian distribution; G(p, r, σ) represents the weight of each pixel and its neighborhood in the medical bill image when performing Gaussian filtering;

[0075] S1.3, enhance the contrast of medical bill images by histogram stretching method, and use Laplacian operator to sharpen the image edges;

[0076] Among them, the histogram stretching method increases the contrast of the image by remapping the grayscale of the image and expanding the brightest and darkest pixel values ​​in the image to the maximum and minimum grayscale ranges respectively;

[0077] Furthermore, the Laplace operator is a commonly used edge detection operator that can highlight the edges and details in the image. By applying the Laplace operator, the edge information of the object in the image can be enhanced, making the edge more obvious, which is conducive to subsequent feature extraction and recognition; sharpening the image edge can increase the contrast of the image and make the image look clearer, which is particularly important for reading and identifying the details in the image, especially in the medical bill image, where sharpening can make the text and table lines clearer; for some slight blurring that may occur during the shooting process, Laplace sharpening can play a certain compensatory role. By sharpening the edge, the impact of blur can be reduced and the image quality can be improved; the Laplace operator is specifically:

[0078]

[0079] Wherein, f(x,y) represents the gray value of the medical bill image at the coordinate (x,y); x represents the x-axis in the two-dimensional space; y represents the y-axis in the two-dimensional space; f represents the function in the two-dimensional space; represents partial derivative;

[0080] S1.4. Use dilation to enhance horizontal line segments in the image. This step can help remove noise and strengthen the line structure inside the table.

[0081] The dilation operation is used to expand the white area (i.e., foreground) in the image, making the lines in the image thicker, thereby enhancing the horizontal line segments. The basic principle of dilation is to scan the image using a structural element and compare the structural element with each pixel in the image to determine the new pixel value;

[0082] S1.5, dynamically adjust the threshold according to the local characteristics of the medical bill image and perform adaptive binarization processing;

[0083] Furthermore, the threshold is dynamically adjusted according to the local characteristics of the medical bill image to perform adaptive binarization processing, including the following steps:

[0084] S1.51. Select a window size of 5×5. This window will be used to scan each pixel in the image. For each pixel in the medical bill image, use the selected window to calculate the local average brightness around the pixel.

[0085] If I(x, y) represents the gray value of the image at the coordinate (x, y), and W represents the window size (5x5), then the local average brightness M(x, y) is expressed as:

[0086]

[0087] Where w represents the side length of the window; i represents the offset on the x-axis; i represents the offset on the y-axis;

[0088] S1.52, calculating an offset value, which is used to prevent foreground text that is too close to the background brightness from being misjudged as the background, and taking the local average brightness plus the offset value as the threshold b of the local area, where the offset represents a constant value;

[0089] S1.53, compare the gray value of each pixel with the calculated threshold of the local area. If it is greater than the threshold b, it is set to white (representing the foreground, usually text); otherwise, it is set to black (representing the background);

[0090] S1.54, repeat steps S1.51 to S1.53 for the entire image until all pixels have been processed;

[0091] S1.6. Apply Hough transform in each area block to detect and reconstruct horizontal line segments to enhance the recognition of table structure, perform cluster analysis on the detected line spacing through K-means clustering algorithm, and calculate the mean line spacing to determine the average row height to provide a reference for the table structure;

[0092] Furthermore, the Hough transform can still detect straight lines when there is noise in the image. Even if there are many noise points in the image, as long as there is a group of these points that approximately meet the straight line condition, the Hough transform can detect this straight line. This is very important for processing medical bill images in practice, because these images may be affected by various factors such as uneven lighting, shadows, stains, etc.; the Hough transform does not rely on continuous and complete straight lines, but is based on a cumulative voting mechanism of points. Even if the straight line is interrupted or partially missing, as long as enough points fall on the same straight line, the Hough transform can still detect the existence of this straight line, which is particularly useful for detecting table lines in medical bills, because the table lines may not be completely continuous or interrupted by text; by setting appropriate parameters, the Hough transform can accurately detect the position and direction of the straight line, which is very useful for occasions that require high-precision detection (such as small table lines in medical bills); applying the Hough transform to detect straight lines in an image includes the following steps:

[0093] S1.61, use standard Hough transform to detect straight lines;

[0094] S1.62, traverse each edge point in the medical bill image, and accumulate votes for the line representing the edge point in the parameter space;

[0095] S1.63. For each edge point, calculate all combinations of line parameters and increase the count in the corresponding parameter space, where the parameter space is a two-dimensional matrix representing the multidimensional space used to represent line parameters in the Hough transform;

[0096] Furthermore, the line parameter ρ = acos(θ) + bsin(θ); (a, b) represents the edge point coordinates; θ represents the angle between the line and the horizontal line;

[0097] S1.64. Find local maximum points in parameter space. These points correspond to the most likely straight lines in the image. Local maximum points are the points with the highest counts in a certain neighborhood.

[0098] S1.65, drawing a straight line in the original medical bill image according to the detected local maximum point;

[0099] Furthermore, the mean line spacing is:

[0100]

[0101] In the formula, μe represents the mean line spacing; C r represents a cluster, i.e. a group of similar row spacing values; i represents the index variable; d e Indicates the specific line spacing value;

[0102] S1.7. Set the threshold u based on the mean line spacing, use the dilation operation to further extract the horizontal line segments, set the threshold h based on the mean line spacing, detect the vertical line segments to completely reconstruct the table structure, merge the detected horizontal line segments with the vertical line segments, and reconstruct the table structure of the medical bill image.

[0103] S2, after reconstructing the table structure, further perform feature fusion of the medical bill image to form a pre-detection table image;

[0104] In this embodiment, after reconstructing the table structure, feature fusion is further performed, including the following steps:

[0105] S2.1. Further process the image using a custom kernel mask to remove noise or enhance specific features.

[0106] S2.2, applying the dilation operation to the pre-extracted area block again to extract horizontal line segments;

[0107] S2.3, applying the standard Hough transform again in the pre-extracted area block to reconstruct the horizontal line segment;

[0108] S2.4, perform cluster analysis again on the reconstructed horizontal line segments to calculate the mean value of the line distance;

[0109] S2.5, set a threshold based on the mean value of line spacing, extract vertical line segments, and further improve the table structure;

[0110] S2.6, perform regional block division again to ensure that the features within each block are more consistent, and determine the threshold m for further processing within each block area;

[0111] S2.7, fusing the features of each block area and pre-extracting the contour to enhance the recognition of the table structure and form a pre-detection table image;

[0112] Among them, context is crucial in semantic segmentation, which broadens the understanding of objects and scenes. Context and scale complement each other and provide key information for object recognition. In order to efficiently use this information, a multi-scale fusion strategy is adopted;

[0113] S3, integrating the pre-detection table image with the visual language pre-training model MDETR to enhance feature expression;

[0114] In this embodiment, the visual language pre-training model is deeply pre-trained on a wide range of visual language understanding tasks, thereby obtaining rich visual and semantic expression capabilities. When this pre-training model is combined with a segmentation module, it can fully demonstrate its powerful representation capabilities and further improve the robustness and detection accuracy of the text detection model. The experimental results show that the model exhibits excellent text detection results when processing medical bill images with low resolution and large noise interference, providing a solution to overcome the challenges of text detection in medical bill images. MDETR is a multimodal model that can process image and text information at the same time. By integrating MDETR, its joint understanding ability of images and texts can be utilized to enhance the understanding and positioning of complex text information in images. This multimodal information fusion helps to improve the accuracy and robustness of text detection. Through the cross-modal attention mechanism in MDETR, the interaction between image features and text features can be achieved. This mechanism allows the model to focus on key parts in the image and ignore irrelevant background information, thereby generating more accurate feature representations, which is very beneficial for improving the detection accuracy of text areas. Integrating the pre-detection table image with the visual language pre-training model MDETR to enhance feature expression includes the following steps:

[0115] S3.1. Using the formed pre-detection table images as input, which have been processed to enhance the table structure, the pre-detection table images are standardized, such as scaling, cropping, and pixel normalization, to meet the input requirements of the MDETR model;

[0116] S3.2, input the pre-processed pre-detection table image into the MDETR model. MDETR is a multimodal model that can process image and text data simultaneously for tasks such as object detection;

[0117] S3.3. Through the cross-modal attention mechanism in MDETR, image features and text features are interacted and image features are encoded;

[0118] S3.4, decode the encoded and interacted features and output the detection results;

[0119] S3.5, post-processing the target detection results output by the model MDETR through the non-maximum suppression NMS method;

[0120] The basic idea of ​​the NMS method is to compare the confidence scores of overlapping detection frames and retain the detection frame with the highest score, while suppressing (i.e. excluding) the detection frame with lower scores and higher overlap;

[0121] S3.6. Use the detection results provided by the model MDETR to further confirm and refine the boundaries of the table and the location of the cells.

[0122] S4, using the connection temporal classification (CTC) method improved by the attention mechanism to identify the text area from the pre-detection table image;

[0123] In this embodiment, the attention mechanism is used to optimize the training process of CTC (connected temporal classification). The attention mechanism is used to monitor and guide CTC to improve the accuracy and computational efficiency of text recognition. The attention mechanism is good at capturing the interdependence between characters and can accurately locate characters, thereby extracting more effective features. The fusion of CTC and the attention mechanism makes the model perform better in text learning, alignment and feature representation. It can flexibly focus on the key text area in the image and effectively filter out noise or interference factors that are not related to the recognition task. This method cleverly combines the high speed of the CTC decoder with the attention mechanism. The high precision of the mechanism ensures that the model achieves an ideal balance between efficiency and effectiveness; the attention module is used to optimize the CTC (connected temporal classification) training strategy so that the model can automatically focus on the key text area in the image, while effectively filtering out noise and interference factors that are not related to the recognition task. The introduction of this guided attention mechanism helps the model achieve more accurate text positioning and recognition, thereby improving the overall recognition accuracy. By exploring effective data enhancement technology, transfer learning or model compression methods, it is expected to achieve efficient text recognition under small sample conditions; the connection temporal classification CTC method improved by the Attention mechanism is used to identify the text area from the pre-detection table image, including the following steps:

[0124] S4.1, input pre-detection table images, which have been processed by feature enhancement, table structure reconstruction and feature fusion to form a clearer table structure image;

[0125] S4.2. Use convolutional neural network (CNN) to extract features from images. CNN can effectively capture local features in images and generate feature maps. The feature maps generated by CNN contain important information in the image. These feature maps will serve as input to the CTC model.

[0126] S4.3. Input the feature map into the recurrent neural network RNN, and add the Attention mechanism to the neural network RNN, so that the model can focus on the important parts of the feature map and ignore irrelevant information. The Attention mechanism calculates the importance weight of each position in the feature map, thereby enhancing the model's attention to key features;

[0127] S4.4. Generate output sequences that represent the predicted probability distribution at each time step and calculate the CTC loss between the output sequence and the true label;

[0128] Among them, the CTC loss between the output sequence and the true label is calculated as follows:

[0129] L = -logP(Q|Z);

[0130] Where Q represents the true label sequence, that is, the text sequence actually existing in the pre-detection table image; Z represents the output sequence of the model; P(Q|Z) represents the probability that a given output sequence is a true label sequence; L represents the loss between the output sequence and the true label;

[0131] S4.5, adopting a greedy strategy to decode the output sequence, determining the text area in the image according to the decoding result, and identifying it;

[0132] The basic idea of ​​the greedy strategy is to select the most likely output at each time step as the final output. This method ignores the global optimality of the entire sequence and only focuses on the best choice at the current moment;

[0133] S5, the detected text area is further deeply analyzed through the fusion model of SVTR network and PP-HGNet;

[0134] In this embodiment, the SVTR network focuses on sparse visual marker recognition and can effectively process characters in text areas and accurately recognize text even in complex backgrounds, while PP-HGNet uses a hierarchical structure to capture features of different scales, which is very beneficial for processing texts of different sizes. The combination of the two can capture more detailed information at multiple scales, thereby improving the accuracy of text recognition; SVTR reduces the computational complexity through a sparse sampling strategy, while PP-HGNet enhances the processing capability of images of different resolutions through a pyramid structure. The fusion of these two models can better cope with medical bill texts of different formats, sizes, and writings, and enhance the model's adaptability and recognition of various complex text logos; the detected text area is further deeply analyzed through the fusion model of the SVTR network and PP-HGNet, including the following steps:

[0135] S5.1, inputting the detected text region image, and preprocessing each text region image;

[0136] S5.2, concatenating the feature map generated by the SVTR network with the feature map generated by the PP-HGNet to form a new feature map;

[0137] S5.3, use the attention mechanism to further process the concatenated feature map to generate the final feature representation;

[0138] S5.4. Use the CTC decoder to convert the final feature representation into a text sequence, and post-process the decoded text sequence, such as removing repeated characters and correcting the format.

[0139] The above shows and describes the basic principles, main features and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited by the above embodiments, and the above embodiments and descriptions are only preferred examples of the present invention, and are not intended to limit the present invention. Without departing from the spirit and scope of the present invention, the present invention may have various changes and improvements, and these changes and improvements all fall within the scope of the present invention to be protected.

Claims

1. A text detection and recognition method for medical bill images, characterized by: The following steps are involved: S1. Perform feature enhancement on the medical bill image through a feature enhancement module, and reconstruct the table structure of the medical bill image; S2, after reconstructing the table structure, further perform feature fusion of the medical bill image to form a pre-detection table image; S3, integrating the pre-detection table image with the visual language pre-training model MDETR to enhance feature expression; S4, using the connection temporal classification (CTC) method improved by the attention mechanism to identify the text area from the pre-detection table image; S5. The detected text area is further deeply analyzed through the fusion model of SVTR network and PP-HGNet.

2. The text detection and recognition method for medical bill images according to claim 1 is characterized by: In S1, the feature enhancement module is used to enhance the features of the medical bill image, including the following steps: S1.1, converting a color medical bill image into a grayscale medical bill image, and adjusting a histogram of the medical bill image; S1.2, smoothing the medical bill image by using a Gaussian filtering algorithm to remove high-frequency noise, preliminarily extracting the area containing the table structure from the processed grayscale medical bill image, and dividing the extracted area into multiple smaller area blocks; S1.3, enhance the contrast of medical bill images by histogram stretching method, and use Laplacian operator to sharpen the image edges; S1.4, use dilation operation to enhance horizontal line segments in the image; S1.5, dynamically adjust the threshold according to the local characteristics of the medical bill image and perform adaptive binarization processing; S1.6, applying Hough transform in each area block to detect and reconstruct horizontal line segments, performing cluster analysis on the detected line spacing using K-means clustering algorithm, and calculating the mean line spacing; S1.

7. Set a threshold u based on the mean line spacing, use the dilation operation to further extract horizontal line segments, set a threshold h based on the mean line spacing, detect vertical line segments, merge the detected horizontal line segments with the vertical line segments, and reconstruct the table structure of the medical bill image.

3. The text detection and recognition method for medical bill images according to claim 2 is characterized by: In S1.2, the Gaussian filtering algorithm is specifically: Among them, p represents the offset of the pixel on the x-axis; r represents the offset of the pixel on the y-axis; σ represents the standard deviation of the Gaussian distribution; G(p, r, σ) represents the weight of each pixel and its neighborhood in the medical bill image when performing Gaussian filtering.

4. The text detection and recognition method for medical bill images according to claim 2 is characterized by: In S1.3, the Laplace operator is specifically: Wherein, f(x,y) represents the gray value of the medical bill image at the coordinate (x,y); x represents the x-axis in the two-dimensional space; y represents the y-axis in the two-dimensional space; f represents the function in the two-dimensional space; represents partial derivative.

5. The text detection and recognition method for medical bill images according to claim 2 is characterized by: In S1.5, the threshold is dynamically adjusted according to the local characteristics of the medical bill image to perform adaptive binarization processing, including the following steps: S1.51, select a window size of 5×5, and for each pixel in the medical bill image, use the selected window to calculate the local average brightness around the pixel; S1.52, calculating an offset value, and taking the local average brightness plus the offset value as the threshold b of the local area; S1.53, compare the gray value of each pixel with the calculated threshold of the local area, if it is greater than the threshold b, set it to white, otherwise set it to black; S1.

54. Repeat steps S1.51 to S1.53 for the entire image until all pixels have been processed.

6. The text detection and recognition method for medical bill images according to claim 2 is characterized by: In S1.6, the application of Hough transform to detect straight lines in an image comprises the following steps: S1.61, use standard Hough transform to detect straight lines; S1.62, traverse each edge point in the medical bill image, and accumulate votes for the line representing the edge point in the parameter space; S1.

63. For each edge point, calculate all line parameter combinations and increase the count in the corresponding parameter space; S1.64, find local maximum points in parameter space; S1.

65. Draw a straight line in the original medical bill image based on the detected local maximum point.

7. The text detection and recognition method for medical bill images according to claim 1 is characterized in that: In S2, after reconstructing the table structure, feature fusion is further performed, including the following steps: S2.1, further process the image using a custom nuclear mask; S2.2, applying the dilation operation to the pre-extracted area block again to extract horizontal line segments; S2.3, applying the standard Hough transform again in the pre-extracted area block to reconstruct the horizontal line segment; S2.4, perform cluster analysis again on the reconstructed horizontal line segments to calculate the mean value of the line distance; S2.5, set a threshold based on the mean value of line spacing, extract vertical line segments, and further improve the table structure; S2.6, divide the region into blocks again and determine the region threshold m; S2.7, the features of each block area are fused, and the contours are pre-extracted to form a pre-detection table image.

8. The text detection and recognition method for medical bill images according to claim 1 is characterized by: In S3, the pre-detection table image is integrated with the visual language pre-training model MDETR to enhance feature expression, including the following steps: S3.1, using the formed pre-detection table image as input, performing standardization processing on the pre-detection table image; S3.2, inputting the pre-processed pre-detection table image into the MDETR model; S3.3, interacting the image features and the text features, and encoding the image features; S3.4, decode the encoded and interacted features and output the detection results; S3.5, post-processing the target detection results output by the model MDETR through the non-maximum suppression NMS method; S3.

6. Use the detection results provided by the model MDETR to further confirm and refine the boundaries of the table and the location of the cells.

9. The text detection and recognition method for medical bill images according to claim 1 is characterized by: In S4, a connection temporal classification (CTC) method improved by an attention mechanism is used to identify a text area from a pre-detection table image, including the following steps: S4.1, input pre-detection table image; S4.2, use convolutional neural network (CNN) to extract features from images; S4.

3. Input the feature map into the recurrent neural network RNN, and add the Attention mechanism based on the neural network RNN; S4.4, generate an output sequence and calculate the CTC loss between the output sequence and the true label; S4.

5. Use a greedy strategy to decode the output sequence, determine the text area in the image based on the decoding results, and identify it.

10. The text detection and recognition method for medical bill images according to claim 1 is characterized by: In S5, the detected text area is further deeply analyzed by the fusion model of the SVTR network and PP-HGNet, including the following steps: S5.1, inputting the detected text region image, and preprocessing each text region image; S5.2, concatenating the feature map generated by the SVTR network with the feature map generated by the PP-HGNet to form a new feature map; S5.3, use the attention mechanism to further process the concatenated feature map to generate the final feature representation; S5.

4. Use the CTC decoder to convert the final feature representation into a text sequence, and post-process the decoded text sequence.

Images