A medical record text recognition method and system based on deep learning

Abstract

This application discloses a medical record text recognition method and system based on deep learning. The method preprocesses the original medical record image and uses the Canny edge detection algorithm and the Hough line detection algorithm to perform edge detection and angle correction on the image respectively; the text area of ​​the preprocessed image is annotated to obtain a text detection training data set; the image is detected using the DBNet network to obtain the bounding box position coordinates of the text area in the image; an improved SVTR model is constructed to extract features from the text detected area, train a text recognition model, and recognize the image to obtain the character content in the medical record image. After determining the image to be subjected to text recognition, the medical record image is preprocessed, and text detection is performed on the preprocessed image. The detected image is segmented according to the detection frame, and the single line of text is recognized to obtain the text recognition result. The text in the image is predicted to obtain the recognized text with extremely high accuracy.

Description

Technical Field

[0001] The present application relates to the field of image recognition technology, and specifically to a method and system for medical record text recognition based on deep learning. Background Art

[0002] Medical record text recognition is crucial for medical research and hospital quality control. Currently, most hospitals only provide scanned images of inpatient records and paper copies when completing digital medical record collection. For paper records, staff must first collect them using a camera to capture each page, generating an image or PDF file. Finally, they perform text recognition on the image or PDF file to extract key information.

[0003] Existing technologies for medical record text recognition face three challenges: First, it's impossible for workers to hold paper medical records perfectly horizontally when photographing them, resulting in tilted images. Second, medical record text images are large and densely packed, making it difficult for general detection and recognition models to accurately locate the text. Third, the unique nature of medical record text often includes many uncommon characters, making recognition more challenging. Misidentification or omissions in medical record text recognition are unacceptable to hospitals.

[0004] Therefore, how to improve the accuracy of medical record text recognition is a technical problem that needs to be solved urgently in this field. Summary of the Invention

[0005] In order to solve the above technical problems, this application proposes the following technical solutions:

[0006] In a first aspect, the embodiments of the present application provide a method for medical record text recognition based on deep learning, comprising:

[0007] The original medical record images were preprocessed, and the Canny edge detection algorithm and Hough line detection algorithm were used to perform edge detection and angle correction on the images respectively;

[0008] Label the text area of ​​the preprocessed image to obtain a text detection training dataset;

[0009] Use the DBNet network to detect the image and obtain the bounding box position coordinates of the text area in the image;

[0010] Build an improved SVTR model, extract features from the text detected area, and train the text recognition model;

[0011] The text recognition model is used to identify the image and obtain the character content in the medical record image.

[0012] In one possible implementation, the Canny edge detection algorithm is used to perform edge detection on the image, including:

[0013] Compute the strength of the gradient using a filter with a Gaussian differentiable function;

[0014] Use a 5×5 Gaussian filter to remove noise from the image;

[0015] The Sobel filter operation is used to obtain the first-order derivative, and the pixel edge gradient value and direction are calculated by the first-order derivative. The calculation formula is as follows:

[0016]

[0017]

[0018] in, and Represent the gradient of the image in the x and y directions respectively. For each pixel, the gradients in these two directions are calculated, then they are squared and added together, and finally the square root is taken. The result is the gradient intensity of the pixel, that is, the edge intensity Edge gradient (G); θ represents the angle between a vector and the x-axis, ranging from -180° to 180°, and Angle (θ) represents the calculated arc tangent value of the angle θ, where G x and G y represent the horizontal and vertical components of a vector respectively.

[0019] Then, the entire image is searched and filtered at the pixel level to exclude objects that are not related to the edge structure;

[0020] Define two opposite thresholds as high threshold and low threshold;

[0021] When the gradient intensity is greater than the high threshold, the pixel object is determined to be positive, that is, the edge area;

[0022] When the gradient intensity is less than the low threshold, the pixel object is judged to be negative, that is, it is a non-edge area;

[0023] When the gradient strength is between the high threshold and the low threshold, determine whether the pixel is connected to the valid edge. If so, the pixel is also marked as a valid edge.

[0024] Repeat the above steps until all image areas are traversed.

[0025] In one possible implementation, after edge detection is performed on the original medical record image, the process of using the Hough line detection algorithm to perform angle correction on the image includes:

[0026] Use polar coordinates to describe the straight line of the rectangular coordinate axis. For each x, y point on the straight line, the following equation is satisfied:

[0027] ρ=xcos(θ)+ysin(θ)

[0028] Where ρ is the shortest distance from the origin to the line, and θ is the distance between the x-axis and the distance line;

[0029] Quantize the parameter space into m×n units and set the accumulator matrix Q m×n ;

[0030] Assign an accumulator Q to each unit in the parameter space i,j , and set the initial value of the accumulator to zero;

[0031] In the parameter space, find the unit corresponding to θ and ρ, and add 1 to the accumulator of the unit, that is, Q i,j =Q i,j +1;

[0032] After all points in the rectangular coordinate system have gone through the above two steps, the value of each accumulator in the parameter space is checked. The θ corresponding to the unit with the largest accumulator value is the tilt angle of the image.

[0033] Use the imrotate function to rotate the image to obtain the angle-corrected image.

[0034] In one possible implementation, labeling the text area of ​​the preprocessed image to obtain a text detection training dataset includes:

[0035] First, the medical record images are screened, and those with blurred images, severe exposure, or underexposure are screened out;

[0036] Use the open source method PPOCRLabel to label medical record photos, use a rectangular box to mark the text on the photo and output the coordinates of the four corner points of the rectangular box and the meaning of the Chinese characters;

[0037] The output results are converted into ICDAR format files through scripts for subsequent training. The format mainly includes the coordinates of four points and text content.

[0038] In one possible implementation, the DBNet network is used to detect the image and obtain the coordinates of the bounding box of the text area in the image, including:

[0039] Adjust the dataset parameters for DBNet training. Set data_dir to the dataset path, label_file_list to the dataset annotation file, and the ratio of training set to test set is 9:1.

[0040] The image is input into the ResNet50 backbone network, and a 3×3 convolution operation is performed on the image to obtain feature maps with a size ratio of 1 / 2, 1 / 4, 1 / 8, 1 / 16, and 1 / 32 of the original image respectively;

[0041] Upsample by ×2 from top to bottom and then fuse with the feature map of the same size generated from bottom to top;

[0042] After fusion, 3×3 convolution is used to eliminate the aliasing effect of upsampling;

[0043] After FPN, four feature maps with sizes of 1 / 4, 1 / 8, 1 / 16, and 1 / 32 of the original image are obtained;

[0044] Upsample the four feature maps separately and unify them into a feature map of 1 / 4 size;

[0045] Finally, concat the four 1 / 4 size feature maps to obtain feature F;

[0046] F passes through the convolution layer, compresses the channel to 1 / 4 of the input, and then undergoes batch normalization and ReLU function activation to obtain the feature map shape;

[0047] The obtained feature map is deconvolved with a convolution kernel of 2×2. The resulting feature map shape is (batch, 256, 1 / 2W, 1 / 2H), which is half the size of the original image.

[0048] Then perform the deconvolution operation, the output feature map channel is 1, and the obtained feature map shape is (batch, W, H), which is the size of the original image.

[0049] Finally, after the sigmoid function, the probability map P and threshold map T are output;

[0050] The probability map P and the threshold map T are subjected to the differentiable binarization DB operation to obtain an approximate binary map;

[0051] Use contraction to construct labels;

[0052] Expand the original polygonal text area G to G d , the shrinkage offset D is calculated according to the following formula:

[0053]

[0054] Where L is the perimeter of the polygon, A is the area of ​​the polygon, and r is the shrinkage factor, which is empirically set to 0.4;

[0055] Shrink frame G s and expansion frame G dThe gap between them is regarded as the boundary of the text area, and the normalized distance between each pixel in this gap and the original image boundary G is calculated;

[0056] Then normalize the calculated value and divide it by the offset D. At this time, G s and G d The value on becomes 1, and then these values ​​are subtracted from 1. At this time, G s and G d The value range of the region is [0,1];

[0057] Finally, scale it again to get the final label.

[0058] In one possible implementation, the loss function of the DBNet network is the sum of the loss of the probability map, the loss of the binarization map, and the loss of the threshold map:

[0059] L=L s +α×L b +β×L t

[0060] Among them, L is the total loss, L b To approximate the loss of binary images, Dice loss is used, L s is the probability map loss. To balance the ratio of positive and negative samples, Dice loss with OHEM is used for difficult sample mining. Positive sample: negative sample = 1:3, L t is the threshold graph loss, using the L1 distance between the predicted value and the label, and α and β are weight coefficients, set to 1 and 10 respectively;

[0061]

[0062] Among them, S l Indicates that OHEM is used for sampling, with a positive-negative sample ratio of 1:3;

[0063] L t Calculate the L1 distance between the predicted value and the label:

[0064]

[0065] Among them, R d is the extended polygon G d A set of indices of the inner pixels; y * is the label of the threshold map.

[0066] In one possible implementation, constructing an improved SVTR model, extracting features from the text-detected region, and training a text recognition model include:

[0067] Input the dataset after text detection and segment the image according to the detection label;

[0068] Generate training and test sets for the new images in a 9:1 ratio, set data_dir to the new dataset path, and point label_file_list to the new dataset annotation file;

[0069] The image is input into the SVTR backbone network with an input image size of H×W×3. Two consecutive 3×3 convolutions and batch normalization are used to achieve improved patch embedding. Character components CC0 of different sizes, which are used to represent character strokes;

[0070] In the first stage, CC0 is reshaped to hw×d0 size, and then layer normalization LN is applied to CC0 before being fed into global mixing or local mixing. The settings of global mixing and local mixing modules vary depending on the application scenario.

[0071] Next, layer normalization and a multi-layer perceptron are applied for feature fusion to reshape the hybrid block into an embedding of h×w×d0. Then, a 3×3 convolution with a stride of 2 is used on the height dimension and a convolution with a stride of 1 is used on the width dimension, followed by layer normalization to produce a h×w×d0 embedding.

[0072] The size of the character component CC1 after the first stage is The second stage repeats the operation of the first stage, and the size of the character component CC2 is In the last stage, the merging operation is replaced by the pooling operation, which first pools the height dimension to 1, and then passes through a fully connected layer, nonlinear activation and dropout to compress the character components into a size of Feature sequence C;

[0073] Finally, parallel linear prediction is performed on C and duplicate removal is performed to obtain a character sequence.

[0074] In a second aspect, the present application provides a medical record text recognition system based on deep learning, including:

[0075] The preprocessing module is used to preprocess the original medical record images, using the Canny edge detection algorithm and the Hough line detection algorithm to perform edge detection and angle correction on the images respectively;

[0076] The text annotation module is used to annotate the text area of ​​the preprocessed image to obtain a text detection training dataset;

[0077] The detection module is used to detect images using the DBNet network and obtain the coordinates of the bounding box position of the text area in the image;

[0078] The text recognition model training module is used to build an improved SVTR model, extract features from the text detected area, and train the text recognition model;

[0079] The text recognition module is used to identify images using a text recognition model to obtain the character content in the medical record image.

[0080] In an embodiment of the present application, after determining the target image for text recognition, the medical record image is preprocessed, text detection is performed on the preprocessed image, the detected image is segmented according to the detection frame, the single line of text is recognized, the text recognition result is obtained, and the image text is predicted to obtain recognized text with extremely high accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0081] Figure 1 A flowchart of a method for medical record text recognition based on deep learning provided in an embodiment of the present application;

[0082] Figure 2 A schematic diagram of a sample of pre-processed original medical record images provided in an embodiment of the present application;

[0083] Figure 3 for Figure 2 Schematic diagram of the annotation results of image samples;

[0084] Figure 4 for Figure 2 Schematic diagram of recognition results of image samples;

[0085] Figure 5 A schematic diagram of a medical record text recognition system based on deep learning provided in an embodiment of the present application. DETAILED DESCRIPTION

[0086] The present invention will be described below with reference to the accompanying drawings and specific implementation methods.

[0087] See also Figure 1 In an embodiment of the present application, a method for medical record text recognition based on deep learning includes:

[0088] S101, pre-processing the original medical record image, using the Canny edge detection algorithm and the Hough line detection algorithm to perform edge detection and angle correction on the image respectively.

[0089] In this embodiment, the Canny edge detection algorithm is used to perform edge detection on the image, and then the Hough line detection algorithm is used to perform angle correction on the image after edge detection.

[0090] Specifically, the Canny edge detection algorithm is used to perform edge detection on the image, including:

[0091] Compute the strength of the gradient using a filter with a Gaussian differentiable function;

[0092] Use a 5×5 Gaussian filter to remove noise from the image;

[0093] The Sobel filter operation is used to obtain the first-order derivative, and the pixel edge gradient value and direction are calculated by the first-order derivative. The calculation formula is as follows:

[0094]

[0095]

[0096] in, and Represent the gradient of the image in the x and y directions respectively. For each pixel, the gradients in these two directions are calculated, then they are squared and added together, and finally the square root is taken. The result is the gradient intensity of the pixel, that is, the edge intensity Edge gradient (G); θ represents the angle between a vector and the x-axis, ranging from -180° to 180°, and Angle (θ) represents the calculated arc tangent value of the angle θ, where G x and G y represent the horizontal and vertical components of a vector respectively.

[0097] Then, the entire image is searched and filtered at the pixel level to exclude objects that are not related to the edge structure;

[0098] Define two opposite thresholds as high threshold and low threshold;

[0099] When the gradient intensity is greater than the high threshold, the pixel object is determined to be positive, that is, the edge area;

[0100] When the gradient intensity is less than the low threshold, the pixel object is judged to be negative, that is, it is a non-edge area;

[0101] When the gradient strength is between the high threshold and the low threshold, determine whether the pixel is connected to the valid edge. If so, the pixel is also marked as a valid edge.

[0102] Repeat the above steps until all image areas are traversed.

[0103] After edge detection of the original medical record image, the process of using the Hough line detection algorithm to correct the image angle includes:

[0104] Use polar coordinates to describe the straight line of the rectangular coordinate axis. For each x, y point on the straight line, the following equation is satisfied:

[0105] ρ=xcos(θ)+ysin(θ)

[0106] Where ρ is the shortest distance from the origin to the line, and θ is the distance between the x-axis and the distance line;

[0107] Quantize the parameter space into m×n units and set the accumulator matrix Q m×n ;

[0108] Assign an accumulator Q to each unit in the parameter space i,j , and set the initial value of the accumulator to zero;

[0109] In the parameter space, find the unit corresponding to θ and ρ, and add 1 to the accumulator of the unit, that is, Q i,j =Q i,j +1;

[0110] After all points in the rectangular coordinate system have gone through the above two steps, the value of each accumulator in the parameter space is checked. The θ corresponding to the unit with the largest accumulator value is the tilt angle of the image.

[0111] Use the imrotate function to rotate the image to obtain the angle-corrected image.

[0112] S102: Label the text area of ​​the preprocessed image to obtain a text detection training data set.

[0113] In this example, the medical record images are first screened, and images with blurred images, severe exposure, or underexposure are screened out. The open source method PPOCRLabel is used to label the medical record photos, and the text in the photo is marked with a rectangular box and the coordinates of the four corner points of the rectangular box and the meaning of the Chinese characters are output. The output result is converted into an ICDAR format file through a script for subsequent training. The format mainly consists of the coordinates of the four points and the text content. See Figure 2 is the original medical record image sample after preprocessing, Figure 3 Schematic diagram of the annotation results.

[0114] S103: Use the DBNet network to detect the image and obtain the position coordinates of the bounding box of the text area in the image.

[0115] In this example, we first adjust the dataset parameters for DBNet training. The data_dir parameter is set to the dataset path, the label_file_list parameter points to the dataset annotation file, and the ratio of the training set to the test set is 9:1.

[0116] The image is input into the ResNet50 backbone network, and a 3×3 convolution operation is performed on the image to obtain feature maps with a size ratio of 1 / 2, 1 / 4, 1 / 8, 1 / 16, and 1 / 32 of the original image respectively;

[0117] Upsample by ×2 from top to bottom and then fuse with the feature map of the same size generated from bottom to top;

[0118] After fusion, 3×3 convolution is used to eliminate the aliasing effect of upsampling;

[0119] After FPN, four feature maps with sizes of 1 / 4, 1 / 8, 1 / 16, and 1 / 32 of the original image are obtained;

[0120] Upsample the four feature maps separately and unify them into a feature map of 1 / 4 size;

[0121] Finally, concat the four 1 / 4 size feature maps to obtain feature F;

[0122] F passes through the convolution layer, compresses the channel to 1 / 4 of the input, and then undergoes batch normalization and ReLU function activation to obtain the feature map shape;

[0123] The obtained feature map is deconvolved with a convolution kernel of 2×2. The resulting feature map shape is (batch, 256, 1 / 2W, 1 / 2H), which is half the size of the original image.

[0124] Then perform the deconvolution operation, the output feature map channel is 1, and the obtained feature map shape is (batch, W, H), which is the size of the original image.

[0125] Finally, after the sigmoid function, the probability map P and threshold map T are output;

[0126] The probability map P and the threshold map T are subjected to the differentiable binarization DB operation to obtain an approximate binary map;

[0127] Use contraction to construct labels;

[0128] Expand the original polygonal text area G to G d , the shrinkage offset D is calculated according to the following formula:

[0129]

[0130] Where L is the perimeter of the polygon, A is the area of ​​the polygon, and r is the shrinkage factor, which is empirically set to 0.4;

[0131] Shrink frame G s and expansion frame G d The gap between them is regarded as the boundary of the text area, and the normalized distance between each pixel in this gap and the original image boundary G is calculated;

[0132] Then normalize the calculated value and divide it by the offset D. At this time, G sand G d The value on becomes 1, and then these values ​​are subtracted from 1. At this time, G s and G d The value range of the region is [0,1];

[0133] Finally, scale it again to get the final label.

[0134] In this embodiment, the loss function of the DBNet network is the sum of the loss of the probability map, the loss of the binarization map, and the loss of the threshold map:

[0135] L=L d +α×L b +β×L t

[0136] Among them, L is the total loss, L b To approximate the loss of binary images, Dice loss is used, L s is the probability map loss. To balance the ratio of positive and negative samples, Dice loss with OHEM is used for difficult sample mining. Positive sample: negative sample = 1:3, L t is the threshold graph loss, using the L1 distance between the predicted value and the label, and α and β are weight coefficients, set to 1 and 10 respectively;

[0137]

[0138] Among them, S l Indicates that OHEM is used for sampling, with a positive-negative sample ratio of 1:3;

[0139] L t Calculate the L1 distance between the predicted value and the label:

[0140]

[0141] Among them, R d is the extended polygon G d A set of indices of the inner pixels; y * is the label of the threshold map.

[0142] S104: Build an improved SVTR model, extract features from the text-detected area, and train a text recognition model.

[0143] Input the dataset after text detection and segment the image according to the detection label;

[0144] Generate training and test sets for the new images in a 9:1 ratio, set data_dir to the new dataset path, and point label_file_list to the new dataset annotation file;

[0145] The image is input into the SVTR backbone network with an input image size of H×W×3. Two consecutive 3×3 convolutions and batch normalization are used to achieve improved patch embedding. Character components CC0 of different sizes, which are used to represent character strokes;

[0146] In the first stage, CC0 is reshaped to hw×d0 size, and then layer normalization LN is applied to CC0 before being fed into global mixing or local mixing. The settings of global mixing and local mixing modules vary depending on the application scenario.

[0147] Next, layer normalization and a multi-layer perceptron are applied for feature fusion to reshape the hybrid block into an embedding of h×w×d0. Then, a 3×3 convolution with a stride of 2 is used on the height dimension and a convolution with a stride of 1 is used on the width dimension, followed by layer normalization to produce a h×w×d0 embedding.

[0148] The size of the character component CC1 after the first stage is The second stage repeats the operation of the first stage, and the size of the character component CC2 is In the last stage, the merging operation is replaced by the pooling operation, which first pools the height dimension to 1, and then passes through a fully connected layer, nonlinear activation and dropout to compress the character components into a size of Feature sequence C;

[0149] Finally, parallel linear prediction is performed on C and duplicate removal is performed to obtain a character sequence.

[0150] S105: Use a text recognition model to recognize the image and obtain the character content in the medical record image.

[0151] See also Figure 4 , is the final recognition result, compared Figure 1 and Figure 4 , Figure 4 You can accurately Figure 1 Accurately identify the text recorded in.

[0152] Corresponding to the medical record text recognition method based on deep learning provided in the above embodiment, the present application also provides an embodiment of a medical record text recognition system based on deep learning.

[0153] See also Figure 5 , a medical record text recognition system based on deep learning 20, including:

[0154] The preprocessing module 201 is used to preprocess the original medical record image, using the Canny edge detection algorithm and the Hough line detection algorithm to perform edge detection and angle correction on the image respectively;

[0155] A text annotation module 202 is used to annotate the text area of ​​the preprocessed image to obtain a text detection training data set;

[0156] Detection module 203, used to detect the image using the DBNet network and obtain the coordinates of the bounding box of the text area in the image;

[0157] The text recognition model training module 204 is used to build an improved SVTR model, extract features from the text detected area, and train the text recognition model;

[0158] The text recognition module 205 is used to identify the image using the text recognition model to obtain the character content in the medical record image.

[0159] In the embodiments of the present application, "at least one" refers to one or more, and "more" refers to two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent the existence of A alone, the existence of A and B at the same time, and the existence of B alone. Among them, A and B can be singular or plural. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship. "At least one of the following" and similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one of a, b and c can be represented by: a, b, c, ab, ac, bc, or abc, where a, b, c can be single or multiple.

[0160] The above description is merely a specific embodiment of the present application. Any person skilled in the art may easily conceive of variations or substitutions within the technical scope disclosed in this application, and such variations or substitutions shall be within the scope of protection of this application. The scope of protection of this application shall be subject to the scope of protection of the claims.

Claims

1. A medical record text recognition method based on deep learning, characterized in that: include: The original medical record images were preprocessed, and the Canny edge detection algorithm and Hough line detection algorithm were used to perform edge detection and angle correction on the images respectively; Label the text area of ​​the preprocessed image to obtain a text detection training dataset; Use the DBNet network to detect the image and obtain the bounding box position coordinates of the text area in the image; Build an improved SVTR model, extract features from the text detected area, and train the text recognition model, including: Input the dataset after text detection and segment the image according to the detection label; Generate training and test sets for the new images in a 9:1 ratio, set data_dir to the new dataset path, and point label_file_list to the new dataset annotation file; Input the image into the SVTR backbone network, the input image size is , using two consecutive 3×3 convolutions and batch normalization to achieve improved patch embedding Character components of size ,These character components are used to represent character strokes; Entering the first stage, Reshape to size, then Layer normalization (LN) is applied and fed into global mixing or local mixing. The settings of global mixing and local mixing modules vary depending on the application scenario. Next, layer normalization and a multi-layer perceptron are applied sequentially for feature fusion to reshape the hybrid block into Then, a 3×3 convolution with a stride of 2 is applied to the height dimension and a stride of 1 is applied to the width dimension, followed by layer normalization, resulting in a network of size ; Character components after the first stage Size In the second stage, the operation of the first stage is repeated, and the character components are Size In the last stage, the merging operation is replaced by the merging operation, which first pools the height dimension to 1, and then passes through a fully connected layer, nonlinear activation and dropout to compress the character component into a size of Feature Sequence ; Finally, yes Perform parallel linear prediction and deduplication operations to obtain a character sequence; The text recognition model is used to identify the image and obtain the character content in the medical record image.

2. The method for medical record text recognition based on deep learning according to claim 1, characterized in that: Use the Canny edge detection algorithm to detect edges on the image, including: Compute the strength of the gradient using a filter with a Gaussian differentiable function; Use a 5×5 Gaussian filter to remove noise from the image; The Sobel filter operation is used to obtain the first-order derivative, and the pixel edge gradient value and direction are calculated by the first-order derivative. The calculation formula is as follows: in, and Represents the gradient of the image in the x and y directions respectively. For each pixel, the gradients in these two directions are calculated, then they are squared and added together, and finally the square root is taken. The result is the gradient strength of the pixel, that is, the edge strength ; Represents the angle between a vector and the x-axis, ranging from -180° to 180°. Indicates the calculated angle The arc tangent of and represent the horizontal and vertical components of a vector respectively; Then, the entire image is searched and filtered at the pixel level to exclude objects that are not related to the edge structure; Define two opposite thresholds as high threshold and low threshold; When the gradient intensity is greater than the high threshold, the pixel object is determined to be positive, that is, the edge area; When the gradient intensity is less than the low threshold, the pixel object is judged to be negative, that is, it is a non-edge area; When the gradient strength is between the high threshold and the low threshold, determine whether the pixel is connected to the valid edge. If so, the pixel is also marked as a valid edge. Repeat the above steps until all image areas are traversed.

3. The method for medical record text recognition based on deep learning according to claim 1 or 2, characterized in that: After edge detection of the original medical record image, the process of using the Hough line detection algorithm to correct the image angle includes: Use polar coordinates to describe the straight line of the rectangular coordinate axis. For each x, y point on the straight line, the following equation is satisfied: in, is the shortest distance from the origin to the line, is the distance between the x-axis and the distance line; Quantize the parameter space into m×n units and set the accumulator matrix ; Assign an accumulator to each unit in the parameter space , and set the initial value of the accumulator to zero; In the parameter space, find and The corresponding unit, and add 1 to the accumulator of the unit, that is, ; After all points in the rectangular coordinate system are traversed, the value of each accumulator in the parameter space is checked. The unit with the largest accumulator value corresponds to is the tilt angle of the image; Use the imrotate function to rotate the image to obtain the angle-corrected image.

4. The method for medical record text recognition based on deep learning according to claim 1, characterized in that: The text area of ​​the preprocessed image is labeled to obtain a text detection training data set, including: First, the medical record images are screened, and those with blurred images, severe exposure, or underexposure are screened out; Use the open source method PPOCRLabel to label medical record photos, use a rectangular box to mark the text on the photo and output the coordinates of the four corner points of the rectangular box and the meaning of the Chinese characters; The output results are converted into ICDAR format files through scripts for subsequent training. The format mainly includes the coordinates of four points and text content.

5. The method for medical record text recognition based on deep learning according to claim 1, characterized in that: Use the DBNet network to detect the image and obtain the bounding box position coordinates of the text area in the image, including: Adjust the dataset parameters for DBNet training. Set data_dir to the dataset path, label_file_list to the dataset annotation file, and the ratio of training set to test set is 9:

1. The image is input into the ResNet50 backbone network and a 3×3 convolution operation is performed on the image to obtain feature maps with a size ratio of 1 / 2, 1 / 4, 1 / 8, 1 / 16, and 1 / 32 of the original image respectively; Upsample by ×2 from top to bottom and then fuse with the feature map of the same size generated from bottom to top; After fusion, 3×3 convolution is used to eliminate the aliasing effect of upsampling; After FPN, four feature maps with sizes of 1 / 4, 1 / 8, 1 / 16, and 1 / 32 of the original image are obtained; Upsample the four feature maps separately and unify them into a feature map of 1 / 4 size; Finally, concat the four 1 / 4 size feature maps to obtain feature F; F passes through the convolution layer, compresses the channel to 1 / 4 of the input, and then undergoes batch normalization and ReLU function activation to obtain the feature map shape; The obtained feature map is deconvolved with a convolution kernel of 2×2. The resulting feature map shape is (batch, 256, 1 / 2W, 1 / 2H), which is 1 / 4 the size of the original image. Then perform the deconvolution operation, the output feature map channel is 1, and the obtained feature map shape is (batch, W, H), which is the size of the original image; Finally, after the sigmoid function, the probability map P and threshold map T are output; The probability map P and the threshold map T are subjected to the differentiable binarization DB operation to obtain an approximate binary map; Use contraction to construct labels; The original polygonal text area Expand to , shrinkage offset Calculate according to the following formula: in, is the perimeter of the polygon, A is the area of ​​the polygon, r is the shrinkage factor, which is empirically set to 0.4; Shrink the frame and expansion box The gap between them is considered as the boundary of the text area, and the distance from each pixel in this gap to the boundary of the original image is calculated. The normalized distance of Then normalize the calculated value and divide it by the offset ,at this time and The value on becomes 1, and then these values ​​are subtracted from 1. and The value range of the region is [0,1]; Finally, scale it again to get the final label.

6. The method for medical record text recognition based on deep learning according to claim 5, characterized in that: The loss function of the DBNet network is the sum of the loss of the probability map, the loss of the binarization map, and the loss of the threshold map: in, For the total loss, To approximate the loss of binary images, use Dice loss. is the probability map loss. To balance the ratio of positive and negative samples, Dice loss with OHEM is used for difficult sample mining, with positive sample: negative sample = 1:

3. is the threshold graph loss, using the difference between the predicted value and the label distance, and are weight coefficients, set to 1 and 10 respectively; in, Indicates that OHEM is used for sampling, with a positive-negative sample ratio of 1:3; Calculate the difference between the predicted value and the label distance: in, Is the extended polygon An array of indices of the interior pixels.

7. A medical record text recognition system based on deep learning, characterized by: include: The preprocessing module is used to preprocess the original medical record images, using the Canny edge detection algorithm and the Hough line detection algorithm to perform edge detection and angle correction on the images respectively; The text annotation module is used to annotate the text area of ​​the preprocessed image to obtain a text detection training dataset; The detection module is used to detect images using the DBNet network and obtain the coordinates of the bounding box position of the text area in the image; The text recognition model training module is used to build an improved SVTR model, extract features from the text detected area, and train the text recognition model, including: Input the dataset after text detection and segment the image according to the detection label; Generate training and test sets for the new images in a 9:1 ratio, set data_dir to the new dataset path, and point label_file_list to the new dataset annotation file; Input the image into the SVTR backbone network, the input image size is , using two consecutive 3×3 convolutions and batch normalization to achieve improved patch embedding Character components of size ,These character components are used to represent character strokes; Entering the first stage, Reshape to size, then Layer normalization (LN) is applied and fed into global mixing or local mixing. The settings of global mixing and local mixing modules vary depending on the application scenario. Next, layer normalization and a multi-layer perceptron are applied sequentially for feature fusion to reshape the hybrid block into Then, a 3×3 convolution with a stride of 2 is applied to the height dimension and a stride of 1 is applied to the width dimension, followed by layer normalization to produce a network of size ; Character components after the first stage Size In the second stage, the operation of the first stage is repeated, and the character components are Size In the last stage, the merging operation is replaced by the merging operation, which first pools the height dimension to 1, and then passes through a fully connected layer, nonlinear activation and dropout to compress the character component into a size of Feature Sequence ; Finally, yes Perform parallel linear prediction and deduplication operations to obtain a character sequence; The text recognition module is used to identify images using a text recognition model to obtain the character content in the medical record image.

Images