Text image direction detection method based on deep learning

Abstract

The invention provides a text image direction detection method based on deep learning, and belongs to the field of computer vision, and the method comprises the steps: cutting an original text image according to a preset size, and obtaining a preprocessed text image; inputting the preprocessed text image into an image direction detection model to obtain a direction confidence data set; detecting whether the maximum confidence data in the direction confidence data set meets the confidence threshold requirement, and when the confidence threshold requirement is met, taking the direction corresponding to the maximum confidence data as the text detection direction; and when the confidence coefficient threshold requirement is not met, height-width ratio data of the original text image is adjusted, and re-detection is performed through the image direction detection model. The method is based on the image direction detection model, the detection precision of the character direction in the image is improved, and the attention of the model on the character content characteristics in the image is improved by adjusting the aspect ratio of the original text image, so that the method can be suitable for complex image scenes to meet the requirements of downstream task processing.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a text image direction detection method based on deep learning. Background Art

[0002] In the field of computer vision, image text direction detection is the basis of related tasks such as text image correction and OCR (optical character recognition). If the image text direction is detected incorrectly, subsequent text correction and character recognition tasks will be difficult to perform normally.

[0003] At present, the existing text image direction detection schemes basically detect the direction of text in the image through text lines, resulting in low text direction detection accuracy; especially in more complex image scenes such as handwritten text images and electronic screen text images, the image text direction detection scheme based on text lines is less effective and has low detection accuracy, which makes it difficult to meet the requirements of subsequent text correction and character recognition downstream task processing.

[0004] Therefore, a text image direction detection method based on deep learning is proposed. Summary of the invention

[0005] In order to solve the above technical problems, the present invention provides a text image direction detection method based on deep learning, which is used to solve the problem of low accuracy of text direction detection in traditional text images.

[0006] An embodiment of the present invention provides a text image direction detection method based on deep learning, comprising:

[0007] Get the original text image;

[0008] Cropping the original text image into a preset size to obtain a preprocessed text image;

[0009] Inputting the preprocessed text image into an image direction detection model, calculating the confidence of the text in each direction in the preprocessed text image, and obtaining a direction confidence data set;

[0010] Whether the maximum confidence data in the detection direction confidence data group meets the confidence threshold requirement, and when the maximum confidence data meets the confidence threshold requirement, the direction corresponding to the maximum confidence data is used as the text detection direction;

[0011] When the maximum confidence data does not meet the confidence threshold requirement, the aspect ratio data of the original text image is detected, and the corresponding image merging operation is performed to obtain a pre-merged image, and the image is processed through the image direction detection model to calculate the confidence of the text in each direction in the pre-merged image to obtain the text detection direction.

[0012] Preferably, the present invention provides a text image direction detection method based on deep learning, the step of: cropping the original text image with a preset size to obtain a preprocessed text image; comprising:

[0013] The central area of ​​the original text image is cropped with a preset size of 768*768px to obtain a preprocessed text image, where px is a pixel unit.

[0014] Preferably, the present invention provides a text image direction detection method based on deep learning, the steps of: inputting the preprocessed text image into an image direction detection model, calculating the confidence of the text in each direction in the preprocessed text image, and obtaining a direction confidence data set; comprising:

[0015] Build a pre-trained direction detection model;

[0016] According to the sample text image, a rotation angle transformation is performed to obtain a sample transformation image corresponding to the sample text image, and a sample training set is constructed;

[0017] The pre-trained direction detection model is trained using a sample training set to obtain an image direction detection model;

[0018] The image direction detection model calculates the confidence of the characters in the preprocessed text image in each direction based on the preprocessed text image, and obtains a direction confidence data set.

[0019] Preferably, the present invention provides a text image direction detection method based on deep learning, wherein the step of: constructing a pre-trained direction detection model comprises:

[0020] Build the backbone network layer; optional, including:

[0021] The feature extraction layer is constructed through two-dimensional convolution layer, normalization layer and activation function layer;

[0022] Construct the feature fusion layer through the first convolution layer, the first segmentation layer, and the attention mechanism layer;

[0023] Construct feature downsampling layer;

[0024] The fast spatial pooling layer is constructed through the second convolutional layer, the maximum pooling layer and the third convolutional layer;

[0025] The attention layer is constructed through the second segmentation layer, the multi-head attention mechanism layer and the feedforward neural network layer;

[0026] The feature extraction layer, feature fusion layer, feature downsampling layer, fast spatial pooling layer, and attention layer are connected in sequence to construct a backbone network layer;

[0027] Construct the neck network layer; optional, including:

[0028] Construct feature upsampling layer;

[0029] The feature output layer is constructed through the fourth convolution layer, the third segmentation layer and the grouped convolution layer;

[0030] The feature upsampling layer and the feature output layer are connected to construct a neck network layer;

[0031] Build the detection head layer; optional, including:

[0032] Constructing a first detection layer through a first regressor and a first classifier;

[0033] Constructing a second detection layer through a second regressor and a second classifier;

[0034] The first detection layer and the second detection layer are connected in parallel to construct a detection head layer.

[0035] Preferably, the present invention provides a method for detecting the direction of a text image based on deep learning, the steps of: performing a rotation angle transformation according to a sample text image, obtaining a sample transformation image corresponding to the sample text image, and constructing a sample training set; comprising:

[0036] The sample text images include: Chinese paper text images, English paper text images, Chinese electronic screen text images, and English electronic screen text images;

[0037] The sample text image is rotated by 90 degrees, 180 degrees and 270 degrees to obtain a sample transformation image corresponding to the sample text image, and a sample training set is constructed.

[0038] Preferably, the present invention provides a method for text image direction detection based on deep learning, the steps of: training the pre-trained direction detection model through a sample training set to obtain an image direction detection model; comprising:

[0039] Obtain a pre-training sample image in the sample training set, and detect whether the aspect ratio data of the pre-training sample image is within the preset ratio range When the pre-training sample image is within 768*768px, the central area of ​​the pre-training sample image is cropped to obtain a sample image;

[0040] When the aspect ratio data of the pre-training sample image is lower than the lower limit of the preset ratio interval, performing vertical image merging processing on the pre-training sample image to obtain a sample image;

[0041] When the aspect ratio data of the pre-training sample image is higher than the upper limit of the preset ratio interval, performing horizontal image merging processing on the pre-training sample image to obtain a sample image;

[0042] Performing two-dimensional convolution processing on the sample images in the sample training set based on the width direction and the height direction through the two-dimensional convolution layer in the first feature extraction layer to obtain sample two-dimensional convolution data, and performing normalization processing through the normalization layer, and performing nonlinear mapping through the activation function layer to obtain feature extraction data;

[0043] The first convolution layer in the feature fusion layer performs channel compression on the feature extraction data to obtain convolution feature data, the first segmentation layer segments the convolution feature data into first convolution feature data and second convolution feature data, the attention mechanism layer performs feature enhancement processing on the first convolution feature data to obtain first convolution enhancement data, and the first convolution enhancement data is feature-fused with the second convolution feature data to obtain convolution feature fusion data;

[0044] The feature downsampling layer adjusts the channel dimension of the convolution feature fusion data through point-by-point convolution, and performs downsampling processing through depth-wise separable convolution to obtain feature sampling data;

[0045] The fast spatial pooling layer converts the feature sampling data into one-dimensional feature sampling data through the second convolution layer, extracts corresponding feature parameters through several maximum pooling layers, fuses the feature parameters, and performs convolution processing through the third convolution layer to obtain pooled feature data;

[0046] The attention layer decomposes the pooled feature data into first feature fusion data and second feature fusion data through the second segmentation layer, the multi-head attention mechanism layer performs attention mechanism adjustment on the first feature fusion data, extracts feature fusion parameters, and processes the data through the feedforward neural network layer to obtain attention feature data, and performs feature fusion on the attention feature data and the second feature fusion data to obtain attention fusion data;

[0047] The feature upsampling layer of the neck network layer performs a feature upsampling operation on the attention fusion data to obtain feature expansion data;

[0048] The feature output layer transforms the feature expansion data into a one-dimensional space through a fourth convolution layer to obtain one-dimensional feature expansion data, decomposes the one-dimensional feature expansion data into first feature expansion data and second feature expansion data through a third segmentation layer, distributes the one-dimensional feature expansion data to a plurality of convolution channels for convolution operation through a grouped convolution layer, and merges the obtained plurality of convolution feature sub-data into grouped convolution feature data through a one-dimensional convolution kernel, and fuses the obtained plurality of convolution feature sub-data with the second feature expansion data to obtain feature output data;

[0049] The first detection layer in the detection head layer outputs data according to the feature to obtain first prediction data; the second detection layer outputs data according to the feature to obtain second prediction data; the first prediction data only includes a unique prediction result, and the second prediction data includes multiple prediction results;

[0050] Calculating first deviation data between the first predicted data and the standard test data corresponding to the sample image through a loss function, calculating second deviation data between the second predicted data and the standard test data corresponding to the sample image through a loss function, and optimizing the training parameters of the backbone network layer and the neck network layer through the first deviation data and the second deviation data; optionally, including:

[0051] Obtain the prediction direction θ1 and the prediction text center point (x1, y1) in the first prediction data;

[0052] Obtain the standard direction θ2 and the standard text center point (x2, y2) in the standard test data;

[0053]

[0054] Wherein, L is the loss function, |θ1-θ2| is the direction deviation, and d is the diagonal distance of the sample image.

[0055] Preferably, the present invention provides a method for detecting the direction of a text image based on deep learning, the steps of: obtaining a pre-training sample image in the sample training set, detecting whether the aspect ratio data of the pre-training sample image is within the preset ratio range When the pre-training sample image is within the range of 768*768px, the central area of ​​the pre-training sample image is cropped to obtain a sample image; including:

[0056] When the aspect ratio data of the pre-training sample image is equal to the lower limit of the preset ratio range, the pixel points of the pre-training sample image are divided into a plurality of pixel rows based on the vertical direction, the pixel rows are up-sampled to obtain an up-sampled text image, and the central area of ​​the up-sampled text image is cropped with a preset size to obtain a sample image;

[0057] When the aspect ratio data of the pre-trained sample image is equal to the upper limit of the preset ratio range, the pixel rows of the pre-trained sample image are under-sampled based on the vertical direction to obtain an under-sampled text image, and the central area of ​​the under-sampled text image is cropped with a preset size to obtain a sample image.

[0058] Preferably, the present invention provides a method for detecting the direction of a text image based on deep learning, the step of: when the aspect ratio data of the pre-training sample image is lower than the lower limit of the preset ratio interval, performing vertical image merging processing on the pre-training sample image to obtain a sample image; comprising:

[0059] Copy the pre-training sample image to obtain a copy sample image, and merge it with the pre-training sample image in a vertical direction to obtain a vertical merged image;

[0060] Detecting height data and length data of the vertically merged image to obtain aspect ratio data of the vertically merged image;

[0061] When the aspect ratio data of the vertically merged image is within the preset ratio range, the central area of ​​the vertically merged image is cropped with a preset size to obtain a sample image; when the aspect ratio data of the vertically merged image is lower than the lower limit of the preset ratio range, the copy sample image is obtained and merged with the vertically merged image to obtain a second vertically merged image, until the aspect ratio data of the vertically merged image is within the preset ratio range.

[0062] Preferably, the present invention provides a text image direction detection method based on deep learning, the steps of: the image direction detection model calculates the confidence of the characters in the preprocessed text image in each direction according to the preprocessed text image, and obtains a direction confidence data group; comprising:

[0063] The image direction detection model extracts the image attention feature data of the preprocessed text image through the backbone network layer, performs feature fusion on the image attention feature data through the neck network layer, obtains image feature output data, and selects the second detection layer of the detection head layer to obtain text direction detection data according to the image feature output data;

[0064] Performing confidence calculation based on the text direction detection data to obtain direction confidence data;

[0065] Set the direction fitting interval;

[0066] Obtaining direction confidence data within the direction fitting interval, and selecting the text direction corresponding to the maximum value data therein as the predicted direction within the direction fitting interval;

[0067] The direction confidence data of all directions in the preprocessed text image are traversed and calculated through the direction fitting interval to obtain the predicted direction and the corresponding direction confidence data, and construct a direction confidence data group.

[0068] Compared with traditional technologies, the beneficial effects of the present invention are: a text image direction detection method based on deep learning obtains the text detection direction based on the confidence of the text in each direction in the pre-processed text image detected by an image direction detection model, which solves the problem of low accuracy of text direction detection based on text lines in traditional technologies, and by adjusting the aspect ratio of the original text image, the model's attention to the text content features in the image is improved, so that it can be applied to more complex image scenes such as handwritten text images and electronic screen text images, and the detection effect of text direction is improved to meet the requirements of subsequent text correction and character recognition downstream task processing.

[0069] Other features and advantages of the present invention will be described in the following description, and partly become apparent from the description, or understood by practicing the present invention. The purpose and other advantages of the present invention can be realized and obtained by the structures particularly pointed out in the written description, claims, and drawings.

[0070] The technical solution of the present invention is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0071] Figure 1 A schematic diagram of a text image direction detection method based on deep learning provided by the present invention;

[0072] Figure 2 A flowchart of a text image direction detection method based on deep learning provided by the present invention;

[0073] Figure 3 This is a second vertical merged image of a text image direction detection method based on deep learning provided by the present invention. DETAILED DESCRIPTION

[0074] The preferred embodiments of the present invention are described below in conjunction with the accompanying drawings. It should be understood that the preferred embodiments described herein are only used to illustrate and explain the present invention, and are not used to limit the present invention.

[0075] Embodiment 1:

[0076] The embodiment of the present invention provides a text image direction detection method based on deep learning, referring to Figure 1 and Figure 2 ,include:

[0077] Get the original text image;

[0078] Cropping the original text image with a preset size to obtain a preprocessed text image;

[0079] Input the preprocessed text image into the image direction detection model, calculate the confidence of the text in each direction in the preprocessed text image, and obtain a direction confidence data group;

[0080] Whether the maximum confidence data in the detection direction confidence data group meets the confidence threshold requirement, when the maximum confidence data meets the confidence threshold requirement, the direction corresponding to the maximum confidence data is used as the text detection direction;

[0081] When the maximum confidence data does not meet the confidence threshold requirement, the aspect ratio data of the original text image is detected, and the corresponding image merging operation is performed to obtain a pre-merged image, which is processed through an image direction detection model, and the confidence of the text in each direction in the pre-merged image is calculated to obtain the text detection direction.

[0082] In the above embodiments, the acquired original text image is cropped to a preset size to obtain a preprocessed text image, the preprocessed text image is input into the image direction detection model, the confidence of the text in each direction in the preprocessed text image is calculated, and a direction confidence data group is obtained; whether the maximum confidence data in the direction confidence data group meets the confidence threshold requirement is detected, and when the maximum confidence data meets the confidence threshold requirement, the direction corresponding to the maximum confidence data is used as the text detection direction.

[0083] In the above embodiments, when the maximum confidence data in the direction confidence data group does not meet the confidence threshold requirement, the aspect ratio data of the original text image is detected, and when the aspect ratio data of the original text image is lower than the lower limit of the preset ratio interval, the original text image is subjected to vertical image merging processing; when the aspect ratio data of the original text image is higher than the upper limit of the preset ratio interval, the original text image is subjected to horizontal image merging processing; the image after image merging processing is cropped to a preset size, a pre-merged image is obtained for processing through an image direction detection model, the confidence of the text in each direction in the pre-merged image is calculated, and the text detection direction is obtained.

[0084] In one embodiment, the preset size is set to 768*768px.

[0085] In one embodiment, the preset ratio interval is set to

[0086] In one embodiment, the confidence threshold requirement is set to 0.65.

[0087] The beneficial effects of the above technology are: cropping the acquired original text image with a preset size to obtain a preprocessed text image, detecting the confidence of the text in each direction in the preprocessed text image through an image direction detection model, selecting the maximum confidence data, and when the confidence threshold requirement is met, obtaining the direction corresponding to the maximum confidence data as the text detection direction, thereby realizing the detection of the text direction in the original text image, and when the confidence threshold requirement is not met, adjusting the aspect ratio of the original text image, and re-detecting through the image direction detection model; compared with traditional technologies, the above method obtains the text detection direction by detecting the confidence of the text in each direction in the preprocessed text image based on the image direction detection model, solving the problem of low accuracy of text direction detection based on text lines in traditional technologies, and by adjusting the aspect ratio of the original text image, improving the model's attention to the text content features in the image, making it applicable to more complex image scenes such as handwritten text images and electronic screen text images, and improving the text direction detection effect to meet the requirements of subsequent text correction and character recognition downstream task processing.

[0088] Embodiment 2:

[0089] The embodiment of the present invention provides a method for detecting the direction of a text image based on deep learning, comprising the steps of: cropping an original text image with a preset size to obtain a preprocessed text image; and comprising:

[0090] The central area of ​​the original text image is cropped with a preset size of 768*768px to obtain a preprocessed text image, where px is a pixel unit.

[0091] In the above embodiment, the preset size is set to 768*768px, and after the central area of ​​the original text image is cropped, the pre-processed text image is input into the image direction detection model for processing.

[0092] Embodiment 3:

[0093] The embodiment of the present invention provides a text image direction detection method based on deep learning, comprising the steps of: inputting a preprocessed text image into an image direction detection model, calculating the confidence of the text in each direction in the preprocessed text image, and obtaining a direction confidence data group; comprising:

[0094] Build a pre-trained direction detection model;

[0095] According to the sample text image, a rotation angle transformation is performed to obtain a sample transformation image corresponding to the sample text image, and a sample training set is constructed;

[0096] The pre-trained direction detection model is trained using a sample training set to obtain an image direction detection model;

[0097] The image direction detection model calculates the confidence of the characters in each direction in the preprocessed text image based on the preprocessed text image, and obtains a direction confidence data set.

[0098] In the above embodiments, the constructed pre-trained direction detection model is trained by using a sample training set to obtain an image direction detection model.

[0099] In the above embodiments, a rotation angle transformation is performed based on a sample text image to obtain a sample transformed image and construct a sample training set.

[0100] In the above embodiments, the image direction detection model calculates the confidence of the characters in the preprocessed text image in each direction based on the preprocessed text image, and obtains a direction confidence data set.

[0101] The beneficial effects of the above technology are: the pre-trained direction detection model is trained through a sample training set, thereby realizing the acquisition of an image direction detection model; the image direction detection model calculates the confidence of the text direction based on the pre-processed text image, and obtains a direction confidence data group, which facilitates the subsequent steps to determine the text detection direction in the image.

[0102] Embodiment 4:

[0103] The embodiment of the present invention provides a method for text image direction detection based on deep learning, the steps are: constructing a pre-trained direction detection model, including:

[0104] Build the backbone network layer; optional, including:

[0105] The feature extraction layer is constructed through two-dimensional convolution layer, normalization layer and activation function layer;

[0106] Construct the feature fusion layer through the first convolution layer, the first segmentation layer, and the attention mechanism layer;

[0107] Construct feature downsampling layer;

[0108] The fast spatial pooling layer is constructed through the second convolutional layer, the maximum pooling layer and the third convolutional layer;

[0109] The attention layer is constructed through the second segmentation layer, the multi-head attention mechanism layer and the feedforward neural network layer;

[0110] The feature extraction layer, feature fusion layer, feature downsampling layer, fast spatial pooling layer, and attention layer are connected in sequence to construct the backbone network layer;

[0111] Construct the neck network layer; optional, including:

[0112] Construct feature upsampling layer;

[0113] The feature output layer is constructed through the fourth convolution layer, the third segmentation layer and the grouped convolution layer;

[0114] The feature upsampling layer and the feature output layer are connected to construct the neck network layer;

[0115] Build the detection head layer; optional, including:

[0116] Constructing a first detection layer through a first regressor and a first classifier;

[0117] Constructing a second detection layer through a second regressor and a second classifier;

[0118] The first detection layer and the second detection layer are connected in parallel to construct a detection head layer.

[0119] In the above embodiment, the feature extraction layer, feature fusion layer, feature downsampling layer, fast spatial pooling layer, and attention layer are connected in sequence to construct a backbone network layer; the feature upsampling layer and the feature output layer are connected to construct a neck network layer; the first detection layer and the second detection layer are connected in parallel to construct a detection head layer. The backbone network layer, the neck network layer, and the detection head layer are connected in sequence.

[0120] In one embodiment, the pre-trained direction detection model may be implemented as a YOLOv10 model.

[0121] The beneficial effect of the above technology is that through the backbone network layer, the neck network and the detection head layer, the construction of a pre-trained direction detection model is realized, wherein the backbone network layer is used to realize the extraction of feature parameters of the input image, the neck network layer is used to realize the fusion of multi-layer feature parameters, and the detection head layer is used to realize the prediction of the direction of the image text.

[0122] Embodiment 5:

[0123] The embodiment of the present invention provides a method for detecting the direction of a text image based on deep learning, comprising the steps of: performing a rotation angle transformation according to a sample text image, obtaining a sample transformation image corresponding to the sample text image, and constructing a sample training set; comprising:

[0124] Sample text images, including: Chinese paper text images, English paper text images, Chinese electronic screen text images, and English electronic screen text images;

[0125] The sample text image is rotated by 90 degrees, 180 degrees, and 270 degrees to obtain the sample transformation image corresponding to the sample text image, and a sample training set is constructed.

[0126] In the above embodiments, sample text images are constructed by acquiring Chinese paper text images, English paper text images, Chinese electronic screen text images, and English electronic screen text images, and a sample training set is constructed by rotating and transforming the sample text images and acquiring sample transformed images.

[0127] The beneficial effect of the above technology is that: while obtaining the sample text image, a rotation transformation is performed to obtain the sample transformed image, enriching the image data in the sample training set to achieve large-scale sample training of the pre-trained direction detection model.

[0128] Embodiment 6:

[0129] The embodiment of the present invention provides a method for text image direction detection based on deep learning, comprising the steps of: training a pre-trained direction detection model through a sample training set to obtain an image direction detection model; and comprising:

[0130] Obtain the pre-training sample images in the sample training set, and detect whether the aspect ratio data of the pre-training sample images is within the preset ratio range When the image is within the range of 100, the central area of ​​the pre-training sample image is cropped with a preset size of 768*768px to obtain a sample image;

[0131] When the aspect ratio data of the pre-training sample image is lower than the lower limit of the preset ratio range, the pre-training sample image is subjected to vertical image merging processing to obtain a sample image;

[0132] When the aspect ratio data of the pre-training sample image is higher than the upper limit of the preset ratio range, the pre-training sample image is subjected to horizontal image merging processing to obtain a sample image;

[0133] Performing two-dimensional convolution processing on the sample images in the sample training set based on the width direction and the height direction through the two-dimensional convolution layer in the first feature extraction layer to obtain sample two-dimensional convolution data, and performing normalization processing through the normalization layer, and performing nonlinear mapping through the activation function layer to obtain feature extraction data;

[0134] The first convolution layer in the feature fusion layer performs channel compression on the feature extraction data to obtain convolution feature data. The first segmentation layer segments the convolution feature data into first convolution feature data and second convolution feature data. The attention mechanism layer performs feature enhancement processing on the first convolution feature data to obtain first convolution enhancement data. The first convolution enhancement data is feature-fused with the second convolution feature data to obtain convolution feature fusion data.

[0135] The feature downsampling layer adjusts the channel dimension of the convolution feature fusion data through point-by-point convolution, and performs downsampling processing through depth-wise separable convolution to obtain feature sampling data;

[0136] The fast spatial pooling layer converts the feature sampling data into one-dimensional feature sampling data through the second convolution layer, and extracts several corresponding feature parameters through several maximum pooling layers, fuses the feature parameters, and obtains pooled feature data through convolution processing through the third convolution layer;

[0137] The attention layer decomposes the pooled feature data into first feature fusion data and second feature fusion data through the second segmentation layer. The multi-head attention mechanism layer adjusts the attention mechanism on the first feature fusion data, extracts feature fusion parameters, and processes it through the feedforward neural network layer to obtain attention feature data. The attention feature data is feature-fused with the second feature fusion data to obtain attention fusion data.

[0138] The feature upsampling layer of the neck network layer performs feature upsampling operations on the attention fusion data to obtain feature expansion data;

[0139] The feature output layer transforms the feature expansion data into a one-dimensional space through the fourth convolution layer to obtain the one-dimensional feature expansion data, and decomposes the one-dimensional feature expansion data into the first feature expansion data and the second feature expansion data through the third segmentation layer, distributes the one-dimensional feature expansion data to several convolution channels for convolution operation through the grouped convolution layer, and merges the obtained several convolution feature sub-data into grouped convolution feature data through the one-dimensional convolution kernel, and fuses them with the second feature expansion data to obtain feature output data;

[0140] The first detection layer in the detection head layer outputs data according to the features to obtain first prediction data; the second detection layer outputs data according to the features to obtain second prediction data; the first prediction data only contains a unique prediction result, and the second prediction data contains multiple prediction results;

[0141] Calculating first deviation data of the standard test data corresponding to the first prediction data and the sample image by a loss function, calculating second deviation data of the standard test data corresponding to the second prediction data and the sample image by a loss function, and optimizing the training parameters of the backbone network layer and the neck network layer by the first deviation data and the second deviation data; optionally, including:

[0142] Obtain the prediction direction θ1 and the prediction text center point (x1, y1) in the first prediction data;

[0143] Obtain the standard direction θ2 and the standard text center point (x2, y2) in the standard test data;

[0144]

[0145] Among them, L is the loss function, |θ1-θ2| is the direction deviation, and d is the diagonal distance of the sample image.

[0146] In the above embodiment, a pre-training sample image in a sample training set is obtained, and the aspect ratio data of the pre-training sample image is detected to be within a preset ratio range. When the image is within the range of 100, the central area of ​​the pre-training sample image is cropped with a preset size of 768*768px to obtain a sample image.

[0147] In the above embodiments, when the aspect ratio data of the pre-trained sample image is lower than the lower limit of the preset ratio interval, the pre-trained sample image is processed by vertical image merging to obtain a sample image; when the aspect ratio data of the pre-trained sample image is higher than the upper limit of the preset ratio interval, the pre-trained sample image is processed by horizontal image merging to obtain a sample image.

[0148] The beneficial effect of the above technology is that by detecting and adjusting the aspect ratio data of the pre-trained sample image, the aspect ratio of the sample image trained by the pre-trained direction detection model is made to be within the preset ratio range. This avoids using sample images with large differences in aspect ratio to train the model, which would cause the trained model to fall into a fixed paradigm and affect the model’s prediction of the text direction in the preprocessed text image.

[0149] In the above embodiments, the backbone network layer, the neck network layer and the detection head layer in the pre-trained direction detection model are trained through the sample training set.

[0150] In the above embodiments, the two-dimensional convolution layer in the first feature extraction layer performs two-dimensional convolution processing on the sample images in the sample training set based on the width direction and the height direction to obtain sample two-dimensional convolution data, and performs normalization processing through the normalization layer. The activation function layer performs nonlinear mapping to obtain feature extraction data; the two-dimensional convolution layer performs a sliding window operation based on the height direction and the width direction of the sample image to fully extract feature information, obtain sample two-dimensional convolution data, and performs normalization processing through the normalization layer to avoid the sample two-dimensional convolution data. The data volume is too large to cause the model performance to be unstable. The activation function layer performs nonlinear activation through the activation function SiLu to obtain feature extraction data.

[0151] The first convolution layer in the feature fusion layer performs channel compression on the feature extraction data to obtain convolution feature data, the first segmentation layer segments the convolution feature data into first convolution feature data and second convolution feature data, the attention mechanism layer performs feature enhancement processing on the first convolution feature data to obtain first convolution enhanced data, and the first convolution enhanced data is feature fused with the second convolution feature data to obtain convolution feature fusion data; the first convolution layer in the feature fusion layer effectively reduces the amount of parameter calculation, and the first segmentation layer and the attention mechanism layer avoid gradient vanishing while fusing multiple layers of features, so that the acquired convolution feature fusion data effectively enhances the feature representation capability.

[0152] The feature downsampling layer adjusts the channel dimension of the convolution feature fusion data through point-by-point convolution, and performs downsampling processing through depth-wise separable convolution to obtain feature sampling data, while reducing the computational cost and number of parameters of the convolution feature fusion data, while maintaining the model accuracy.

[0153] The fast spatial pooling layer converts the feature sampling data into one-dimensional feature sampling data through the second convolution layer, and extracts several corresponding feature parameters through several maximum pooling layers, fuses the feature parameters, and obtains pooled feature data through convolution processing through the third convolution layer; the global features and local features of the feature sampling data are extracted through the fast spatial pooling layer, and fused to obtain pooled feature data.

[0154] The attention layer decomposes the pooled feature data into first feature fusion data and second feature fusion data through the second segmentation layer. The multi-head attention mechanism layer adjusts the attention mechanism on the first feature fusion data, extracts feature fusion parameters, and processes it through the feedforward neural network layer to obtain attention feature data, and performs feature fusion on the attention feature data and the second feature fusion data to obtain attention fusion data. The attention layer sends the first feature fusion data in the pooled feature data to the multi-head attention mechanism layer and the feedforward neural network layer for processing through the second segmentation layer to obtain the deep features of the first feature fusion data, and performs feature fusion on the attention feature data and the second feature fusion data, thereby realizing the fusion of deep features and shallow features in the pooled feature data and obtaining attention fusion data.

[0155] The feature upsampling layer of the neck network layer performs feature upsampling operations on the attention fusion data to obtain feature expansion data;

[0156] The feature output layer transforms the feature expansion data into one-dimensional space through the fourth convolution layer to obtain one-dimensional feature expansion data, and decomposes the one-dimensional feature expansion data into first feature expansion data and second feature expansion data through the third segmentation layer, distributes the one-dimensional feature expansion data to several convolution channels for convolution operation through the grouped convolution layer, and merges the obtained several convolution feature sub-data into grouped convolution feature data through the one-dimensional convolution kernel, and fuses them with the second feature expansion data to obtain feature output data; the feature output layer performs grouped convolution on the feature expansion data, which effectively improves the feature extraction and expression capabilities of the model, and merges the obtained convolution feature sub-data and fuses them with the second feature expansion data, thereby realizing the acquisition of feature output data containing multiple layers of features.

[0157] The first detection layer in the detection head layer outputs data according to the features to obtain first prediction data; the second detection layer outputs data according to the features to obtain second prediction data; the first prediction data only contains a unique prediction result, and the second prediction data contains multiple prediction results;

[0158] Calculate the first deviation data of the standard test data corresponding to the first prediction data and the sample image through the loss function, calculate the second deviation data of the standard test data corresponding to the second prediction data and the sample image through the loss function, and optimize the training parameters of the backbone network layer and the neck network layer through the first deviation data and the second deviation data;

[0159] The detection head layer adopts a dual detection layer structure. The first prediction data contains a unique prediction result, and the second prediction data contains multiple prediction results. The first deviation data and the second deviation data of the standard test data corresponding to the sample image are calculated through the loss function to obtain rich supervision signals to achieve parameter optimization of the pre-trained direction detection model and obtain the image direction detection model.

[0160] In the above embodiments, the predicted direction and predicted text center point in the first predicted data are obtained, the standard direction and standard text center point in the standard test data are obtained, and the first deviation data is calculated through a loss function, and the loss function is composed of the predicted direction deviation degree and the text center point deviation value.

[0161] The beneficial effects of the above technology are: through the sample images in the sample training set, the first deviation data and the second deviation data are calculated based on the loss function to optimize the parameters of the backbone network layer and the neck network layer in the pre-trained direction detection model, and obtain the image direction detection model.

[0162] Embodiment 7:

[0163] The embodiment of the present invention provides a text image direction detection method based on deep learning, comprising the steps of: obtaining a pre-training sample image in a sample training set, detecting whether the aspect ratio data of the pre-training sample image is within a preset ratio range When the pre-training sample image is within, the central area of ​​the pre-training sample image is cropped with a preset size of 768*768px to obtain a sample image; including:

[0164] When the aspect ratio data of the pre-training sample image is equal to the lower limit of the preset ratio range, the pixel points of the pre-training sample image are divided into a plurality of pixel rows based on the vertical direction, the pixel rows are up-sampled to obtain an up-sampled text image, and the central area of ​​the up-sampled text image is cropped with a preset size to obtain a sample image;

[0165] When the aspect ratio data of the pre-trained sample image is equal to the upper limit of the preset ratio range, the pixel rows of the pre-trained sample image are under-sampled based on the vertical direction to obtain an under-sampled text image, and the central area of ​​the under-sampled text image is cropped with a preset size to obtain a sample image.

[0166] In the above embodiments, when the aspect ratio data of the pre-trained sample image is equal to the lower limit of the preset ratio range, the pixel rows of the pre-trained sample image are upsampled based on the vertical direction to obtain an upsampled text image, and the central area of ​​the upsampled text image is cropped to a preset size to obtain a sample image.

[0167] In the above embodiments, when the aspect ratio data of the pre-trained sample image is equal to the upper limit of the preset ratio range, the pixel rows of the pre-trained sample image are undersampled based on the vertical direction to obtain an undersampled text image, and the central area of ​​the undersampled text image is cropped to a preset size to obtain a sample image.

[0168] The beneficial effect of the above technology is that when the aspect ratio data of the pre-trained sample image is equal to the lower limit or upper limit of the preset ratio range, upsampling and downsampling are performed respectively, so that the aspect ratio data of the acquired sample image is within the preset ratio range, thereby avoiding large differences in the aspect ratios of the sample images used to train the model, causing the model to fall into a fixed paradigm and affecting the model prediction results.

[0169] Embodiment 8:

[0170] The embodiment of the present invention provides a method for detecting the direction of a text image based on deep learning, comprising the steps of: when the aspect ratio data of a pre-trained sample image is lower than the lower limit of a preset ratio interval, performing vertical image merging processing on the pre-trained sample image to obtain a sample image; comprising:

[0171] Copy the pre-training sample image to obtain a copy sample image, and merge it with the pre-training sample image in the vertical direction to obtain a vertical merged image;

[0172] Detecting height data and length data of the vertically merged image, and obtaining aspect ratio data of the vertically merged image;

[0173] When the aspect ratio data of the vertically merged image is within a preset ratio range, the central area of ​​the vertically merged image is cropped with a preset size to obtain a sample image; when the aspect ratio data of the vertically merged image is lower than the lower limit of the preset ratio range, a copy sample image is obtained and merged with the vertically merged image to obtain a second vertically merged image, until the aspect ratio data of the vertically merged image is within the preset ratio range.

[0174] In the above embodiments, when the aspect ratio data of the pre-trained sample image is lower than the lower limit of the preset ratio range, the pre-trained sample image is copied to obtain the copy sample image, which is merged with the pre-trained sample image in the vertical direction to obtain a vertically merged image, and when the aspect ratio data of the vertically merged image is within the preset ratio range, the central area of ​​the vertically merged image is cropped to a preset size to obtain a sample image.

[0175] In the above embodiment, when the aspect ratio data of the vertically merged image is lower than the lower limit of the preset ratio range, a copy sample image is obtained and merged with the vertically merged image to obtain a second vertically merged image until the aspect ratio data of the vertically merged image is within the preset ratio range.

[0176] In one embodiment, the second vertically merged image is Figure 3 shown.

[0177] The beneficial effect of the above technology is that when the aspect ratio data of the pre-training sample image is lower than the lower limit of the preset ratio range, the pre-training sample image is vertically merged so that the aspect ratio data of the acquired vertically merged image is within the preset ratio range, which facilitates the model to use sample images with aspect ratio data within the preset ratio range for training, improves the model's attention to the text features in the sample image, and thereby improves the trained model's prediction accuracy of the text direction.

[0178] In one embodiment, when the aspect ratio data of the pre-trained sample image is higher than the upper limit of a preset ratio range, the pre-trained sample image is subjected to horizontal image merging processing, specifically by obtaining a copy sample image, merging it with the pre-trained sample image in the horizontal direction to obtain a horizontally merged image, and when the aspect ratio data of the horizontally merged image is within the preset ratio range, the central area of ​​the horizontally merged image is cropped with a preset size to obtain a sample image.

[0179] Embodiment 9:

[0180] The embodiment of the present invention provides a text image direction detection method based on deep learning, comprising the steps of: an image direction detection model calculates the confidence of characters in each direction in the preprocessed text image according to the preprocessed text image, and obtains a direction confidence data group; comprising:

[0181] The image direction detection model extracts the image attention feature data of the preprocessed text image through the backbone network layer, and performs feature fusion on the image attention feature data through the neck network layer to obtain image feature output data, and selects the second detection layer of the detection head layer to obtain text direction detection data according to the image feature output data;

[0182] Calculate the confidence level based on the text direction detection data to obtain direction confidence level data;

[0183] Set the direction fitting interval;

[0184] Obtain the direction confidence data within the direction fitting interval, and select the text direction corresponding to the maximum value data as the predicted direction within the direction fitting interval;

[0185] The direction confidence data of all directions in the preprocessed text image are traversed and calculated through the direction fitting interval to obtain the predicted direction and the corresponding direction confidence data, and construct a direction confidence data group.

[0186] In the above embodiments, the image direction detection model processes the preprocessed text image through the backbone network layer and the neck network layer respectively, obtains the image feature output data corresponding to the preprocessed text image, and selects the second detection layer of the detection head layer to obtain the text direction detection data according to the image feature output data.

[0187] In the above embodiments, the second detection layer in the detection head layer outputs multiple prediction results of the image feature output data, which are constructed as text direction detection data.

[0188] In the above embodiments, a direction fitting interval is set, and the direction confidence data of all directions in the preprocessed text image are traversed and calculated. The text direction corresponding to the maximum value data of the direction confidence data within the direction fitting interval is selected as the predicted direction within the direction fitting interval, the predicted direction and the corresponding direction confidence data are obtained, and a direction confidence data group is constructed.

[0189] In one embodiment, the direction fitting interval is set to [0°, 5°), [5°, 10°), ... [355°, 0°), and the direction confidence data of all directions in the preprocessed text image are traversed and calculated. Only the unique maximum value data is selected in any interval in the direction fitting interval [0°, 5°), [5°, 10°), ... [355°, 0°), and the text direction corresponding to the maximum value data is used as the predicted direction of the direction fitting interval. 72 predicted directions and corresponding confidence data are obtained to construct a direction confidence data group.

[0190] In the above embodiments, the direction fitting interval can be freely set according to the image size of the pre-processed text image and the text information contained.

[0191] The beneficial effects of the above technology are: through the image direction detection model, it is possible to obtain text direction detection data based on the preprocessed text image, and by setting the direction fitting interval, the non-maximum data in the interval is suppressed, avoiding the appearance of several prediction directions with similar confidence levels in the direction fitting interval, which affects the text direction detection of the original text image, thereby improving the accuracy of the text detection direction.

[0192] In one embodiment, when the original text image is a paper Chinese image, the prediction accuracy of the text detection direction is 99.91%; when the original text image is a paper English image, the prediction accuracy of the text detection direction is 99.60%; when the original text image is an electronic screen Chinese image, the prediction accuracy of the text detection direction is 99.87%; when the original text image is an electronic screen English image, the prediction accuracy of the text detection direction is 99.56%.

[0193] A text image direction detection method based on deep learning proposed in the present invention effectively improves the accuracy and robustness of text direction detection through the design of training strategies and reasoning strategies, thereby improving the detection accuracy of text images and the stability in complex image scenes, facilitating the normal processing of subsequent related tasks.

[0194] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.

Claims

1. A text image direction detection method based on deep learning, characterized in that: include: Get the original text image; Cropping the original text image into a preset size to obtain a preprocessed text image; Inputting the preprocessed text image into an image direction detection model, calculating the confidence of the text in each direction in the preprocessed text image, and obtaining a direction confidence data set; Whether the maximum confidence data in the detection direction confidence data group meets the confidence threshold requirement, and when the maximum confidence data meets the confidence threshold requirement, the direction corresponding to the maximum confidence data is used as the text detection direction; When the maximum confidence data does not meet the confidence threshold requirement, the aspect ratio data of the original text image is detected, and the corresponding image merging operation is performed to obtain a pre-merged image, and the image is processed through the image direction detection model to calculate the confidence of the text in each direction in the pre-merged image to obtain the text detection direction.

2. The text image direction detection method based on deep learning according to claim 1 is characterized in that: The step of cropping the original text image with a preset size to obtain a preprocessed text image comprises: The central area of ​​the original text image is cropped with a preset size of 768*768px to obtain a preprocessed text image, where px is a pixel unit.

3. The text image direction detection method based on deep learning according to claim 1 is characterized in that: The step of inputting the preprocessed text image into an image direction detection model, calculating the confidence of the text in each direction in the preprocessed text image, and obtaining a direction confidence data set comprises: Build a pre-trained direction detection model; According to the sample text image, a rotation angle transformation is performed to obtain a sample transformation image corresponding to the sample text image, and a sample training set is constructed; The pre-trained direction detection model is trained using a sample training set to obtain an image direction detection model; The image direction detection model calculates the confidence of the characters in the preprocessed text image in each direction based on the preprocessed text image, and obtains a direction confidence data set.

4. The text image direction detection method based on deep learning according to claim 3 is characterized in that: The step of building a pre-trained direction detection model includes: Build the backbone network layer; optional, including: The feature extraction layer is constructed through two-dimensional convolution layer, normalization layer and activation function layer; Construct the feature fusion layer through the first convolution layer, the first segmentation layer, and the attention mechanism layer; Construct feature downsampling layer; The fast spatial pooling layer is constructed through the second convolutional layer, the maximum pooling layer and the third convolutional layer; The attention layer is constructed through the second segmentation layer, the multi-head attention mechanism layer and the feedforward neural network layer; The feature extraction layer, feature fusion layer, feature downsampling layer, fast spatial pooling layer, and attention layer are connected in sequence to construct a backbone network layer; Construct the neck network layer; optional, including: Construct feature upsampling layer; The feature output layer is constructed through the fourth convolution layer, the third segmentation layer and the grouped convolution layer; The feature upsampling layer and the feature output layer are connected to construct a neck network layer; Build the detection head layer; optional, including: Constructing a first detection layer through a first regressor and a first classifier; Constructing a second detection layer through a second regressor and a second classifier; The first detection layer and the second detection layer are connected in parallel to construct a detection head layer.

5. The text image direction detection method based on deep learning according to claim 4 is characterized in that: The step of performing rotation angle transformation according to the sample text image, obtaining a sample transformation image corresponding to the sample text image, and constructing a sample training set comprises: The sample text images include: Chinese paper text images, English paper text images, Chinese electronic screen text images, and English electronic screen text images; The sample text image is rotated by 90 degrees, 180 degrees and 270 degrees to obtain a sample transformation image corresponding to the sample text image, and a sample training set is constructed.

6. The method for detecting the direction of text images based on deep learning according to claim 5, characterized in that: The step of training the pre-trained direction detection model through a sample training set to obtain an image direction detection model comprises: Obtain a pre-training sample image in the sample training set, and detect whether the aspect ratio data of the pre-training sample image is within the preset ratio range When the pre-training sample image is within 768*768px, the central area of ​​the pre-training sample image is cropped to obtain a sample image; When the aspect ratio data of the pre-training sample image is lower than the lower limit of the preset ratio interval, performing vertical image merging processing on the pre-training sample image to obtain a sample image; When the aspect ratio data of the pre-training sample image is higher than the upper limit of the preset ratio interval, performing horizontal image merging processing on the pre-training sample image to obtain a sample image; Performing two-dimensional convolution processing on the sample images in the sample training set based on the width direction and the height direction through the two-dimensional convolution layer in the first feature extraction layer to obtain sample two-dimensional convolution data, and performing normalization processing through the normalization layer, and performing nonlinear mapping through the activation function layer to obtain feature extraction data; The first convolution layer in the feature fusion layer performs channel compression on the feature extraction data to obtain convolution feature data, the first segmentation layer segments the convolution feature data into first convolution feature data and second convolution feature data, the attention mechanism layer performs feature enhancement processing on the first convolution feature data to obtain first convolution enhancement data, and the first convolution enhancement data is feature-fused with the second convolution feature data to obtain convolution feature fusion data; The feature downsampling layer adjusts the channel dimension of the convolution feature fusion data through point-by-point convolution, and performs downsampling processing through depth-wise separable convolution to obtain feature sampling data; The fast spatial pooling layer converts the feature sampling data into one-dimensional feature sampling data through the second convolution layer, extracts corresponding feature parameters through several maximum pooling layers, fuses the feature parameters, and performs convolution processing through the third convolution layer to obtain pooled feature data; The attention layer decomposes the pooled feature data into first feature fusion data and second feature fusion data through the second segmentation layer, the multi-head attention mechanism layer performs attention mechanism adjustment on the first feature fusion data, extracts feature fusion parameters, and processes the data through the feedforward neural network layer to obtain attention feature data, and performs feature fusion on the attention feature data and the second feature fusion data to obtain attention fusion data; The feature upsampling layer of the neck network layer performs a feature upsampling operation on the attention fusion data to obtain feature expansion data; The feature output layer transforms the feature expansion data into a one-dimensional space through a fourth convolution layer to obtain one-dimensional feature expansion data, decomposes the one-dimensional feature expansion data into first feature expansion data and second feature expansion data through a third segmentation layer, distributes the one-dimensional feature expansion data to a plurality of convolution channels for convolution operation through a grouped convolution layer, and merges the obtained plurality of convolution feature sub-data into grouped convolution feature data through a one-dimensional convolution kernel, and fuses the obtained plurality of convolution feature sub-data with the second feature expansion data to obtain feature output data; The first detection layer in the detection head layer outputs data according to the feature to obtain first prediction data; the second detection layer outputs data according to the feature to obtain second prediction data; the first prediction data only includes a unique prediction result, and the second prediction data includes multiple prediction results; Calculating first deviation data between the first predicted data and the standard test data corresponding to the sample image through a loss function, calculating second deviation data between the second predicted data and the standard test data corresponding to the sample image through a loss function, and optimizing the training parameters of the backbone network layer and the neck network layer through the first deviation data and the second deviation data; optionally, including: Obtain the prediction direction θ1 and the prediction text center point (x1, y1) in the first prediction data; Obtain the standard direction θ2 and the standard text center point (x2, y2) in the standard test data; Wherein, L is the loss function, |θ1-θ2| is the direction deviation, and d is the diagonal distance of the sample image.

7. The method for detecting the direction of text images based on deep learning according to claim 6, characterized in that: The step of: obtaining a pre-training sample image in the sample training set, and detecting that the aspect ratio data of the pre-training sample image is within the preset ratio range When the pre-training sample image is within the range of 768*768px, the central area of ​​the pre-training sample image is cropped to obtain a sample image; including: When the aspect ratio data of the pre-training sample image is equal to the lower limit of the preset ratio range, the pixel points of the pre-training sample image are divided into a plurality of pixel rows based on the vertical direction, the pixel rows are up-sampled to obtain an up-sampled text image, and the central area of ​​the up-sampled text image is cropped with a preset size to obtain a sample image; When the aspect ratio data of the pre-trained sample image is equal to the upper limit of the preset ratio range, the pixel rows of the pre-trained sample image are under-sampled based on the vertical direction to obtain an under-sampled text image, and the central area of ​​the under-sampled text image is cropped with a preset size to obtain a sample image.

8. The method for detecting the direction of text images based on deep learning according to claim 6, characterized in that: The step of: performing vertical image merging processing on the pre-training sample image to obtain a sample image when the aspect ratio data of the pre-training sample image is lower than the lower limit of the preset ratio interval; comprises: Copy the pre-training sample image to obtain a copy sample image, and merge it with the pre-training sample image in a vertical direction to obtain a vertical merged image; Detecting height data and length data of the vertically merged image to obtain aspect ratio data of the vertically merged image; When the aspect ratio data of the vertically merged image is within the preset ratio range, the central area of ​​the vertically merged image is cropped with a preset size to obtain a sample image; when the aspect ratio data of the vertically merged image is lower than the lower limit of the preset ratio range, the copy sample image is obtained and merged with the vertically merged image to obtain a second vertically merged image, until the aspect ratio data of the vertically merged image is within the preset ratio range.

9. The method for detecting the direction of text images based on deep learning according to claim 4, characterized in that: The step: the image direction detection model calculates the confidence of the characters in each direction in the preprocessed text image according to the preprocessed text image, and obtains a direction confidence data set; comprises: The image direction detection model extracts the image attention feature data of the preprocessed text image through the backbone network layer, performs feature fusion on the image attention feature data through the neck network layer, obtains image feature output data, and selects the second detection layer of the detection head layer to obtain text direction detection data according to the image feature output data; Performing confidence calculation based on the text direction detection data to obtain direction confidence data; Set the direction fitting interval; Obtaining direction confidence data within the direction fitting interval, and selecting the text direction corresponding to the maximum value data therein as the predicted direction within the direction fitting interval; The direction confidence data of all directions in the preprocessed text image are traversed and calculated through the direction fitting interval to obtain the predicted direction and the corresponding direction confidence data, and construct a direction confidence data group.

Images