Invoice text detection model training method and invoice text detection method

Abstract

The application provides a training method of a document text detection model and a document text detection method. The training method comprises the following steps: obtaining a training set image, wherein the training set image comprises a document image and a position label corresponding to a text region of the document image; performing feature extraction on each document image to obtain a first feature map; performing feature fusion on the first feature map to obtain a second feature map; obtaining a probability map and a threshold value map based on the second feature map, and performing differentiable binarization processing on the probability map and the threshold value map to obtain a binary image and a text region position coordinate of the document image; and iteratively updating each parameter based on the position label, the text region position coordinate and a preset loss function, and obtaining a target document text detection model when the preset loss function converges. The application can effectively improve the training precision of the model, and further improve the detection precision of the document text.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image text detection, and particularly relates to a training method of a document text detection model and a document text detection method. BACKGROUND

[0002] A document contains various important information and is an important data source of a financial system, and plays a very important role in auditing, tax payment, accounting and reimbursement, etc. Therefore, automatic and accurate detection of document text information is of great significance to smart finance. At present, with the aid of image processing technology and optical character recognition (OCR) technology, text recognition and analysis of a document image can be realized, key text information can be extracted, and document processing automation can be realized, which greatly shortens the time of a financial staff to process a document, reduces the workload and improves the work efficiency.

[0003] With the continuous development of deep learning technology, image text detection based on deep learning has become a current research hotspot. At present, the image text detection method based on deep learning mainly adopts two image processing strategies: one is based on the "region proposal" strategy in the target detection algorithm; and the other is to adopt an image semantic segmentation strategy.

[0004] Based on the "region proposal" strategy in the target detection algorithm, a general target detection network is used as a basic model, and the algorithm is improved in combination with the actual application of text detection on the basis of the general target detection network, such as adjusting the general multi-class target detection model to a single-class (text) detection model. For example, the common target detection model Faster R-CNN has the following basic process: 1) CNN (Convolutional Neural Network) image feature extraction; 2) candidate region RoI (Region of interest) and candidate box generation; 3) generating a candidate box score through a classifier; and 4) removing redundant candidate boxes through a non-maximum suppression method (NMS) to obtain a final detection result. The target detection model for text detection also includes SSD (Single Shot MultiBox Detector), YOLO (You Only Look Once), etc.

[0005] The development of image segmentation technology is from the early extraction of shallow features based on RGB images, to the pixel-level processing by convolutional neural networks using traditional segmentation algorithms, and then to the end-to-end image segmentation by the fully convolutional deep neural network (FCN) proposed by Shelhamer et al. Compared with the general object detection using the anchor box mechanism to regress the positioning mode, the FCN pays more attention to the pixel-level labeling information of the text, and thus can be used for image text detection in a complex background. In addition, with the great success of the Transformer in natural language processing, it has also been introduced into image processing tasks.

[0006] However, although certain effects have been achieved in text detection at present, the invoice text image is different from the ordinary document image. In addition to text, the invoice text image contains a large amount of noise such as handwritten signature, seal image, ink stain, etc., and has the characteristics of dense text lines, large text scale variation, and various text font styles. In addition, due to the influence of factors such as old paper invoices, insufficient scanning, paper creases, etc., the invoice text image has blurring and text deformation, which greatly affects the text detection work. Moreover, the existing text detection model has the following problems: (1) low detection accuracy. Unlike general text detection tasks, the invoice image contains important information, and text detection errors will have a great impact and consequences on subsequent data processing and analysis, so the invoice text detection and recognition algorithm must have high recognition accuracy to ensure the accuracy of the recognition result. (2) Inaccurate text edge positioning in complex background. Due to the complex background, there are many interference factors, and the existing text detection algorithm often only has high text detection accuracy in a specific environment, so for the invoice text image, the existing text detection algorithm is difficult to adapt to the needs of automatic detection of intelligent financial invoices.

[0007] Therefore, it is of great significance and application value to design an invoice text detection method with high accuracy and robustness to realize automatic and accurate detection of invoice text information for intelligent finance. SUMMARY

[0008] The embodiments of the present application provide a training method of an invoice text detection model and an invoice text detection method to solve the problem of low invoice text detection accuracy in the prior art.

[0009] In a first aspect, the embodiments of the present application provide a training method of an invoice text detection model, comprising:

[0010] obtaining a training set image, wherein the training set image comprises an invoice image and a position label corresponding to a text region in the invoice image;

[0011] For each of the invoice images in the training set images, feature extraction is performed based on a preset encoder module to obtain first feature maps of different scales corresponding to the invoice image;

[0012] The first feature maps of different scales are fused from top to bottom and from bottom to top based on the cascade fusion attention module and the local-global fusion attention module to obtain second feature maps.

[0013] The probability map and the threshold map are obtained based on the second feature maps, and the probability map and the threshold map are subjected to differentiable binarization processing to obtain a binary map and text region position coordinates of the invoice image predicted based on the binary map.

[0014] The parameters of the preset encoder module, the cascade fusion attention module and the local-global fusion attention module are iteratively updated based on the position label, the text region position coordinates and a preset loss function, and when the preset loss function converges, a trained target invoice text detection model is obtained.

[0015] In a second aspect, an embodiment of the present application provides an invoice text detection method, comprising:

[0016] An invoice image to be detected is obtained.

[0017] The invoice image to be detected is subjected to blur denoising processing, grayscale processing and data enhancement processing to obtain a first invoice image to be detected.

[0018] The first invoice image to be detected is input into a target invoice text detection model to output text region position coordinates corresponding to the first invoice image to be detected; wherein the target invoice text detection model is trained based on the invoice text detection model training method according to the first aspect or any possible implementation manner of the first aspect.

[0019] In a third aspect, an embodiment of the present application provides a training device of an invoice text detection model, comprising:

[0020] An image acquisition module is configured to acquire training set images, wherein the training set images include invoice images and position labels corresponding to text regions in the invoice images.

[0021] A feature extraction module is configured to, for each of the invoice images in the training set images, perform feature extraction based on a preset encoder module to obtain first feature maps of different scales corresponding to the invoice image.

[0022] A feature fusion module is configured to fuse the first feature maps of different scales from top to bottom and from bottom to top based on a cascade fusion attention module and a local-global fusion attention module to obtain second feature maps.

[0023] a text detection module, configured to obtain a probability map and a threshold map based on the second feature map, and perform differentiable binarization processing on the probability map and the threshold map to obtain a binary map and text region position coordinates of the document image predicted based on the binary map;

[0024] a model correction module, configured to iteratively update parameters of the preset encoder module, the cascaded fusion attention module and the local-global fusion attention module based on the position label, the text region position coordinates and a preset loss function, and obtain a trained target document text detection model when the preset loss function converges.

[0025] In a fourth aspect, an embodiment of the present application provides a document text detection device, comprising:

[0026] a document image acquisition module, configured to acquire a to-be-detected document image;

[0027] an image preprocessing module, configured to perform blur denoising processing, grayscale processing and data enhancement processing on the to-be-detected document image to obtain a first to-be-detected document image;

[0028] an image text detection module, configured to input the first to-be-detected document image into a target document text detection model to output text region position coordinates corresponding to the first to-be-detected document image; wherein the target document text detection model is trained based on the training method of the document text detection model according to the first aspect or any possible implementation manner of the first aspect.

[0029] In a fifth aspect, an embodiment of the present application provides an electronic device, comprising a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor implements steps of the method according to the first aspect or any possible implementation manner of the first aspect when executing the computer program.

[0030] In a sixth aspect, an embodiment of the present application provides an electronic device, comprising a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor implements steps of the method according to the second aspect when executing the computer program.

[0031] In a seventh aspect, an embodiment of the present application provides a computer readable storage medium, which stores a computer program, wherein the computer program is executable on a processor to implement steps of the method according to the first aspect, the second aspect or any possible implementation manner of the first aspect.

[0032] The embodiment of the present application provides a kind of bill text detection model training method and bill text detection method, by obtaining training set image, wherein, training set image includes bill image and the position label corresponding to text area in bill image;For each bill image in training set image, feature extraction is carried out based on preset encoder module, to obtain the first feature map of different scales corresponding to the bill image;Different scale first feature map is carried out from top to bottom and from bottom to top feature fusion based on cascaded fusion attention module and local-global fusion attention module, to obtain second feature map;Probability map and threshold map are obtained based on second feature map, and the differentiable binarization processing is carried out to probability map and threshold map, to obtain binary image and the text area position coordinates of bill image predicted based on binary image;The parameters of preset encoder module, cascaded fusion attention module and local-global fusion attention module are iteratively updated based on position label, text area position coordinates and preset loss function, when preset loss function converges, the target bill text detection model trained is obtained.Based on the bill text detection model training method provided by the embodiment of the present application, the target bill text detection model trained can obtain accurate semantic information and spatial information in the process of feature extraction of training set image, to ensure the accurate positioning of image text edge;And based on the feature fusion from top to bottom and from bottom to top to first feature map, the accuracy of text detection can be effectively improved;Then in the training process of model, the parameters of each item of model are iteratively updated based on loss function, effectively improve the training precision of model, to further improve the detection precision of bill text. BRIEF DESCRIPTION OF DRAWINGS

[0033] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following will briefly introduce the drawings needed to be used in the embodiments or prior art description. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can also be obtained by those skilled in the art without creative labor.

[0034] Figure 1 It is the implementation flowchart of the bill text detection model training method provided by the embodiment of the present application;

[0035] Figure 2 It is the architecture schematic diagram of the target bill text detection model provided by the embodiment of the present application;

[0036] Figure 3 It is the architecture schematic diagram of the preset encoder module provided by the embodiment of the present application;

[0037] Figure 4 It is the architecture schematic diagram of the cascaded fusion attention module provided by the embodiment of the present application;

[0038] Figure 5 is an architecture schematic diagram of a local-global fusion attention module provided by an embodiment of the present application;

[0039] Figure 6 is a flow schematic diagram of a training method of a document text detection model provided by an embodiment of the present application;

[0040] Figure 7 is an implementation flowchart of a document text detection method provided by an embodiment of the present application;

[0041] Figure 8 is a structural schematic diagram of a training device of a document text detection model provided by an embodiment of the present application;

[0042] Figure 9 is a structural schematic diagram of a document text detection device provided by an embodiment of the present application;

[0043] Figure 10 is a schematic diagram of an electronic device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0044] In the following description, specific details are set forth, such as particular system configurations, techniques, etc., in order to provide a thorough understanding of the present application embodiments. However, persons skilled in the art will understand that the present application can be practiced in other embodiments that depart from these specific details. In other instances, detailed descriptions of well-known systems, devices, circuits, and methods are omitted so as not to obscure the description of the present application with unnecessary detail.

[0045] In order to make the objectives, technical solutions and advantages of the present application clearer, the following will be described by specific embodiments in conjunction with the accompanying drawings.

[0046] An embodiment of the present application provides a training method of a document text detection model, Figure 1 is an implementation flowchart of a training method of a document text detection model provided by an embodiment of the present application; in combination with Figure 1 The training method is described in detail as follows.

[0047] Step 101: acquiring a training set image, wherein the training set image includes a document image and a position label corresponding to a text region in the document image.

[0048] In step 101, since text detection is performed in units of text regions, in this embodiment, when establishing a training set image, each region on the document image needs to be labeled to establish a text label. For example, the position label of the characters can be given in the form of a quadrilateral frame (or an elliptical frame or other shapes), in which the left upper, right upper, right lower and left lower are respectively corresponded in a clockwise direction, and then the corresponding text content in each region is labeled, and if there is ambiguous and indistinguishable content, it can be replaced by a symbol such as “###”. Therefore, in this embodiment, the training set image is obtained, that is, the document image and the position label corresponding to the text region of the document image are obtained.

[0049] In addition, after obtaining the training set image, since the text in the document image has many problems, such as insufficient clarity of the document image, or different text colors of the document image, etc. In order to solve these problems and improve the use accuracy of the document image, and thus improve the accuracy of text detection, therefore, before inputting the document image into the preset document text detection model for detection, the image can be preprocessed by a related algorithm. For example, the original document image can be blurred and denoised, and gray processed, to correspondingly solve the above-mentioned problems of clarity of the text of the document image and the text color of the document image; the original image can also be processed by data enhancement, to indirectly improve the robustness of the document text detection model.

[0050] Specifically, for blurred denoising: since the document paper has color or stains, which may have an adverse effect on subsequent image processing, therefore, the document image can be pre-processed for blurred denoising. For example, the denoising algorithm can use a normal probability distribution convolution kernel for convolution calculation, to slightly blur the pixel gray value of the image, so as to eliminate small range color noise and achieve the effect of denoising. The essence of image blurred denoising is to filter high frequency signals and retain low frequency signals, and the Gaussian kernel is a convolution filter used for filtering. The convolution kernel is a square pixel matrix, in which the size of the pixel value is the value on the two-dimensional normal distribution curve. When Gaussian blurring, the convolution kernel is multiplied by the region of the original image to be convolved to obtain the blurred image. Since its calculation is simple and the effect is robust, it can be used as a preprocessing algorithm to eliminate redundant information in image processing.

[0051] For gray processing: after the image is converted to gray, each pixel in the new image is represented by 8 bits (or 8 bits), with a value of 0 representing black, a value of 255 representing white, and other numbers representing different degrees of gray. The conversion formula is as follows:

[0052]

[0053] Wherein, L represents the image brightness of the document image after the gray processing, R represents the red channel of the document image, G represents the green channel of the document image, and B represents the blue channel of the document image.

[0054] For data enhancement processing: in the embodiment, in order to make full use of data, data enhancement processing can be performed on the training set images. Exemplarily, for each input image (i.e. training set image), enter five different data enhancement pipelines with a certain probability, and perform random scaling of 0.5, 1.0, 2.0, 3.0, random flipping, then randomly select an angle between (-10°, 10°) for angle rotation, select a reasonable center point, and perform cropping, etc. The document image after data enhancement has different scales, different text orientations, and different angles. In addition, blank data may occur in the cropping process, which can well improve the robustness of the model.

[0055] In the embodiment, the document image with high use accuracy and the position label corresponding to the text region in the document image are obtained, so as to facilitate the iterative correction of the parameters in the subsequent document text detection model, and further ensure the training accuracy of the model to more accurately detect the document text.

[0056] Step 102: for each document image in the training set image, feature extraction is performed based on the preset encoder module to obtain a first feature map of different scales corresponding to the document image.

[0057] In step 102, Figure 2 The architecture schematic diagram of the target document text detection model provided by the embodiment of the present application is as follows, Figure 3 The architecture schematic diagram of the preset encoder module provided by the embodiment of the present application is as follows; please refer to Figure 1 and Figure 3 For each document image in the training set image, the document image is input into the preset encoder module for feature extraction to obtain a first feature map of different scales corresponding to the document image. In the embodiment, exemplarily, the Segformer is used as the main part of the preset encoder module. On the one hand, the multi-head attention is used to solve the problem that the global semantic information of the image is not fully utilized in the CNN. On the other hand, the hierarchical Transformer encoder without position encoding is adopted to avoid complex design, thereby facilitating efficient training and high-performance detection of the document text detection model. Unlike the Vision Transformer (ViT) which can only generate a single resolution feature map, the Segformer generates four feature maps of different resolutions in the four stages of the encoder (i.e. Figure 2The four Transformer blocks of the Transformer Block1-Transformer Block4 are first processed by the hierarchical Segformer to obtain four feature maps of different scales, and then the obtained feature maps are processed by a multilayer perceptron (MLP) to generate feature maps M1, M2, M3 and M4 of different resolutions (as shown in Figure 2 In this embodiment, the scales corresponding to the feature maps M1, M2, M3 and M4 are 1 / 4, 1 / 8, 1 / 16 and 1 / 32 of the original image, respectively, and the channel numbers are 32, 64, 128 and 256, respectively. Through the low-resolution feature map, the position of the center region of the text can be determined, and through the high-resolution feature map, the edge information of the text region and the position information between adjacent characters can be obtained. As shown in Figure 3 A Transformer block is connected by N efficient self-attention modules and a mix-feed-forward network (Mix-FFN), and then an overlapped patch merging module is used to merge the overlapped image blocks to maintain the local continuity around the image blocks.

[0058] Specifically, for the efficient self-attention module: in the original multi-head self-attention process, Q, K and V (Q represents a Query vector, K represents a Key vector, and V represents a Value vector. Q, K and V are three matrices.) are the three inputs of the efficient self-attention module, each head Q, K and V has the same dimension NxC, where N=HxW is the size of the image, C represents the channel number of the image, H represents the height of the image, W represents the width of the image, and the self-attention estimation is:

[0059]

[0060] wherein, represents the self-attention, represents a normalization function, represents the dimension of the Key vector, represents the transpose of the input K. The computational complexity of this process is For high-resolution images, the amount of parameters generated will be quite large, therefore, the Segformer introduces a scaling factor r, which reduces the dimension of the input image by r times under the condition of ensuring the same channel number, so that the complexity of the self-attention mechanism is reduced from to, and the specific process is as follows:

[0061]

[0062] wherein K represents the result after scaling change, that is, the K input of the efficient self-attention module, represents a normalization function, represents the original input image, represents a convolution operation with a convolution kernel size of a, a stride of a, and padding of 0. In addition, is a function in pytorch, which can change the dimension of the input, equivalent to a full connection layer.

[0063] For the mixed feed-forward network: since the resolution of the position encoding (PE) of the ViT is fixed, when the resolution of the training set image is different from the resolution of the test set image of the test document text detection model to be obtained subsequently, the position encoding needs to be interpolated, which may cause the training accuracy of the model to decrease. Therefore, in the Segformer, the position encoding is removed, and the Mix-FFN is introduced, which mixes a 3x3 convolution and an MLP at each feed-forward network (FFN) layer. The specific process is as follows: the input is the output result of the connected self-attention module, which is first processed by a full connection layer for dimension reduction, then processed by a 3x3 convolution operation for feature extraction, and finally processed by a GELU activation function and a full connection layer to restore to the input dimension of the document image.

[0064] For the overlapping merging module: if the direct downsampling operation is performed according to the fusion strategy in the ViT (Vision Transformer), the spatial consistency between blocks will be lost, therefore, in the Segformer, 7x7 and 3x3 convolutions are used to learn the position relationship between adjacent image blocks while performing downsampling operation. Through the fusion of this module, feature maps with different resolutions can be obtained.

[0065] In this embodiment, by proposing a hierarchical encoder based on Segformer, a position encoding-free manner is adopted to generate multiple feature maps with different resolutions, so as to obtain accurate semantic information and spatial information, and effectively solve the problem of inaccurate text edge positioning.

[0066] Step 103: based on the cascaded fusion attention module and the local-global fusion attention module, the first feature maps of different scales are fused from top to bottom and from bottom to top to obtain the second feature maps.

[0067] In step 103, Figure 4is a schematic diagram of an architecture of a cascading fusion attention module provided by an embodiment of the present application, Figure 5 is a schematic diagram of an architecture of a local-global fusion attention module provided by an embodiment of the present application, please refer to Figures 1 to 5 In this embodiment, based on the cascading fusion attention module (Cascading Fusion Attention, CFA), a top-down feature fusion operation is performed on the first feature maps of different scales, and then based on the local-global fusion attention module (Dual-Detail and Global Fusion Model, D-DGF), a bottom-up feature fusion operation is performed on the feature maps introduced by the cascading fusion attention module, and then the second feature maps are obtained.

[0068] In this embodiment, taking the encoder in the preset encoder module as an example, the encoder is a Segformer-based encoder, to match the four-level structure of the encoder, a decoder with an enhanced feature pyramid structure is proposed. Specifically, the decoder adopts a four-layer feature pyramid structure for feature fusion, introduces a CFA module between every two first feature maps, uses global average pooling to realize weight selection between channels, and uses convolution to enhance the information in the spatial and channel dimensions. To some extent, the problem of image information loss in the decoding process is effectively avoided. In addition, due to the difference in resolution of each feature map, the spatial information and semantic information it has are also different. Therefore, by introducing a D-DGF module, the weights are adaptively adjusted, so that the feature maps with different receptive fields can be more efficiently fused, effectively improving the training accuracy of the single document text detection model, and thus improving the accuracy of text detection.

[0069] In a possible implementation, based on the cascading fusion attention module and the local-global fusion attention module, a top-down and bottom-up feature fusion is performed on the first feature maps of different scales to obtain the second feature maps, including:

[0070] In a top-down order, the adjacent high-level feature maps and low-level feature maps in the first feature maps of different scales are subjected to global average pooling processing and maximum pooling processing to obtain the first intermediate feature maps of different scales.

[0071] The first intermediate feature maps are upsampled to obtain the corresponding second intermediate feature maps of the same scale.

[0072] In a bottom-up order, the adjacent low-level detail feature maps and high-level spatial feature maps in the second intermediate feature maps are divided into a group, and each group of second intermediate feature maps is subjected to global average pooling processing and detail feature protection processing to obtain the third intermediate feature maps.

[0073] The global average pooling processing and the detail feature protection processing are performed on all the third intermediate feature maps to obtain second feature maps.

[0074] In the embodiment, please refer to Figures 1 to 5 , considering that the text region of the document image is dense in Chinese and the interval of adjacent texts is small, in order to fully utilize the spatial information and semantic information of different levels of features in the decoding process, the global average pooling processing and the maximum pooling processing are performed on adjacent high-level feature maps and low-level feature maps in different scales in a top-down order to obtain first intermediate feature maps of different scales. In the embodiment, the CFA module is introduced between the adjacent high-level feature maps and the low-level feature maps, the global average pooling is used to realize the weight selection between channels, and then the convolution is used to enhance the information in the spatial dimension and the channel dimension, which to some extent, effectively avoids the problem of information loss of the document image in the decoding process.

[0075] Specifically, the CFA module is composed of a global attention (GA) module and a spatial attention (SA) module, and the inputs are two high-low scale feature maps and , and the output is the fused feature . The basic idea is to realize the channel attention in multiple scales through average pooling and maximum pooling.

[0076] More specifically, in the global attention (GA) module, the global information of the channel can be obtained through the global average pooling, but the single pooling method will cause information loss, therefore, the spatial attention (SA) module is adopted to introduce the position information between pixels into the pooling process to obtain accurate weight information.

[0077] In the embodiment, in the top-down path, the cascaded fusion attention module is introduced between the first feature maps at each layer to fuse the bottom features and the high-level features, and the result is input to the next layer structure for the feature fusion operation of the low-level features and the high-level features, so as to improve the correlation between different feature channels and increase the accuracy of text edge positioning. For example, three CFA modules are added in the document text detection model in the embodiment to realize the merging of the low-level feature maps and the three high-level feature maps.

[0078] In the embodiment, after obtaining the first intermediate feature maps of different scales, the resolutions of the first intermediate feature maps are different, therefore, in order to retain more available information, the upsampling operation is performed on all the first intermediate feature maps to obtain second intermediate feature maps of the same scale corresponding to each first intermediate feature map. For example Figure 2As shown, in this embodiment, the four first intermediate feature maps are upsampled respectively, and the output second intermediate feature maps D1, D2, D3 and D4 are unified to one-quarter the size of the original image.

[0079] In this embodiment, as Figure 5 As shown, after obtaining the second intermediate feature maps of the same scale, since each second intermediate feature map carries different information—high-level feature maps focus on carrying spatial location information, while low-level feature maps focus on carrying detail information—in order to effectively fuse the feature maps at all levels, the second intermediate feature maps D1, D2, D3, and D4 are first divided into two groups. For example, the grouping format can be: D1 and D2 in one group, denoted as the first group; and D3 and D4 in another group, denoted as the second group. In the two groups of feature maps, D1 and D3 have a higher proportion of detail information, while D2 and D4 have a higher proportion of spatial information. Therefore, following a bottom-up order, global average pooling and detail feature preservation processing are performed on the first and second groups of second intermediate feature maps respectively to obtain the third intermediate feature maps C1 and C2 (i.e., feature map C1 with a higher proportion of detail information and feature map C2 with a higher proportion of spatial information). Then, global average pooling and detail feature preservation processing are performed on C1 and C2 to obtain the final second feature map (i.e.,... Figure 5 (out in the middle).

[0080] In this embodiment, in the bottom-up path, based on the D-DGF module, the first stage fuses two low-resolution feature maps and two high-resolution feature maps to generate a spatial information map and a semantic information map, respectively. The second stage fuses the spatial information map and the semantic information map to obtain the final second feature map. By adaptively adjusting the weights of different feature maps, feature maps with different receptive fields are fused more efficiently to improve the accuracy of text detection.

[0081] In one possible implementation, following a top-down order, adjacent high-level and low-level feature maps in the first feature maps of different scales are subjected to global average pooling and max pooling to obtain first intermediate feature maps of different scales, including:

[0082] based on Global average pooling and max pooling are performed on adjacent high-level and low-level feature maps in the first feature maps at different scales to obtain first intermediate feature maps at different scales.

[0083] in, This represents the global features of adjacent high-level and low-level feature maps in the first feature map at different scales. Indicates the first The first feature map, and represents the first feature map relative to the first feature map. The low-level feature map of the first feature map, Indicates the first The first feature map, and represents the first feature map relative to the first feature map. High-level feature maps of the first feature map, This represents an element-by-element addition operation. This indicates global average pooling. This indicates that the convolution kernel size is 1×1, the input channel is C, and the output channel is [missing information]. Depth convolution operation, Represents the ReLU activation function. This indicates that the convolution kernel size is 1×1 and the input channels are... A depthwise convolution operation with output channel C. This represents the local features of adjacent high-level and low-level feature maps in the first feature map at different scales. This indicates a max pooling operation. This indicates the average pooling operation. This indicates a splicing operation. This indicates a convolution operation with a kernel size of 7×7, 2 input channels, and 1 output channel. This represents the Sigmoid activation function. This represents the fusion weights of the low-level and high-level feature maps. This indicates the element-wise multiplication operation. This represents the first intermediate feature map.

[0084] In this embodiment, the global features of the document text image are obtained through the GA module. Specifically, global average pooling is used to obtain the global information of the channels. First, the image of size C×H×W is compressed into a scalar of size C×1×1 through the global average attention module; then, a Conv layer with a convolution kernel of 1×1 is used to compress the number of channels back to the original size. Then, after ReLU activation, a deep convolutional layer (Point-Wise conv, PW) is applied to restore the image to its original dimensions. The specific formula is as follows:

[0085]

[0086] in, This represents the global features of adjacent high-level and low-level feature maps in the first feature map at different scales. Indicates the first The first feature map, and represents the first feature map relative to the first feature map. The low-level feature map of the first feature map, Indicates the first The first feature map, and represents the first feature map relative to the first feature map. High-level feature maps of the first feature map, This represents an element-by-element addition operation. denotes a global average pooling process, denotes a depth convolution operation with a kernel size of 1x1, an input channel of C, and an output channel of denotes a ReLU activation function, denotes a depth convolution operation with a kernel size of 1x1, an input channel of , and an output channel of C.

[0087] The above processing operations can better fit the nonlinear relationship between channels, and also do not increase a large amount of calculation. However, the result after the global average pooling is relatively rough, and is more inclined to emphasize large objects of global distribution, and ignores some small-scale detailed information.

[0088] Therefore, in order to improve the accuracy of subsequent invoice text detection, the SA module is introduced in the embodiment, and the SA module is connected in parallel with the GA module, and the spatial detailed information is increased to reduce the missed detection phenomenon of small-scale text. Unlike the GA module, the SA module adopts a channel dimension pooling operation, that is, the pixels at the same position of all channels are averaged or maximized. For an input image of CxHxW, two feature maps of 1xHxW can be obtained after maximum pooling and average pooling, respectively, and then the spliced feature map is fused by a 7x7 convolution. The specific formula is as follows:

[0089]

[0090] wherein, denotes local features of adjacent high-level feature maps and low-level feature maps in the first feature map of different scales, denotes a maximum pooling operation, denotes an average pooling operation, denotes a splicing operation, denotes a convolution operation with a kernel size of 7x7, an input channel of 2, and an output channel of 1, denotes a Sigmoid activation function.

[0091] Then, the first intermediate feature maps of different scales are obtained based on the outputs of the GA module and the SA module. The specific formula is as follows:

[0092]

[0093] wherein, denotes a fusion weight of the low-level feature map and the high-level feature map, which consists of real numbers between 0 and 1, and is used in combination with 1- to enable the network to perform weighted averaging between and ​​This indicates the element-wise multiplication operation. This represents the first intermediate feature map.

[0094] In this embodiment, by introducing a CFA module between layers, global average pooling is used to select weights between channels, and convolution is used to enhance information in both spatial and channel dimensions. To a certain extent, this effectively solves the problem of information loss during decoding.

[0095] In one possible implementation, adjacent low-level detail feature maps and high-level spatial feature maps in the second intermediate feature map are grouped together in a bottom-up order. Global average pooling and detail preservation processing are then performed on each group of second intermediate feature maps to obtain the third intermediate feature map, which includes:

[0096] based on The adjacent low-level detail feature maps and high-level spatial feature maps in the second intermediate feature map are grouped together, and global average pooling and detail feature protection processing are performed on each group of second intermediate feature maps to obtain the third intermediate feature map.

[0097] in, This represents the global features of adjacent low-level detail feature maps and high-level spatial feature maps in the second intermediate feature map. Indicates the first The second intermediate feature map, and represents the second intermediate feature map relative to the first intermediate feature map. The low-level detail feature map of the second intermediate feature map Indicates the first The second intermediate feature map, and represents the second intermediate feature map relative to the first intermediate feature map. The high-level spatial feature map of the second intermediate feature map This indicates global average pooling. This indicates that the convolution kernel size is 1×1, the input channel is C, and the output channel is [missing information]. Depth convolution operation, Represents the ReLU activation function. This indicates that the convolution kernel size is 1×1 and the input channels are... A depthwise convolution operation with output channel C. This represents the detailed features of the low-level detail map. This represents the transposed detail feature map. This represents the third intermediate feature map.

[0098] In this embodiment, the structure of the DGF module is as follows: Figure 5As shown, it consists of two modules: Global Attention (GA) and Detail Attention (DE). The GA module in DGF has the same function as the GA module in CFA, both obtaining global information of the channels through global average pooling. First, the global average attention module compresses the image of size C×H×W to a scalar of size C×1×1; then, a Conv layer with a 1×1 kernel is used to compress the number of channels back to the original size. Then, after ReLU activation, a deep convolutional layer (Point-Wise conv, PW) is applied to restore the image to its original dimensions. The specific formula is as follows:

[0099]

[0100] in, This represents the global features of adjacent low-level detail feature maps and high-level spatial feature maps in the second intermediate feature map. Indicates the first The second intermediate feature map, and represents the second intermediate feature map relative to the first intermediate feature map. The low-level detail feature map of the second intermediate feature map Indicates the first The second intermediate feature map, and represents the second intermediate feature map relative to the first intermediate feature map. The high-level spatial feature map of the second intermediate feature map This indicates global average pooling. This indicates that the convolution kernel size is 1×1, the input channel is C, and the output channel is [missing information]. Depth convolution operation, Represents the ReLU activation function. This indicates that the convolution kernel size is 1×1 and the input channels are... A depthwise convolution operation with output channel C.

[0101] However, since average pooling and max pooling operations cause information loss in the feature map during decoding, a DE module is introduced to preserve detailed features. The specific formula for the DE module operation is as follows:

[0102]

[0103] in, This represents the detailed features of the low-level detail map. This represents the transposed detail feature map.

[0104] Next, the third intermediate feature map is obtained, using the following formula:

[0105]

[0106] wherein, represents the third intermediate feature map.

[0107] In this embodiment, the D-DGF module adaptively adjusts the weights to enable more efficient fusion of feature maps with different receptive fields, thereby improving the accuracy of text detection.

[0108] Step 104: Obtain a probability map and a threshold map based on the second feature map, and perform differentiable binarization processing on the probability map and the threshold map to obtain a binary map and a text region position coordinate of the invoice image predicted based on the binary map.

[0109] In step 104, as shown in Figure 2 , the second feature map obtained based on the decoder part can generate two kinds of maps, a probability map and a threshold map. The probability map represents the probability that a pixel point is text, and the threshold map represents the threshold of each pixel point. Then, the probability map and the threshold map are subjected to differentiable binarization processing to obtain a binary map and a text region position coordinate of the invoice image predicted based on the binary map.

[0110] In this embodiment, as shown in Figure 2 , in the post-processing part, a differentiable binarization algorithm is used, and the purpose is to separate the text center area from the background, that is, to perform binarization processing on the pixels of the probability map, 0 for background and 1 for text region. The binarization process is as follows:

[0111]

[0112] wherein, represents the pixel of the binarized probability map, and t represents a preset pixel threshold.

[0113] Traditional binarization divides the text region and the background region by setting a fixed threshold, and the size of the threshold directly affects the final result. The main feature of the differentiable binarization algorithm is that the threshold information can be trained together with the network model, so as to adaptively determine the threshold of the pixels at different positions.

[0114] Generally, the threshold of the text region is large, and the threshold of the edge region is small, which can improve the efficiency of detecting text of any shape. Under the specific algorithm,

[0115]

[0116] wherein, represents the value of the approximate binary map at (i, j), represents the pixel value of the probability map, represents the pixel value of the threshold map.

[0117] Step 105: iteratively updating the parameters of the preset encoder module, the cascaded fusion attention module and the local-global fusion attention module based on the position label, the text region position coordinates and the preset loss function, and obtaining the trained target invoice text detection model when the preset loss function converges.

[0118] In step 105, the parameters in each module in the model are iteratively updated based on the position label, the text region position coordinates and the preset loss function, and the trained target invoice text detection model is obtained when the preset loss function converges. Illustratively, the preset loss function converges can be embodied as that the loss value of the preset loss function does not exceed a certain preset threshold; that is, the trained target invoice text detection model is obtained when the loss value of the preset loss function does not exceed a certain preset threshold.

[0119] Illustratively, the process of generating the position label for the probability map and the threshold map is described as follows: first, according to the rectangular region coordinates in the text, the original labeled text box G is generated, and then from the original labeled text box G, the Vatti clipping algorithm is used to generate the contracted text core region G s and the expanded text boundary region G d . The mask map filled with 1 inside G and 0 outside G is taken as the label of the probability map and the binary map, and the mask map filled with 0 between G s and G d is taken as the label of the threshold map.

[0120] In this embodiment, the specific calculation formula of the preset loss function can be:

[0121]

[0122] wherein, represents the weighted sum of the probability map loss, the binary map loss and the threshold map loss, is the loss function of the contracted text instance, is the loss function of the binary contracted text instance, is the loss function of the binary threshold map, represents the proportion of the binary map loss, represents the proportion of the threshold map loss.

[0123] The specific calculation formula of the cross-entropy loss function used by the probability map and the binary map can be:

[0124]

[0125] wherein, represents the position, represents the core region of the predicted text, express The predicted probability value of the location. Represents a probability graph Or a binary image exist The label value of the location.

[0126] The loss of a threshold map is specifically represented by the sum of the distances between the label value and the predicted value within the expanded polygon. The specific formula for calculating the loss function of a threshold map can be:

[0127]

[0128] in, This represents the text bounding box area between expansion and contraction. Indicates threshold map in The label value of the location, express The predicted threshold value for location.

[0129] In this embodiment, the model's parameters are continuously updated iteratively based on location labels, text region location coordinates, and a preset loss function, which can effectively improve the model's training accuracy and thus effectively improve the detection accuracy of document text.

[0130] Figure 6 This is a flowchart illustrating the training method of the document text detection model provided in an embodiment of the present invention, as shown below. Figure 6 As shown. In this embodiment, the automatic and accurate detection of document text mainly includes four parts: label generation, image preprocessing, model training, and model evaluation. The label generation part is mainly used to obtain the position labels corresponding to the text regions in the document image; the image preprocessing part may include blur denoising processing, grayscale processing, and / or data augmentation processing, mainly to ensure the clarity of the document image, so as to provide high-quality document images for subsequent model training and text prediction.

[0131] In the model training section, the input is a preprocessed document image, such as... Figure 2As shown, the model training part can be further divided into three modules: an encoder module, a decoder module, and a post-processing module. The encoder module is a backbone network based on Segformer, which generates feature maps M1, M2, M3, and M4 of different resolutions through four Transformer blocks: Transformer Block1 to Transformer Block4. The scales are 1 / 4, 1 / 8, 1 / 16, and 1 / 32 of the original image, and the channel numbers are 32, 64, 128, and 256, respectively. Then, M4 is sent to the second structure decoder module to obtain feature map F4. F4 is combined with M3 through the CFA module to generate F3, and then F3 is combined with M2 through the CFA module to generate F2, and F2 is combined with M1 through the CFA module to generate F1. Four feature maps F1, F2, F3, and F4 are obtained during the decoding process, with scales of 1 / 4, 1 / 8, 1 / 16, and 1 / 32 of the original image, respectively. In order to retain more available information, the four feature maps are respectively upsampled to the size of 1 / 4 of the original image. Then, the four feature maps are combined and output as a segmentation map (i.e., the second feature map mentioned above) through D-DGF, with a scale of 1 / 4 of the original image and a channel number of 256. Finally, in the post-processing module, the segmentation map generated by the decoder module can generate a probability map and a threshold map. The probability map represents the probability of a pixel being text, and the threshold map represents the threshold of each pixel. Then, the probability map and the threshold map are processed through differentiable binarization to obtain a binary map and the text region coordinates of the invoice image predicted based on the binary map. Finally, in the model evaluation part, based on the position label, the text region coordinates, and the preset loss function, the parameters of each module in the model are iteratively updated to improve the training accuracy of the invoice text detection model, so as to more accurately detect the text of the invoice.

[0132] The embodiment of the present application provides a kind of training method of bill text detection model, by obtaining training set image, wherein, training set image includes bill image and the position label corresponding to text area in bill image;For each bill image in training set image, feature extraction is carried out based on preset encoder module, to obtain the first feature map of different scales corresponding to the bill image;Different scale first feature map is carried out from top to bottom and from bottom to top feature fusion based on cascaded fusion attention module and local-global fusion attention module, to obtain second feature map;Probability map and threshold map are obtained based on second feature map, and the differentiable binarization processing is carried out to probability map and threshold map, to obtain binary image and the text area position coordinates of bill image predicted based on binary image;The parameters of preset encoder module, cascaded fusion attention module and local-global fusion attention module are iteratively updated based on position label, text area position coordinates and preset loss function, when preset loss function converges, the target bill text detection model trained is obtained.Based on the training method of bill text detection model provided by the embodiment of the present application, the target bill text detection model trained can obtain accurate semantic information and spatial information in the process of feature extraction to training set image, to ensure the accurate positioning of image text edge;And based on the feature fusion from top to bottom and from bottom to top to first feature map, the accuracy of text detection can be effectively improved;Then in the training process of model, the parameters of each item of model are iteratively updated based on loss function, effectively improve the training precision of model, to further improve the detection precision of bill text.

[0133] In addition, Figure 7 The implementation flowchart of the bill text detection method provided by the embodiment of the present application is shown as Figure 7 The embodiment of the present application provides a kind of bill text detection method, comprising:

[0134] Step 201: obtaining bill image to be detected.

[0135] In step 201, the bill image to be detected obtained is used as the input of bill text detection model.

[0136] Step 202: carrying out fuzzy denoising processing, gray processing and data enhancement processing to bill image to be detected, to obtain first bill image to be detected.

[0137] In step 202, the bill image to be detected can be preprocessed to obtain the first bill image to be detected, to facilitate subsequent improvement of text detection precision of bill text image.Exemplarily, preprocessing can include fuzzy denoising processing, gray processing and / or data enhancement processing, etc., which is not limited in the present application.

[0138] Step 203: Input the first document image to be detected into the target document text detection model and output the text region location coordinates corresponding to the first document image to be detected; wherein, the target document text detection model is trained based on the document text detection model training method described in the first aspect or any possible implementation of the first aspect above.

[0139] In step 203, the first document image to be detected is input into the target document text detection model. Based on the processing operations of each module in the model, the coordinates of the text region corresponding to the first document image to be detected are output. The target document text detection model is trained using the training method for document text detection models described in the first aspect or any possible implementation thereof. In this way, the detection of the text region of the document text image is completed.

[0140] This invention provides a document text detection method. The method involves acquiring a document image to be detected; performing blurring and denoising processing, grayscale processing, and data augmentation on the document image to obtain a first document image to be detected; inputting the first document image to be detected into a target document text detection model, and outputting the coordinates of the text region corresponding to the first document image; wherein the target document text detection model is trained based on the document text detection model training method described in the first aspect or any possible implementation thereof. Based on the document text detection method provided by this invention, the text region of a document image can be accurately detected.

[0141] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0142] The following are device embodiments of the present invention. For details not described in detail, please refer to the corresponding method embodiments described above.

[0143] Figure 8 This is a schematic diagram of the training device for the document text detection model provided in an embodiment of the present invention. For ease of explanation, only the parts related to the embodiment of the present invention are shown, and are described in detail below:

[0144] like Figure 8 As shown, the training device 3 for the document text detection model includes:

[0145] The image acquisition module 301 is used to acquire training set images, wherein the training set images include document images and position labels corresponding to text regions in the document images.

[0146] The feature extraction module 302 is configured to perform feature extraction on each of the document images in the training set images based on a preset encoder module to obtain first feature maps of different scales corresponding to the document images.

[0147] The feature fusion module 303 is configured to perform top-down and bottom-up feature fusion on the first feature maps of different scales based on a cascaded fusion attention module and a local-global fusion attention module to obtain second feature maps.

[0148] The text detection module 304 is configured to obtain a probability map and a threshold map based on the second feature maps, and perform differentiable binarization processing on the probability map and the threshold map to obtain a binary map and text region position coordinates of the document image predicted based on the binary map.

[0149] The model correction module 305 is configured to iteratively update parameters of the preset encoder module, the cascaded fusion attention module and the local-global fusion attention module based on the position label, the text region position coordinates and a preset loss function, and obtain a trained target document text detection model when the preset loss function converges.

[0150] The training device for the document text detection model provided in the embodiments of the present application can obtain accurate semantic information and spatial information of an image in the process of feature extraction on a training set image, thereby ensuring accurate positioning of an image text edge, and can effectively improve the accuracy of text detection based on top-down and bottom-up feature fusion on first feature maps. In the process of training the model, the parameters of the model are iteratively updated based on a loss function, thereby effectively improving the training accuracy of the model and further improving the detection accuracy of document text.

[0151] Figure 9 The structure diagram of the document text detection device provided in the embodiments of the present application is shown in the figure, only the parts related to the embodiments of the present application are shown, and the details are as follows:

[0152] As shown in the figure, the document text detection device 4 comprises: Figure 9

[0153] The document image acquisition module 401 is configured to acquire a document image to be detected.

[0154] The image preprocessing module 402 is configured to perform fuzzy denoising processing, grayscale processing and data enhancement processing on the document image to be detected to obtain a first document image to be detected.

[0155] ​The image text detection module 403 is configured to input the first to-be-detected document image into a target document text detection model, and output text region position coordinates corresponding to the first to-be-detected document image; wherein the target document text detection model is trained based on the training method of the document text detection model in the first aspect or any possible implementation manner of the first aspect.

[0156] The document text detection device provided by the embodiment of the present application can accurately detect the text region of a document image.

[0157] Figure 10 is a schematic diagram of an electronic device provided by the embodiment of the present application. As shown in Figure 10 , the electronic device 5 of the embodiment includes a processor 50, a memory 51, and a computer program 52 stored in the memory 51 and executable on the processor 50. The processor 50 implements the steps in the training method of the document text detection model / method for detecting a document text in the embodiment of the present application when executing the computer program 52, for example, the steps 101 to 105 shown in Figure 1 , or the steps 201 to 203 shown in Figure 7 . Alternatively, the processor 50 implements the functions of the modules / units in the various device embodiments when executing the computer program 52, for example, the functions of the modules 301 to 305 shown in Figure 8 , or the functions of the modules 401 to 403 shown in Figure 9 .

[0158] For example, the computer program 52 can be divided into one or more modules / units, which are stored in the memory 51 and executed by the processor 50 to complete the present application. The one or more modules / units can be a series of computer program instruction segments capable of completing a specific function, which are used to describe the execution process of the computer program 52 in the electronic device 5. For example, the computer program 52 can be divided into Figure 4 the modules / units 41 to 43 shown in

[0159] The electronic device 5 can be a desktop computer, a notebook computer, a palm computer, and a cloud server, etc. The electronic device 5 can include, but is not limited to, the processor 50 and the memory 51. Those skilled in the art can understand that Figure 5 the electronic device 5 is only an example and does not constitute a limitation on the electronic device 5, and can include more or fewer components than those shown, or combine certain components, or different components, for example, the electronic device can also include an input / output device, a network access device, a bus, etc.

[0160] The processor 50 can be a central processing unit (CPU), and can also be other general-purpose processors, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic device, discrete hardware component, etc. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor.

[0161] The memory 51 can be an internal storage unit of the electronic device 5, such as a hard disk or a memory of the electronic device 5. The memory 51 can also be an external storage device of the electronic device 5, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the electronic device 5. Further, the memory 51 can also include both the internal storage unit and the external storage device of the electronic device 5. The memory 51 is used to store the computer program and other programs and data required by the electronic device. The memory 51 can also be used to temporarily store data that has been output or will be output.

[0162] Those skilled in the art can clearly understand that, for the convenience and brevity of description, only the division of the above functional units and modules is exemplified, and in actual application, the above functions can be completed by different functional units and modules according to needs, that is, the internal structure of the apparatus is divided into different functional units or modules to complete all or part of the above described functions. Each functional unit and module in the embodiment can be integrated in one processing unit, or each unit can exist physically, or two or more units can be integrated in one unit, and the integrated unit can be realized in the form of hardware or in the form of software. In addition, the specific names of each functional unit and module are only for easy distinction, and do not limit the protection scope of the present application. The specific working process of the units and modules in the above system can refer to the corresponding process in the foregoing method embodiments, which will not be described here.

[0163] In the above embodiments, the description of each embodiment has its own emphasis, and the parts not described or recorded in detail in a certain embodiment can be referred to the relevant description of other embodiments.

[0164] Those skilled in the art can understand that the units and algorithm steps of each example described in combination with the embodiments disclosed herein can be realized by electronic hardware or a combination of computer software and electronic hardware. Whether the functions are realized in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0165] In the embodiments provided by the present application, it should be understood that the disclosed apparatus / equipment and method can be implemented in other ways. For example, the apparatus / equipment embodiments described above are merely schematic. The division of the modules or units is merely a logical function division, and there can be another division manner in actual implementation. For example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the displayed or discussed coupling or direct coupling or communication connection between the units can be indirect coupling or communication connection through some interface, device or unit, and can be electrical, mechanical or in other forms.

[0166] The units described as separate components can or can not be physically separate, and the components shown as units can or can not be physical units, i.e. they can be located in one place, or distributed on a plurality of network units. Part or all of the units can be selected according to actual needs to achieve the purpose of the embodiments.

[0167] In addition, each functional unit in each embodiment of the present application can be integrated into a processing unit, or each unit can exist physically, or two or more units can be integrated into one unit. The integrated unit can be realized in the form of hardware or in the form of a software functional unit.

[0168] The integrated module / unit, if implemented in the form of a software function unit and sold or used as an independent product, can be stored in a computer readable storage medium. Based on such understanding, all or part of the processes in the above-mentioned embodiment methods can also be completed by a computer program instructing related hardware, and the computer program can be stored in a computer readable storage medium. The computer program can implement the steps of the above-mentioned each invoice text detection model training method / invoice text detection method embodiment when executed by a processor. The computer program includes computer program code, which can be in the form of source code, object code, executable file, or some intermediate form. The computer readable medium can include any entity or device capable of carrying the computer program code, recording medium, U disk, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signal, telecommunication signal, and software distribution medium, etc. It should be noted that the computer readable medium can include or exclude contents according to the requirements of legislation and patent practice in the jurisdiction, for example, in some jurisdictions, according to legislation and patent practice, the computer readable medium does not include electrical carrier signals and telecommunication signals.

[0169] The above-described embodiments are only used to illustrate the technical solutions of the present application, rather than limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: it can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement for part of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application, and should be included in the protection scope of the present application.

Claims

1. A training method for a document text detection model, characterized in that, include: Obtain training set images, wherein the training set images include document images and location labels corresponding to text regions in the document images; For each document image in the training set, feature extraction is performed based on a preset encoder module to obtain a first feature map of different scales corresponding to the document image. Based on the cascaded fusion attention module and the local-global fusion attention module, feature fusion is performed on the first feature maps of different scales in a top-down and bottom-up manner to obtain the second feature map. This includes: performing global average pooling and max pooling on adjacent high-level and low-level feature maps in the first feature maps of different scales in a top-down order to obtain first intermediate feature maps of different scales; upsampling all the first intermediate feature maps to obtain corresponding second intermediate feature maps of the same scale; dividing adjacent low-level detail feature maps and high-level spatial feature maps in the second intermediate feature maps into a group in a bottom-up order, and performing global average pooling and detail feature preservation processing on each group of second intermediate feature maps to obtain a third intermediate feature map; and performing global average pooling and detail feature preservation processing on all the third intermediate feature maps to obtain the second feature map. Based on the second feature map, a probability map and a threshold map are obtained, and the probability map and the threshold map are subjected to differentiable binarization processing to obtain a binary map and the text region position coordinates of the document image predicted based on the binary map; Based on the location label, the text region location coordinates, and the preset loss function, the parameters of the preset encoder module, the cascaded fusion attention module, and the local-global fusion attention module are iteratively updated. When the preset loss function converges, the trained target document text detection model is obtained. The process involves performing global average pooling and max pooling on adjacent high-level and low-level feature maps in the first feature maps of different scales in a top-down order to obtain first intermediate feature maps of different scales, including: based on Global average pooling and max pooling are performed on adjacent high-level feature maps and low-level feature maps in the first feature maps of different scales to obtain the first intermediate feature maps of different scales. in, This represents the global features of adjacent high-level and low-level feature maps in the first feature maps at different scales. Indicates the first The first feature map, and representing relative to the first The low-level feature map of the first feature map, Indicates the first The first feature map, and representing relative to the first High-level feature maps of the first feature map This represents an element-by-element addition operation. This refers to the global average pooling process. This indicates that the convolution kernel size is 1×1, the input channel is C, and the output channel is [missing information]. Depth convolution operation, Represents the ReLU activation function. This indicates that the convolution kernel size is 1×1 and the input channels are... A depthwise convolution operation with output channel C. The first feature maps at different scales represent the local features of adjacent high-level and low-level feature maps. This indicates a max pooling operation. This indicates the average pooling operation. This indicates a splicing operation. This indicates a convolution operation with a kernel size of 7×7, 2 input channels, and 1 output channel. This represents the Sigmoid activation function. This represents the fusion weights of the low-level feature map and the high-level feature map. This indicates the element-wise multiplication operation. This represents the first intermediate feature map; The process involves grouping adjacent low-level detail feature maps and high-level spatial feature maps in the second intermediate feature map into a group, following a bottom-up order. Global average pooling and detail preservation processing are then applied to each group of second intermediate feature maps to obtain a third intermediate feature map, including: based on The adjacent low-level detail feature maps and high-level spatial feature maps in the second intermediate feature map are grouped together, and global average pooling and detail feature protection processing are performed on each group of second intermediate feature maps to obtain the third intermediate feature map. in, This represents the global features of the adjacent low-level detail feature maps and high-level spatial feature maps in the second intermediate feature map. Indicates the first The second intermediate feature map, and representing the second intermediate feature map relative to the first intermediate feature map. The low-level detail feature map of the second intermediate feature map, Indicates the first The second intermediate feature map, and representing the second intermediate feature map relative to the first intermediate feature map. The high-level spatial feature map of the second intermediate feature map, This refers to the global average pooling process. This indicates that the convolution kernel size is 1×1, the input channel is C, and the output channel is [missing information]. Depth convolution operation, Represents the ReLU activation function. This indicates that the convolution kernel size is 1×1 and the input channels are... A depthwise convolution operation with output channel C. This represents the detailed features of the low-level detail map. This represents the transposed detail feature map. This represents the third intermediate feature map.

2. A document text detection method, characterized in that, include: Acquire the image of the document to be inspected; The image of the document to be detected is subjected to blur denoising, grayscale processing and data augmentation to obtain the first image of the document to be detected; The first document image to be detected is input into the target document text detection model, and the coordinates of the text region corresponding to the first document image to be detected are output. The training method of the target document text detection model includes: acquiring training set images, wherein the training set images include document images and position labels corresponding to the text regions in the document images; for each document image in the training set images, feature extraction is performed based on a preset encoder module to obtain first feature maps of different scales corresponding to the document image; based on a cascaded fusion attention module and a local-global fusion attention module, top-down and bottom-up feature fusion is performed on the first feature maps of different scales to obtain second feature maps, including: performing global average pooling and max pooling on adjacent high-level and low-level feature maps in the first feature maps of different scales in a top-down order to obtain first intermediate feature maps of different scales; upsampling is performed on all the first intermediate feature maps. Similarly, a second intermediate feature map of the same scale is obtained; following a bottom-up order, adjacent low-level detail feature maps and high-level spatial feature maps in the second intermediate feature map are grouped together, and global average pooling and detail feature protection processing are performed on each group of second intermediate feature maps to obtain a third intermediate feature map; global average pooling and detail feature protection processing are performed on all the third intermediate feature maps to obtain the second feature map; a probability map and a threshold map are obtained based on the second feature map, and differentiable binarization processing is performed on the probability map and the threshold map to obtain a binary map and the text region position coordinates of the document image predicted based on the binary map; the parameters of the preset encoder module, the cascaded fusion attention module and the local-global fusion attention module are iteratively updated based on the position label, the text region position coordinates and the preset loss function; when the preset loss function converges, the trained target document text detection model is obtained. The process involves performing global average pooling and max pooling on adjacent high-level and low-level feature maps in the first feature maps of different scales in a top-down order to obtain first intermediate feature maps of different scales, including: based on Global average pooling and max pooling are performed on adjacent high-level feature maps and low-level feature maps in the first feature maps of different scales to obtain the first intermediate feature maps of different scales. in, This represents the global features of adjacent high-level and low-level feature maps in the first feature maps at different scales. Indicates the first The first feature map, and representing relative to the first The low-level feature map of the first feature map, Indicates the first The first feature map, and representing relative to the first High-level feature maps of the first feature map This represents an element-by-element addition operation. This refers to the global average pooling process. This indicates that the convolution kernel size is 1×1, the input channel is C, and the output channel is [missing information]. Depth convolution operation, Represents the ReLU activation function. This indicates that the convolution kernel size is 1×1 and the input channels are... A depthwise convolution operation with output channel C. The first feature maps at different scales represent the local features of adjacent high-level and low-level feature maps. This indicates a max pooling operation. This indicates the average pooling operation. This indicates a splicing operation. This indicates a convolution operation with a kernel size of 7×7, 2 input channels, and 1 output channel. This represents the Sigmoid activation function. This represents the fusion weights of the low-level feature map and the high-level feature map. This indicates the element-wise multiplication operation. This represents the first intermediate feature map; The process involves grouping adjacent low-level detail feature maps and high-level spatial feature maps in the second intermediate feature map into a group, following a bottom-up order. Global average pooling and detail preservation processing are then applied to each group of second intermediate feature maps to obtain a third intermediate feature map, including: based on The adjacent low-level detail feature maps and high-level spatial feature maps in the second intermediate feature map are grouped together, and global average pooling and detail feature protection processing are performed on each group of second intermediate feature maps to obtain the third intermediate feature map. in, This represents the global features of the adjacent low-level detail feature maps and high-level spatial feature maps in the second intermediate feature map. Indicates the first The second intermediate feature map, and representing the second intermediate feature map relative to the first intermediate feature map. The low-level detail feature map of the second intermediate feature map, Indicates the first The second intermediate feature map, and representing the second intermediate feature map relative to the first intermediate feature map. The high-level spatial feature map of the second intermediate feature map, This refers to the global average pooling process. This indicates that the convolution kernel size is 1×1, the input channel is C, and the output channel is [missing information]. Depth convolution operation, Represents the ReLU activation function. This indicates that the convolution kernel size is 1×1 and the input channels are... A depthwise convolution operation with output channel C. This represents the detailed features of the low-level detail map. This represents the transposed detail feature map. This represents the third intermediate feature map.

3. A training device for a document text detection model, characterized in that, include: The image acquisition module is used to acquire training set images, wherein the training set images include document images and location labels corresponding to text regions in the document images; The feature extraction module is used to extract features for each document image in the training set based on the preset encoder module to obtain a first feature map of different scales corresponding to the document image. The feature fusion module is used to perform top-down and bottom-up feature fusion on the first feature maps of different scales based on the cascaded fusion attention module and the local-global fusion attention module to obtain the second feature map. The module includes: performing global average pooling and max pooling on adjacent high-level and low-level feature maps of the first feature maps of different scales in a top-down order to obtain first intermediate feature maps of different scales; upsampling all the first intermediate feature maps to obtain corresponding second intermediate feature maps of the same scale; dividing adjacent low-level detail feature maps and high-level spatial feature maps of the second intermediate feature maps into a group in a bottom-up order, and performing global average pooling and detail feature preservation processing on each group of second intermediate feature maps to obtain a third intermediate feature map; and performing global average pooling and detail feature preservation processing on all the third intermediate feature maps to obtain the second feature map. The text detection module is used to obtain a probability map and a threshold map based on the second feature map, and to perform differentiable binarization processing on the probability map and the threshold map to obtain a binary map and the text region location coordinates of the document image predicted based on the binary map; The model correction module is used to iteratively update the parameters of the preset encoder module, the cascaded fusion attention module and the local-global fusion attention module based on the location label, the text region location coordinates and the preset loss function. When the preset loss function converges, the trained target document text detection model is obtained. The process involves performing global average pooling and max pooling on adjacent high-level and low-level feature maps in the first feature maps of different scales in a top-down order to obtain first intermediate feature maps of different scales, including: based on Global average pooling and max pooling are performed on adjacent high-level feature maps and low-level feature maps in the first feature maps of different scales to obtain the first intermediate feature maps of different scales. in, This represents the global features of adjacent high-level and low-level feature maps in the first feature maps at different scales. Indicates the first The first feature map, and representing relative to the first The low-level feature map of the first feature map, Indicates the first The first feature map, and representing relative to the first High-level feature maps of the first feature map This represents an element-by-element addition operation. This refers to the global average pooling process. This indicates that the convolution kernel size is 1×1, the input channel is C, and the output channel is [missing information]. Depth convolution operation, Represents the ReLU activation function. This indicates that the convolution kernel size is 1×1 and the input channels are... A depthwise convolution operation with output channel C. The first feature maps at different scales represent the local features of adjacent high-level and low-level feature maps. This indicates a max pooling operation. This indicates the average pooling operation. This indicates a splicing operation. This indicates a convolution operation with a kernel size of 7×7, 2 input channels, and 1 output channel. This represents the Sigmoid activation function. This represents the fusion weights of the low-level feature map and the high-level feature map. This indicates the element-wise multiplication operation. This represents the first intermediate feature map; The process involves grouping adjacent low-level detail feature maps and high-level spatial feature maps in the second intermediate feature map into a group, following a bottom-up order. Global average pooling and detail preservation processing are then applied to each group of second intermediate feature maps to obtain a third intermediate feature map, including: based on The adjacent low-level detail feature maps and high-level spatial feature maps in the second intermediate feature map are grouped together, and global average pooling and detail feature protection processing are performed on each group of second intermediate feature maps to obtain the third intermediate feature map. in, This represents the global features of the adjacent low-level detail feature maps and high-level spatial feature maps in the second intermediate feature map. Indicates the first The second intermediate feature map, and representing the second intermediate feature map relative to the first intermediate feature map. The low-level detail feature map of the second intermediate feature map, Indicates the first The second intermediate feature map, and representing the second intermediate feature map relative to the first intermediate feature map. The high-level spatial feature map of the second intermediate feature map, This refers to the global average pooling process. This indicates that the convolution kernel size is 1×1, the input channel is C, and the output channel is [missing information]. Depth convolution operation, Represents the ReLU activation function. This indicates that the convolution kernel size is 1×1 and the input channels are... A depthwise convolution operation with output channel C. This represents the detailed features of the low-level detail map. This represents the transposed detail feature map. This represents the third intermediate feature map.

4. A document text detection device, characterized in that, include: The document image acquisition module is used to acquire images of the documents to be inspected. The image preprocessing module is used to perform blur denoising, grayscale processing and data enhancement processing on the image of the document to be detected to obtain the first image of the document to be detected. An image text detection module is used to input the first document image to be detected into a target document text detection model and output the location coordinates of the text region corresponding to the first document image to be detected. The training method of the target document text detection model includes: acquiring training set images, wherein the training set images include document images and location labels corresponding to the text regions in the document images; for each document image in the training set images, feature extraction is performed based on a preset encoder module to obtain first feature maps of different scales corresponding to the document image; feature fusion is performed on the first feature maps of different scales using a top-down and bottom-up approach based on a cascaded fusion attention module and a local-global fusion attention module to obtain second feature maps, including: performing global average pooling and max pooling on adjacent high-level and low-level feature maps in the first feature maps of different scales in a top-down order to obtain first intermediate feature maps of different scales; and processing all the first intermediate feature maps... The image is upsampled to obtain a second intermediate feature map of the same scale. Following a bottom-up order, adjacent low-level detail feature maps and high-level spatial feature maps in the second intermediate feature map are grouped together, and global average pooling and detail feature preservation processing are performed on each group of second intermediate feature maps to obtain a third intermediate feature map. Global average pooling and detail feature preservation processing are performed on all the third intermediate feature maps to obtain the second feature map. A probability map and a threshold map are obtained based on the second feature map, and the probability map and the threshold map are subjected to differentiable binarization processing to obtain a binary map and the text region location coordinates of the document image predicted based on the binary map. The parameters of the preset encoder module, the cascaded fusion attention module, and the local-global fusion attention module are iteratively updated based on the location label, the text region location coordinates, and the preset loss function. When the preset loss function converges, the trained target document text detection model is obtained. The process involves performing global average pooling and max pooling on adjacent high-level and low-level feature maps in the first feature maps of different scales in a top-down order to obtain first intermediate feature maps of different scales, including: based on Global average pooling and max pooling are performed on adjacent high-level feature maps and low-level feature maps in the first feature maps of different scales to obtain the first intermediate feature maps of different scales. in, This represents the global features of adjacent high-level and low-level feature maps in the first feature maps at different scales. Indicates the first The first feature map, and representing relative to the first The low-level feature map of the first feature map, Indicates the first The first feature map, and representing relative to the first High-level feature maps of the first feature map This represents an element-by-element addition operation. This refers to the global average pooling process. This indicates that the convolution kernel size is 1×1, the input channel is C, and the output channel is [missing information]. Depth convolution operation, Represents the ReLU activation function. This indicates that the convolution kernel size is 1×1 and the input channels are... A depthwise convolution operation with output channel C. The first feature maps at different scales represent the local features of adjacent high-level and low-level feature maps. This indicates a max pooling operation. This indicates the average pooling operation. This indicates a splicing operation. This indicates a convolution operation with a kernel size of 7×7, 2 input channels, and 1 output channel. This represents the Sigmoid activation function. This represents the fusion weights of the low-level feature map and the high-level feature map. This indicates the element-wise multiplication operation. This represents the first intermediate feature map; The process involves grouping adjacent low-level detail feature maps and high-level spatial feature maps in the second intermediate feature map into a group, following a bottom-up order. Global average pooling and detail preservation processing are then applied to each group of second intermediate feature maps to obtain a third intermediate feature map, including: based on The adjacent low-level detail feature maps and high-level spatial feature maps in the second intermediate feature map are grouped together, and global average pooling and detail feature protection processing are performed on each group of second intermediate feature maps to obtain the third intermediate feature map. in, This represents the global features of the adjacent low-level detail feature maps and high-level spatial feature maps in the second intermediate feature map. Indicates the first The second intermediate feature map, and representing the second intermediate feature map relative to the first intermediate feature map. The low-level detail feature map of the second intermediate feature map, Indicates the first The second intermediate feature map, and representing the second intermediate feature map relative to the first intermediate feature map. The high-level spatial feature map of the second intermediate feature map, This refers to the global average pooling process. This indicates that the convolution kernel size is 1×1, the input channel is C, and the output channel is [missing information]. Depth convolution operation, Represents the ReLU activation function. This indicates that the convolution kernel size is 1×1 and the input channels are... A depthwise convolution operation with output channel C. This represents the detailed features of the low-level detail map. This represents the transposed detail feature map. This represents the third intermediate feature map.

5. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the method as described in claim 1 above.

6. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the method as described in claim 2 above.

7. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method as described in claim 1 or 2 above.

Images