Scanning file identification method and related device
By integrating feature fusion module, attention mechanism and boundary refinement module into the second-stage detection model, the problem of blurred boundaries between printed word images and handwritten images in the scanned file is solved, and higher recognition accuracy is achieved.
Patent Information
- Application Number
- CN202411681429.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-11-22
- Publication Date
- 2025-03-11
AI Technical Summary
The prior art is difficult to accurately segment the printed word image and handwritten image in the scanned file, resulting in blurred segmentation boundaries and affecting the recognition accuracy.
The detection model based on the two-stage detection model is adopted, which incorporates the feature fusion module, attention mechanism and boundary refinement module. The difference in multi-scale feature maps is reduced through the feature fusion module. The attention mechanism learns the spatial correlation between image pixels, and the boundary refinement module refines the boundary of the handwritten image area.
It realizes accurate segmentation of printed word images and handwritten images in the scanned file, solves the problem of blurred segmentation boundaries, and improves the recognition accuracy of scanned files.
Smart Images
Figure CN119600624A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of image processing technology, and in particular to a scanned document recognition method and related devices. Background Art
[0002] Banks process a large number of business scanned documents every day. The clarity, font size, and image shape of each scanned document are not exactly the same. At the same time, the text of each scanned document includes both printed and handwritten images. Currently, the printed image and handwritten image in each scanned document can be separated by a two-stage detection model, but there is often a problem of blurred segmentation boundaries. How to accurately separate the printed image and handwritten image in each scanned document has become a technical problem that needs to be solved urgently. Summary of the invention
[0003] Based on the above problems, the present application provides a scanned document recognition method for accurately segmenting the printed word images and handwritten word images in each scanned document, solving the problem of blurred segmentation boundaries in traditional methods and improving the segmentation accuracy of scanned documents.
[0004] The embodiments of the present application disclose the following technical solutions:
[0005] The first aspect of the present application provides a scanned document recognition method, comprising:
[0006] Acquire a target scan file; the target scan file includes a printed word image and a handwritten word image;
[0007] The target scan file is input into the detection model to obtain an instance segmentation result of the target scan file; the instance segmentation result shows that the target scan file is segmented into multiple regions, each of which corresponds to a printed word image or a handwritten word image; the detection model is based on a two-stage detection model, and is trained after integrating a feature fusion module, an attention mechanism and a boundary refinement module; the feature fusion module is used to reduce the difference between different scales when multi-scale feature maps are fused; the attention mechanism is used to learn the spatial correlation between image pixels in the target scan file; and the boundary refinement module is used to refine the regional boundary of the area where the handwritten word image is located.
[0008] In an optional implementation, using the feature fusion module to process a plurality of first feature maps corresponding to the target scan file includes:
[0009] Performing merging connection and multiple convolution operations on the multiple first feature maps to obtain a first intermediate feature map;
[0010] Performing an element-by-element operation on the first intermediate feature map and the multiple first feature maps to obtain a second feature map;
[0011] The second feature map is passed through a multi-layer perceptron and a softmax function to obtain a second intermediate feature map;
[0012] The second intermediate feature map is element-wise multiplied by the second feature map to obtain a third feature map.
[0013] In an optional implementation, using the boundary refinement module to process a plurality of fourth feature maps corresponding to the target scan file includes:
[0014] Using a plurality of dilated convolutional layers with different dilation rates to extract features of objects of different sizes in the plurality of fourth feature maps, to obtain a fifth feature map;
[0015] The fifth characteristic map is subjected to multiple convolution operations to obtain a sixth characteristic map.
[0016] In an optional implementation, the attention mechanism includes context modeling, feature conversion and feature aggregation, wherein:
[0017] The context modeling is used to extract global context information of the target scan file;
[0018] The feature conversion is used to capture the dependencies between channels and reduce the difficulty of network optimization;
[0019] The feature aggregation is used to perform element-by-element feature fusion between feature maps.
[0020] In an optional implementation, the training process of the detection model includes:
[0021] Using the target handwritten image and the two-stage detection network, the preliminary weight values of the two-stage network are obtained;
[0022] Constructing a model to be trained using the preliminary weight values, the two-stage detection model, the integrated feature fusion module, the attention mechanism and the boundary refinement module;
[0023] The model to be trained is trained based on the loss function and the training data set until the error is less than a preset error value, and the training is stopped to obtain the detection model.
[0024] A second aspect of the present application provides a scanned file recognition device, the scanned file recognition device comprising:
[0025] A file acquisition module, used to acquire a target scan file; the target scan file includes a printed word image and a handwritten word image;
[0026] The result acquisition module is used to input the target scan file into the detection model to obtain the instance segmentation result of the target scan file; the instance segmentation result shows that the target scan file is segmented into multiple areas, each of which corresponds to a printed word image or a handwritten word image; the detection model is based on a two-stage detection model, and is trained after integrating a feature fusion module, an attention mechanism and a boundary refinement module; the feature fusion module is used to reduce the difference between different scales when multi-scale feature maps are fused; the attention mechanism is used to learn the spatial correlation between image pixels in the target scan file; the boundary refinement module is used to refine the regional boundary of the area where the handwritten word image is located.
[0027] In an optional implementation, the scanned file recognition device includes:
[0028] Performing merging connection and multiple convolution operations on the multiple first feature maps to obtain a first intermediate feature map;
[0029] Performing an element-by-element operation on the first intermediate feature map and the multiple first feature maps to obtain a second feature map;
[0030] The second feature map is passed through a multi-layer perceptron and a softmax function to obtain a second intermediate feature map;
[0031] The second intermediate feature map is element-wise multiplied by the second feature map to obtain a third feature map.
[0032] In an optional implementation, the scanned file recognition device includes:
[0033] a fifth feature map acquisition unit, configured to extract features of objects of different sizes in the plurality of fourth feature maps using a plurality of dilated convolutional layers with different dilation rates to obtain a fifth feature map;
[0034] The sixth characteristic map acquisition unit is used to perform multiple convolution operations on the fifth characteristic map to obtain a sixth characteristic map.
[0035] A third aspect of the present application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method described in the first aspect.
[0036] A fourth aspect of the present application provides an electronic device, including:
[0037] a memory having a computer program stored thereon;
[0038] A processor is used to execute the computer program in the memory to implement the steps of the method described in the first aspect.
[0039] Compared with the prior art, this application has the following beneficial effects:
[0040] The present application discloses a scanned document recognition method, comprising: obtaining a target scanned document including a printed word image and a handwritten word image; inputting the target scanned document into a detection model to obtain an instance segmentation result of the target scanned document. The instance segmentation result shows that the target scanned document is segmented into multiple regions, each region corresponding to a printed word image or a handwritten word image.
[0041] Since the feature fusion module can reduce the difference between different scales when multi-scale feature maps are fused, the attention mechanism can fully learn the spatial correlation between image pixels in the target scan file, and the boundary refinement module can refine the regional boundaries of the area where the handwritten image is located, the two-stage detection model can identify the content of the target scan file and segment the target scan file; therefore, based on the two-stage detection model, after integrating the feature fusion module, the attention mechanism and the boundary refinement module, the detection model obtained after training can identify the content of each scan file, accurately separate the printed image and the handwritten image in each scan file, solve the problem of blurred segmentation boundaries in the existing method, and improve the recognition accuracy of scanned files. BRIEF DESCRIPTION OF THE DRAWINGS
[0042] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.
[0043] Figure 1 A flowchart of a scanned file recognition method provided in an embodiment of the present application;
[0044] Figure 2 A schematic diagram of the structure of a detection model provided in an embodiment of the present application;
[0045] Figure 3 A schematic diagram of a feature fusion module provided in an embodiment of the present application processing a feature map of a target scan file;
[0046] Figure 4 A schematic diagram of a boundary refinement module processing a feature map of a target scan file is provided for an embodiment of the present application;
[0047] Figure 5 A schematic diagram of the structure of a scanning document recognition device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0048] As mentioned above, banks process a large number of scanned documents every day. The clarity, font size, and image shape of each scanned document are not exactly the same. At the same time, the text of each scanned document includes both printed word images and handwritten word images. Currently, the printed word images and handwritten word images in each scanned document can be separated by a two-stage detection model, but there is often a problem of blurred segmentation boundaries. How to accurately separate the printed word images and handwritten word images in each scanned document has become a technical problem that needs to be solved urgently.
[0049] The present application discloses a scanned document recognition method, comprising: obtaining a target scanned document including a printed word image and a handwritten word image; inputting the target scanned document into a detection model to obtain an instance segmentation result of the target scanned document. The instance segmentation result shows that the target scanned document is segmented into multiple regions, each region corresponding to a printed word image or a handwritten word image.
[0050] Since the feature fusion module can reduce the difference between different scales when multi-scale feature maps are fused, the attention mechanism can fully learn the spatial correlation between image pixels in the target scan file, and the boundary refinement module can refine the regional boundaries of the area where the handwritten image is located, the two-stage detection model can identify the content of the target scan file and segment the target scan file; therefore, based on the two-stage detection model, after integrating the feature fusion module, the attention mechanism and the boundary refinement module, the detection model obtained after training can accurately segment the printed image and the handwritten image in each scanned file, solve the problem of blurred segmentation boundaries in the existing method, and improve the segmentation accuracy of the scanned file.
[0051] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without creative work are within the scope of protection of the present application.
[0052] Figure 1 A flowchart of a scanned file recognition method provided in an embodiment of the present application. Figure 1 As shown, the scanned file recognition method in this application includes:
[0053] Step 101, obtaining a target scan file.
[0054] Taking the business within the banking system as an example, when a customer goes to the bank to handle business, the bank will provide the corresponding file for the business that the customer needs to handle. The file includes the printed standard text content, as well as the name, business handling time, contact information and other related content that require the customer's handwritten signature. When the customer's business is completed, in order to facilitate archiving and business analysis, the bank will scan the document that the customer has signed and confirmed to generate a scanned file. The scanned file includes both printed text images, such as the printed standard text content in the file, and handwritten text images, such as the customer's handwritten name, business handling time and contact information.
[0055] The target scanned file in this application is a scanned file including printed word images and handwritten word images. It is understandable that due to the different shapes and sizes of handwritten words and the different scanning machines used, the scanned files often have many problems such as inconsistent word sizes and irregular image shapes.
[0056] Step 102: input the target scan file into a detection model to obtain an instance segmentation result of the target scan file.
[0057] The detection model in this application is based on a two-stage detection model, which is trained after integrating a feature fusion module, an attention mechanism, and a boundary refinement module. Among them, the two-stage detection model can identify the content of the target scan file and segment the target scan file; the feature fusion module is used to reduce the difference between different scales when multi-scale feature maps are fused; the attention mechanism is used to learn the spatial correlation between image pixels in the target scan file; and the boundary refinement module is used to refine the regional boundaries of the area where the handwritten image is located.
[0058] The example segmentation results in this application show that the target scan file is segmented into multiple regions, each region corresponding to a printed word image or a handwritten word image. For example, assuming that there are three parts of printed word images in the target scan file A, namely image 1, image 2 and image 3; two parts of handwritten word images, namely image 4 and image 5, then after the target scan file A is input into the detection model, the detection model recognizes and analyzes the content in the target scan file A, and finally the target scan file A can be segmented into 5 regions, the first region corresponding to image 1, the second region corresponding to image 2, the third region corresponding to image 3, the fourth region corresponding to image 4 and the fifth region corresponding to image 5.
[0059] It should be noted that the detection model in this application is built based on the Mask R-CNN framework in the two-stage detection model, which is an improved model of the two-stage detection model. Mask R-CNN is an improved model based on the regional convolutional neural network (R-CNN), which can realize target detection and example segmentation.
[0060] Figure 2 A schematic diagram of a detection model provided in an embodiment of the present application. Figure 2 As shown, Figure 2 The middle (a) area shows the structure of the improved Mask R-CNN (detection model), which includes a feature extraction network, a FPN network (Feature Pyramid Networks), an RPN network (Region Proposal Network), a ROI Align layer, and a three-branch structure. Among them, the feature extraction network is the starting part of the detection model, and the VGG16 network or ResNet-50 network can be used as the feature extraction network. The feature map extracted by the feature extraction network is input into the FPN network, and the FPN network performs feature fusion on the feature map input into the network, and the fused feature map is input into the RPN network. The RPN is used to generate a series of anchor boxes on the input image; the feature maps output by the FPN network and the RPN network are respectively input into the ROI Align layer for size scaling to obtain feature maps of different sizes; and then the feature maps of different sizes are input into the three-branch structure. Figure 2 The middle area (b) shows the specific structure of the three branches, namely the structure of the category score branch, the target bounding box branch, and the mask prediction branch.
[0061] Specifically, after the feature map output from the ROI Align layer enters the category score branch and the target bounding box branch in the three-branch structure, it passes through the convolution layer and multiple fully connected layers to obtain the category score and target bounding box of the target scan file.
[0062] The detection model in this application is a model obtained by training after adding a feature fusion module, an attention mechanism, and a boundary refinement module on the basis of Mask R-CNN. The feature fusion module is located before the feature lateral connection in the FPN network; the attention mechanism is located after the fully connected layer in the mask prediction branch; the boundary refinement module is used as a supplement to the mask prediction branch. The details of the feature fusion module, the attention mechanism, and the boundary refinement module will be introduced in detail in the subsequent embodiments of this application.
[0063] It should be noted that since Mask R-CNN is a two-stage detection model that is relatively well known to those skilled in the art, this application does not provide much introduction to the basic Mask R-CNN.
[0064] The feature pyramid network (FPN network) used by the traditional Mask R-CNN can solve the scale change problem that often exists in target detection, but the network simply connects the feature maps of the same level horizontally, resulting in inconsistency between feature maps of different sizes. This application adds a feature fusion module before the horizontal connection of features in the FPN network to solve the inconsistency caused by multi-scale detection features and reduce the differences between multi-scale features.
[0065] Figure 3 A schematic diagram of a feature fusion module provided in an embodiment of the present application processing a feature map of a target scan file. Figure 3 As shown in the figure, the process is specifically as follows: first, multiple first feature maps of the target scanned file are obtained, and the multiple first feature maps are merged and connected to obtain a connected feature map; a conventional convolution operation is performed on the connected feature map to obtain a convolved feature map (a first intermediate feature map); the convolved feature map is element-wise added and multiplied with the multiple first feature maps to obtain a second feature map; the second feature map is input into a multi-layer perceptron (Multi-LayerPerception, MLP) and a softmax function to obtain a second intermediate feature map; the second intermediate feature map is element-wise multiplied with the second feature map to obtain a third feature map.
[0066] The localization of the convolution operation will make the Convolutional Neural Networks (CNN) have an inherently narrow receptive field, which limits the CNN's global understanding ability and the inability to consider the relationship between global pixels, which will cause information loss and make the boundary segmentation of the target scanned file more error-prone.
[0067] To solve the above problems, this application adds an attention mechanism to learn the spatial correlation between image pixels in the target scan file and improve the accuracy of target scan file segmentation. The attention mechanism is a way to achieve network adaptive attention. The use of the attention mechanism can effectively focus on the key information of the current task, which can improve the efficiency and accuracy of task processing. Specifically, the attention mechanism used in this application is the Global Context Block (GC Block).
[0068] GC Block fully combines the context modeling capability of the non-local block (NL Block) and the computational savings of the Sequeze and Excitation Block (SE Block). GCBlock consists of three parts: a global average pooling module for context modeling, a bottleneck conversion that captures inter-channel dependencies, and element-by-element feature fusion. Formula (1) is the formula expression of GC Block:
[0069]
[0070] Among them, w v2 ×ReLU(LN(w v1 ×(·))) represents the bottleneck transformation; ReLU (Rectified Linear Unit) is the activation function. Before the bottleneck transformation and before the ReLU activation, a normalization layer is added to reduce the difficulty of optimization and serve as a regularizer that is conducive to generalization; Represents the global attention pooling weight.
[0071] Specifically, x in formula (1) i 、x j and x n Both represent the feature graphs of the input target scan files; N p is the total number of feature maps of the target scan file; z i is the output feature map; where i is the index of the query position, i∈(1, N p ); j lists all possible query locations, j∈(1, N p ), n is any feature graph that can be queried, n∈(1, N p );w v2 and w v1 represents the linear transformation matrix; w k represents the weight of the corresponding feature. It should be noted that in the initial training, w k Assign an arbitrary initial value and gradually update w in the reverse direction during the training process k The numerical value of .
[0072] When traditional methods segment the target scanned files, mask prediction relies on a fully convolutional neural network to perform pixel-level classification, and all pixels are processed in the same way, ignoring the boundary shape and features of the object to be processed. Therefore, when traditional methods segment the printed image and handwritten image in the target scanned files, there is often a problem of blurred segmentation boundaries.
[0073] To address the above issues, this application proposes a boundary refinement module for handwritten edge perception, which can integrate boundary information into the mask prediction branch and ultimately obtain a more accurate mask segmentation result.
[0074] Figure 4 A schematic diagram of a boundary refinement module processing a feature map of a target scan file is provided for an embodiment of the present application. Figure 4 The ROI (Region of Interest) in the figure represents the area to be processed; Box 1 shows the structure of the mask prediction branch; Box 2 shows the structure of the boundary refinement module. Figure 4 As shown, the process of processing the feature map of the target scan file by the boundary refinement module in the present application is specifically as follows: first, multiple dilated convolutional layers with different dilation rates are used ( Figure 4 The atrous convolution module in the image processing module (in the image processing module) extracts features of targets of different sizes in the ROI, that is, extracts features of targets of different sizes in multiple fourth feature maps to obtain the fifth feature map, and performs multiple convolution operations on the fifth feature map to obtain the sixth feature map. Specifically, the atrous convolution layer with a hole rate of 6, 12, and 18 can be used to extract features of targets of different scales in the fourth feature map to obtain the fifth feature map; the fifth feature map is input into the 1×1 convolution layer for channel compression, and then processed by two consecutive 3×3 convolution layers to obtain the sixth feature map; the sixth feature map is passed into the mask prediction branch and spliced with the current result of the mask prediction branch along the channel as a supplement to the mask prediction branch. Then, the boundary refinement part is subjected to a 3×3 convolution and a deconvolution, as well as pixel-level classification of the corresponding category, and the boundary is extracted using the Sobel operator, the error is calculated using the boundary loss function, and the weight of the boundary branch is updated during the gradient backpropagation process.
[0075] It should be noted that Figure 1 The structure of the mask prediction branch shown in the middle frame 1 includes multiple fully connected layers and multiple attention convolution layers; the attention convolution layer uses the GC Block mentioned in the previous embodiment of the present application; for the relevant content of GCBlock, please refer to the introduction of the previous embodiment and will not be repeated here.
[0076] In an optional implementation, the detection model in the present application is based on a two-stage detection model, and after integrating a feature fusion module, an attention mechanism, and a boundary refinement module, the model is obtained through training. Specifically, the target handwritten image and the two-stage detection network are used to obtain the preliminary weight values of the two-stage network; the model to be trained is constructed using the preliminary weight values, the two-stage detection model, the feature fusion module, the attention mechanism, and the boundary refinement module; the model to be trained is trained based on the loss function and the training data set until the error is less than the preset error value, and the training is stopped to obtain the detection model.
[0077] Specifically, the target handwritten image is input, and the two-stage detection network will capture the region of interest of the target handwritten image and extract the features of the target handwritten image. The weights of the two-stage detection network will be iteratively updated, reasonable hyperparameters will be set to avoid overfitting, and the baseline network with excellent performance will be used to obtain preliminary weight values.
[0078] Then, the network to be trained is constructed using the preliminary weight values and the extreme network with excellent performance obtained in the previous steps. At the same time, the feature fusion module is integrated before the lateral connection of the features in the FPN network; the attention mechanism is integrated after the fully connected layer in the mask prediction branch; the boundary refinement module is used as a supplement to the mask prediction branch; and finally the network to be trained is constructed.
[0079] Finally, the model to be trained is trained based on the loss function and the training data set until the error is less than the preset error value, and the training is stopped to obtain the detection model. The loss function in this application includes 4 parts, namely, the handwritten character classification loss function, the detection box regression loss function, the segmentation mask loss function and the boundary loss function. In this application, the cross entropy loss function is used as the handwritten character classification loss function; the binary cross entropy loss function is used as the segmentation mask loss function; formula (2) is used as the boundary loss function; and formula (3) is used as the detection box regression loss function.
[0080] Specifically, formula (2) is expressed as:
[0081] L Boundary =μ1×L dice +μ2×L BMSE (2)
[0082] Among them, L Boundary represents the boundary loss function; L dice represents the regional loss function (Dice-loss); L BMSE represents the boundary mean-square error loss function (BMSE); μ1 and μ2 represent weight coefficients; L BMSEIt cannot be used alone because it cannot distinguish between data inside and outside the area to be segmented; when the boundary mean square error loss function is used in combination with the regional loss function, the regional loss function can guide the initial training and avoid falling into large errors caused by unstable boundary loss. The Sobel operator edge detection filter is used to extract edges during training. The boundary loss function in this application is easy to implement and does not require any pre-processing or post-processing, and explicitly focuses on the boundary area during training.
[0083] Specifically, the expression of formula (3) is:
[0084]
[0085] in,
[0086]
[0087] Among them, t x =(xx a ) / w a ;t y =(yy a ) / h a ;t w =log(w / w a );;t h =log(h / h a ); x, y, w, and h represent the coordinate values of the prediction box boundary and its length and width respectively; x a ,y a 、w a and h a is the position information corresponding to the anchor box; x * ,y * 、w * and h * Represents the location information corresponding to the real frame.
[0088] In this application, a multi-task training method can be used to combine the handwriting classification loss function L class , detection regression loss function L box , segmentation mask loss function L mask And the boundary loss function L Boundary Since the boundary and mask are fused and cross-connected, joint training can increase the feature representation of boundary and mask prediction. The overall multi-task loss function is shown in formula (5):
[0089] L=L class +L box +L mask +L Boundary (5)
[0090] Wherein, L represents the multi-task loss function; the meanings of the remaining letters refer to the above embodiments. Since the cross entropy loss function and the binary cross entropy loss function are relatively mature loss functions, training the model to be trained based on the loss function and the data set is a relatively mature technology, so the above content is not repeated in this application.
[0091] Based on the scanned document recognition method provided in the aforementioned embodiment, the present application also provides a scanned document recognition device. Figure 5 A schematic diagram of the structure of a scanning document recognition device provided in an embodiment of the present application. Figure 5 As shown, the scanning document recognition device 500 of the present application includes:
[0092] The file acquisition module 501 is used to acquire a target scan file; the target scan file includes a printed word image and a handwritten word image;
[0093] The result acquisition module 502 is used to input the target scan file into the detection model to obtain the instance segmentation result of the target scan file; the instance segmentation result shows that the target scan file is segmented into multiple regions, each of which corresponds to a printed word image or a handwritten word image; the detection model is based on a two-stage detection model, and is trained after integrating a feature fusion module, an attention mechanism and a boundary refinement module; the feature fusion module is used to reduce the difference between different scales when fusing multi-scale feature maps; the attention mechanism is used to learn the spatial correlation between image pixels in the target scan file; the boundary refinement module is used to refine the regional boundary of the area where the handwritten word image is located.
[0094] In an optional implementation, the result acquisition module 502 includes:
[0095] A first intermediate feature map acquisition unit, configured to perform a merge connection and multiple convolution operations on the multiple first feature maps to obtain a first intermediate feature map;
[0096] A second feature map acquisition unit, configured to perform an element-by-element operation on the first intermediate feature map and the plurality of first feature maps to obtain a second feature map;
[0097] A second intermediate feature map acquisition unit, configured to obtain a second intermediate feature map by passing the second feature map through a multi-layer perceptron and a softmax function;
[0098] The third feature map acquisition unit is used to perform element-by-element multiplication of the second intermediate feature map and the second feature map to obtain a third feature map.
[0099] In an optional implementation, the result acquisition module 502 includes:
[0100] a fifth feature map acquisition unit, configured to extract features of objects of different sizes in the plurality of fourth feature maps using a plurality of dilated convolutional layers with different dilation rates to obtain a fifth feature map;
[0101] The sixth characteristic map acquisition unit is used to perform multiple convolution operations on the fifth characteristic map to obtain a sixth characteristic map.
[0102] In an optional implementation, the result acquisition module 502 includes:
[0103] A first unit, used for the context modeling, for extracting global context information of the target scan file;
[0104] The second unit, the feature conversion, is used to capture the dependency between channels and reduce the difficulty of network optimization;
[0105] The third unit, the feature aggregation, is used to perform element-by-element feature fusion between feature maps.
[0106] In an optional implementation, the scanning file recognition device 500 of the present application further includes:
[0107] A first training unit is used to obtain preliminary weight values of the two-stage network using the target handwritten image and the two-stage detection network;
[0108] A second training unit is used to construct a model to be trained by using the preliminary weight value, the two-stage detection model, the integrated feature fusion module, the attention mechanism and the boundary refinement module;
[0109] The third training unit is used to train the model to be trained based on the loss function and the training data set until the error is less than a preset error value, and stop training to obtain the detection model.
[0110] Based on the scanned file recognition method and device provided in the aforementioned embodiments, the present application also provides a computer-readable storage medium, on which a computer program is stored, which, when executed by a processor, implements some or all of the steps in the scanned file recognition method mentioned above.
[0111] Based on the scanned file recognition method and device provided in the above embodiments, the present application further provides an electronic device, including:
[0112] a memory having a computer program stored thereon;
[0113] The processor is used to execute the computer program in the memory to implement part or all of the steps in the scanned document recognition method provided in the above embodiment.
[0114] It should be noted that each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment. The device embodiment described above is merely schematic, in which the unit described as a separate component may or may not be physically separated, and the component prompted as a unit may or may not be a physical unit, that is, it may be located in one place, or it may be distributed on multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative work.
[0115] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed in the present application should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.
Claims
1. A scanned document recognition method, characterized in that: The method comprises: Acquire a target scan file; the target scan file includes a printed word image and a handwritten word image; The target scan file is input into the detection model to obtain an instance segmentation result of the target scan file; the instance segmentation result shows that the target scan file is segmented into multiple regions, each of which corresponds to a printed word image or a handwritten word image; the detection model is based on a two-stage detection model, and is trained after integrating a feature fusion module, an attention mechanism and a boundary refinement module; the feature fusion module is used to reduce the difference between different scales when multi-scale feature maps are fused; the attention mechanism is used to learn the spatial correlation between image pixels in the target scan file; and the boundary refinement module is used to refine the regional boundary of the area where the handwritten word image is located.
2. The method according to claim 1, characterized in that Using the feature fusion module to process a plurality of first feature maps corresponding to the target scan file includes: Performing merging connection and multiple convolution operations on the multiple first feature maps to obtain a first intermediate feature map; Performing an element-by-element operation on the first intermediate feature map and the multiple first feature maps to obtain a second feature map; The second feature map is passed through a multi-layer perceptron and a softmax function to obtain a second intermediate feature map; The second intermediate feature map is element-wise multiplied by the second feature map to obtain a third feature map.
3. The method according to claim 1, characterized in that Processing a plurality of fourth feature maps corresponding to the target scan file using the boundary refinement module includes: Using a plurality of dilated convolutional layers with different dilation rates to extract features of objects of different sizes in the plurality of fourth feature maps, to obtain a fifth feature map; The fifth characteristic map is subjected to multiple convolution operations to obtain a sixth characteristic map.
4. The method according to claim 1, characterized in that: The attention mechanism includes context modeling, feature conversion and feature aggregation, where: The context modeling is used to extract global context information of the target scan file; The feature conversion is used to capture the dependencies between channels and reduce the difficulty of network optimization; The feature aggregation is used to perform element-by-element feature fusion between feature maps.
5. The method according to claim 1, characterized in that: The training process of the detection model includes: Using the target handwritten image and the two-stage detection network, obtaining preliminary weight values of the two-stage network; Constructing a model to be trained by using the preliminary weight values, the two-stage detection model, the feature fusion module, the attention mechanism and the boundary refinement module; The model to be trained is trained based on the loss function and the training data set until the error is less than a preset error value, and the training is stopped to obtain the detection model.
6. A scanning document recognition device, characterized in that: The device comprises: A file acquisition module, used to acquire a target scan file; the target scan file includes a printed word image and a handwritten word image; The result acquisition module is used to input the target scan file into the detection model to obtain the instance segmentation result of the target scan file; the instance segmentation result shows that the target scan file is segmented into multiple areas, each of which corresponds to a printed word image or a handwritten word image; the detection model is based on a two-stage detection model, and is trained after integrating a feature fusion module, an attention mechanism and a boundary refinement module; the feature fusion module is used to reduce the difference between different scales when multi-scale feature maps are fused; the attention mechanism is used to learn the spatial correlation between image pixels in the target scan file; the boundary refinement module is used to refine the regional boundary of the area where the handwritten word image is located.
7. The device according to claim 6, characterized in that The result acquisition module comprises: A first intermediate feature map acquisition unit, configured to perform a merge connection and multiple convolution operations on the multiple first feature maps to obtain a first intermediate feature map; A second feature map acquisition unit, configured to perform an element-by-element operation on the first intermediate feature map and the plurality of first feature maps to obtain a second feature map; A second intermediate feature map acquisition unit, configured to obtain a second intermediate feature map by passing the second feature map through a multi-layer perceptron and a softmax function; The third feature map acquisition unit is used to perform element-by-element multiplication of the second intermediate feature map and the second feature map to obtain a third feature map.
8. The device according to claim 6, characterized in that The result acquisition module comprises: a fifth feature map acquisition unit, configured to extract features of objects of different sizes in the plurality of fourth feature maps using a plurality of dilated convolutional layers with different dilation rates to obtain a fifth feature map; The sixth characteristic map acquisition unit is used to perform multiple convolution operations on the fifth characteristic map to obtain a sixth characteristic map.
9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps of the method described in any one of claims 1 to 5 are implemented.
10. An electronic device, characterized in that: include: a memory having a computer program stored thereon; A processor, configured to execute the computer program in the memory to implement the steps of the method according to any one of claims 1 to 5.