A method, system, device and medium for restoring deformation of key information of 3D document

Abstract

The present invention relates to the field of image correction and restoration, and specifically to a method, system, device, and medium for restoring deformation of key information of a 3D document. The method extracts shallow features from a distorted document image, performs a first feature enhancement, uses a plurality of inverted residual blocks to extract deep features, then performs a second feature enhancement on the obtained deep feature map, performs multi-layer dilation convolution processing on the feature map after the second feature enhancement, and obtains fused features. Based on the fused features, the predicted intervals between reference points and the predicted coordinate positions of control points are output respectively to obtain a position mapping relationship between each control point and its corresponding reference point, further obtains a position corresponding relationship between all pixel points and reference points in a corrected image, moves each pixel point to the position of its corresponding reference point in the corrected image, and obtains a document image after deformation restoration, thereby realizing efficient deformation restoration of the deformation of key information of the 3D document.

Description

Technical Field

[0001] The present invention relates to the technical field of image correction and restoration, and in particular to a method, system, device and medium for restoring deformation of key information of a 3D document. Background Art

[0002] Paper documents are the core medium for recording and disseminating information culture. However, paper documents are easily damaged during the storage and transmission process, resulting in 3D deformation problems such as wrinkles and bends, which distort the recorded images, thus affecting the recognition of key information in the documents and causing analysis and processing obstacles. Therefore, the deformation recovery of key information in paper documents is very important, which not only facilitates the digitization of documents, but also facilitates the safe and efficient storage, retrieval and sharing of information, and has great practical significance.

[0003] At present, the methods for restoring paper document image deformation are mainly divided into traditional methods and neural network methods based on deep learning. Traditional methods are mainly divided into two categories: the first category mainly uses specific hardware equipment to collect three-dimensional data of document deformation, performs three-dimensional reconstruction, and performs a series of planarization processing on the reconstructed three-dimensional surface to further restore the deformed document image; the second category mainly explores the two-dimensional image feature information contained in the document image itself, and relies on this information to build a parameterized document deformation three-dimensional model to repair the deformed document image. Although the traditional method is accurate, it is expensive and inconvenient to collect data. It has limited effect in complex scenes and is easily affected by missed characters and the complexity of text line extraction.

[0004] In contrast, neural network methods based on deep learning can overcome the limitations of traditional methods, effectively improve the accuracy and efficiency of deformation recovery, and are also applicable to complex and changeable scenarios in practical applications. The current mainstream models use an end-to-end framework for pixel-level prediction, such as using stacked UNet, FCN, Transformer and other networks. Although these methods can correct images within a certain range, they are computationally intensive and generally perform poorly in retaining the edges of key information such as barcodes and QR codes. Summary of the invention

[0005] The invention provides a method for restoring deformation of key information of a 3D document.

[0006] The technical solution of the present invention is as follows:

[0007] A method for restoring deformation of key information of a 3D document comprises the following steps:

[0008] S1, performing multi-layer convolution processing, batch normalization and nonlinear processing on the document image to be processed in sequence, extracting shallow features of the document image, performing shallow feature mapping on the shallow features to obtain a channel feature map, wherein each shallow feature corresponds to a channel;

[0009] S2, performing the first channel attention mechanism processing and the first spatial attention mechanism processing on the channel feature map in sequence to obtain a feature map after the first feature is enhanced;

[0010] Extracting deep features from the feature map after the first feature enhancement using a plurality of inverted residual blocks to obtain a deep feature map, wherein each inverted residual block includes, in a processing order, an extended convolution layer, a depth-separable convolution layer, and a compressed convolution layer;

[0011] S3, performing the second channel attention mechanism processing and the second spatial attention mechanism processing on the depth feature map in sequence to obtain a feature map after the second feature is enhanced;

[0012] The feature map after the second feature enhancement is sequentially subjected to a plurality of dilated convolution layers with specific dilation rates for multi-layer dilated convolution processing, wherein the multi-layer dilated convolution processing is performed based on the dilation rate set in each dilated convolution layer and the weight of the convolution kernel at a specific position of the feature map after the second feature enhancement, to obtain a plurality of dilated convolution features, and the plurality of dilated convolution features are subjected to feature fusion processing to obtain a fused feature;

[0013] S4, input the fusion features into the fully connected layer and the convolution layer respectively, obtain the predicted spatial information distribution after the fully connected layer is processed, and obtain the correction template based on the spatial information distribution; obtain the key feature points at specific positions after the convolution layer is processed; based on the coordinate positions of the correction template and the key feature points, obtain the position mapping relationship between the correction template and the key feature points,

[0014] Based on the position mapping relationship, each pixel in the document image to be processed is interpolated to obtain the position correspondence between all pixels and their corresponding mapping coordinates in the correction template.

[0015] Based on the position correspondence, each pixel is moved to its corresponding mapping coordinate position in the correction template to obtain the document image after deformation restoration.

[0016] Specifically, the loss function used in S4 is:

[0017] Based on the number of key feature points and the distance between the key feature points and their corresponding mapped coordinate positions in the calibration template, the positioning loss is obtained;

[0018] Based on the number of key feature points and the distance between key feature points and their neighbor key feature points, the neighborhood relationship loss is obtained;

[0019] Based on the number of key feature points, the interval of key feature points, and the interval of key feature points in the correction template, the interval loss regression loss is obtained;

[0020] Based on the number of key feature points, the gradient direction of the key feature points, and the gradient direction of the key feature points corresponding to the mapping coordinate position in the correction template, the gradient direction loss of the key feature points is obtained;

[0021] The positioning loss, neighborhood relationship loss, interval loss regression loss and key feature point gradient direction loss are linearly weighted to obtain the loss function.

[0022] Furthermore, each inverted residual block in S2 includes, in processing order, an extended convolution layer, a depthwise separable convolution layer, and a compressed convolution layer, wherein:

[0023] The extended convolution layer uses a 1×1 convolution kernel to expand the number of channels of the feature map after the first feature enhancement to obtain a feature map after channel expansion;

[0024] The feature map after channel expansion is processed by depth-separable convolution layer using 3×3 convolution kernel to perform depth convolution on each channel, extract the depth features, and obtain the feature map after depth convolution;

[0025] The feature map after the deep convolution is compressed using a 1×1 convolution kernel in the compressed convolution layer to output the final deep feature map.

[0026] Preferably, the step of sequentially performing multi-layer convolution processing on the document image to be processed in S1 specifically includes using two 3×3 convolution layers with a step size of 2 to perform convolution processing on the distorted document image.

[0027] Specifically, in S2, the channel feature map is processed by the first channel attention mechanism and the first spatial attention mechanism in sequence to obtain a feature map after the first feature is enhanced, specifically:

[0028] The channel feature map is pooled, transformed and activated to obtain the attention weight of each channel. The attention weight is multiplied by the channel feature map to obtain the feature map after the channel attention mechanism is applied.

[0029] The feature map processed by the channel attention mechanism is transformed into a spatial feature to obtain a spatial attention map. The spatial attention map and the feature map processed by the channel attention mechanism are element-wise multiplied to obtain a feature map after the first feature is enhanced.

[0030] Furthermore, in S3, the depth feature map is processed by the second channel attention mechanism and the second spatial attention mechanism in sequence to obtain a feature map after the second feature enhancement, specifically:

[0031] The deep feature map is pooled, transformed, and activated to obtain the attention weight of each channel. The attention weight is multiplied by the deep feature map by channel to obtain the feature map processed by the channel attention mechanism.

[0032] The feature map processed by the channel attention mechanism is transformed into a spatial feature to obtain a spatial attention map. The spatial attention map and the feature map processed by the channel attention mechanism are element-wise multiplied to obtain a feature map after the second feature is enhanced.

[0033] Specifically, the key feature points at the specific positions in S4 include: four corner points of the distorted document image, four corner points and edge position points of the optical mark, barcode and two-dimensional code in the distorted document image.

[0034] The present invention also provides a 3D document key information deformation recovery system, comprising:

[0035] Shallow feature extraction module: used to perform multi-layer convolution processing, batch normalization and nonlinear processing on the document image to be processed in sequence, extract shallow features of the document image, perform shallow feature mapping on it, and obtain a channel feature map, wherein each shallow feature corresponds to a channel;

[0036] Deep feature extraction module: used to perform first channel attention mechanism processing and first spatial attention mechanism processing on the channel feature map in sequence to obtain a feature map after the first feature is enhanced;

[0037] Extracting deep features from the feature map after the first feature enhancement using a plurality of inverted residual blocks to obtain a deep feature map, wherein each inverted residual block includes, in a processing order, an extended convolution layer, a depth-separable convolution layer, and a compressed convolution layer;

[0038] Dilated convolution processing module: used to process the deep feature map in turn with the second channel attention mechanism and the second spatial attention mechanism to obtain a feature map after the second feature is enhanced;

[0039] The feature map after the second feature enhancement is sequentially subjected to a plurality of dilated convolution layers with specific dilation rates for multi-layer dilated convolution processing, wherein the multi-layer dilated convolution processing is performed based on the dilation rate set in each dilated convolution layer and the weight of the convolution kernel at a specific position of the feature map after the second feature enhancement, to obtain a plurality of dilated convolution features, and the plurality of dilated convolution features are subjected to feature fusion processing to obtain a fused feature;

[0040] Document correction module: The fused features are input into the fully connected layer and the convolution layer respectively. After the fully connected layer is processed, the predicted spatial information distribution is obtained, and the correction template is obtained based on the spatial information distribution; after the convolution layer is processed, the key feature points at a specific position are obtained; based on the coordinate positions of the correction template and the key feature points, the position mapping relationship between the correction template and the key feature points is obtained.

[0041] Based on the position mapping relationship, each pixel in the document image to be processed is interpolated to obtain the position correspondence between all pixels and their corresponding mapping coordinates in the correction template.

[0042] Based on the position correspondence, each pixel is moved to its corresponding mapping coordinate position in the correction template to obtain the document image after deformation restoration.

[0043] In addition, the present invention provides a 3D document key information deformation recovery device, including a processor and a memory, wherein the processor implements the 3D document key information deformation recovery method as described above when executing a computer program stored in the memory.

[0044] The present invention also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, can implement the steps of the method for restoring deformation of key information of a 3D document as described above.

[0045] The beneficial effects of the present invention are:

[0046] The present invention extracts shallow features from the deformed document image, performs a first feature enhancement, uses a plurality of inverted residual blocks to extract deep features, and then performs a second feature enhancement on the obtained deep feature map. By performing multi-layer dilated convolution processing on the feature map after the second feature enhancement, the perception ability of local details and global deformation of the document is enhanced, and key information features and overall deformation of the document image are given priority, which reduces the feature extraction time and effectively improves the performance in document image deformation recovery, ensuring that the final output image can maintain the original layout and content of the document while maintaining high quality.

[0047] The fused features are fully connected and convoluted to obtain the correction template and key point features respectively, and a position mapping relationship is constructed. After difference processing, the position correspondence between all pixel points and the reference points in the correction image is obtained to restore the 3D deformation of the document image. The present invention realizes pixel mapping from a distorted document image to a correction template by constructing a position mapping relationship, can effectively correct complex document layouts, automatically detect and correct deformations in document images, improve the accuracy of information recognition, and enhance visual effects.

[0048] 3. The loss function designed by the present invention takes into account the edge gradient direction of the control point and the edge gradient direction of the reference point, and can maintain the original texture rules of the edge of the document image, that is, the straight or horizontal lines, to a large extent. It not only improves the document deformation recovery performance, but also performs well in edge retention of key information such as barcodes and QR codes, ensuring the subsequent decoding and recognition of key information such as barcodes or QR codes. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] In the attached picture:

[0050] Figure 1 A schematic diagram of a flow chart of a method for restoring key information deformation of a 3D document in an embodiment;

[0051] Figure 2 It is a deformation recovery test result diagram of some key information in the embodiment;

[0052] Figure 3 This is a comparative experimental diagram of the optical marker array deformation recovery in the embodiment. DETAILED DESCRIPTION

[0053] The exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings.

[0054] This embodiment provides a method for restoring key information deformation of a 3D document. Figure 1 , including the following steps:

[0055] S1. Perform multi-layer convolution processing, batch normalization and nonlinear processing on the document image to be processed in sequence, extract shallow features of the document image, perform shallow feature mapping on it, and obtain a channel feature map, wherein each shallow feature corresponds to a channel.

[0056] Specifically, the distorted document image is preprocessed before multi-layer convolution, batch normalization and nonlinear processing, including scaling, cropping and normalization operations.

[0057] The multi-layer convolution processing is performed on the distorted document image. Specifically, two 3×3 convolution layers with a step size of 2 are used to perform convolution processing on the distorted document image.

[0058] The shallow features of the document image obtained include: edge features, texture features, color, brightness and local shape of the document.

[0059] S2, performing the first channel attention mechanism processing and the first spatial attention mechanism processing on the channel feature map in sequence to obtain a feature map after the first feature is enhanced;

[0060] A plurality of inverted residual blocks are used to extract deep features from the feature map after the first feature enhancement to obtain a deep feature map, wherein each inverted residual block includes, in a processing order, an extended convolution layer, a depth-separable convolution layer, and a compressed convolution layer.

[0061] Specifically, the channel feature map is processed by the first channel attention mechanism and the first spatial attention mechanism in sequence to obtain a feature map after the first feature is enhanced, specifically:

[0062] The channel feature map is pooled, transformed and activated to obtain the attention weight of each channel. The attention weight is multiplied by the channel feature map to obtain the feature map after the channel attention mechanism is applied.

[0063] The feature map processed by the channel attention mechanism is transformed into a spatial feature to obtain a spatial attention map. The spatial attention map and the feature map processed by the channel attention mechanism are element-wise multiplied to obtain a feature map after the first feature is enhanced.

[0064] Furthermore, each inverted residual block includes a dilated convolutional layer, a depth-wise separable convolutional layer, and a compressed convolutional layer, where:

[0065] The extended convolution layer uses a 1×1 convolution kernel to expand the number of channels of the feature map after the first feature enhancement to obtain a feature map after channel expansion;

[0066] The feature map after channel expansion is processed by depth-separable convolution layer using 3×3 convolution kernel to perform depth convolution on each channel, extract the depth features, and obtain the feature map after depth convolution;

[0067] The feature map after the deep convolution is compressed using a 1×1 convolution kernel in the compressed convolution layer to output the final deep feature map.

[0068] The depth feature is the number of bits that store each pixel in the distorted document image.

[0069] S3, performing the second channel attention mechanism processing and the second spatial attention mechanism processing on the depth feature map in sequence to obtain a feature map after the second feature is enhanced;

[0070] The feature map after the second feature enhancement is sequentially subjected to multi-layer dilated convolution processing through several dilated convolution layers with specific dilation rates. The multi-layer dilated convolution processing is performed based on the dilation rate set for each dilated convolution layer and the weight of the convolution kernel at a specific position of the feature map after the second feature enhancement, to obtain several dilated convolution features, and the several dilated convolution features are subjected to feature fusion processing to obtain a fused feature.

[0071] In this embodiment, 6 dilated convolutional layers are set. Each convolutional layer uses a different dilation rate to achieve a different receptive field, and the specific dilation rates are set to 1, 2, 5, 8, 12 and 18 in sequence, which can capture rich features at different scales.

[0072] Assume that the input feature map size is H×W, the convolution kernel size is k×k, the step size is s, the padding is p, and the expansion rate is d. For the position on the input feature map The output of the dilated convolution can be expressed as:

[0073] ,

[0074] Among them, w(m,n) is the weight of the convolution kernel at position (m,n), input is the input feature map, output is the output feature map, and d is the dilation rate. The dilation rate d determines the sampling interval of the convolution kernel on the input feature map, thereby affecting the size of the receptive field.

[0075] Specifically, the depth feature map is processed by the second channel attention mechanism and the second spatial attention mechanism in sequence to obtain a feature map after the second feature enhancement, specifically:

[0076] The deep feature map is pooled, transformed, and activated to obtain the attention weight of each channel. The attention weight is multiplied by the deep feature map by channel to obtain the feature map processed by the channel attention mechanism.

[0077] The feature map processed by the channel attention mechanism is transformed into a spatial feature to obtain a spatial attention map. The spatial attention map and the feature map processed by the channel attention mechanism are element-wise multiplied to obtain a feature map after the second feature is enhanced.

[0078] S4, input the fusion features into the fully connected layer and the convolution layer respectively, obtain the predicted spatial information distribution after the fully connected layer is processed, and obtain the correction template based on the spatial information distribution; obtain the key feature points at specific positions after the convolution layer is processed; obtain the position mapping relationship between the correction template and the key feature points based on the coordinate positions of the correction template and the key feature points. This process mainly focuses on extracting the key feature points in the image. By detecting and matching these key feature points, we can understand the overall structure and important areas of the image. The output of this process is the correspondence between a small number of feature points.

[0079] Based on the position mapping relationship, each pixel in the document image to be processed is interpolated to obtain the position correspondence between all pixels and their corresponding mapping coordinates in the correction template. After obtaining the key feature points, the entire image is processed to generate a detailed correspondence between each pixel. The result of this stage is a complete pixel mapping relationship.

[0080] Based on the position correspondence, each pixel is moved to its corresponding mapping coordinate position in the correction template to obtain the deformed document image. With the generated complete pixel mapping relationship, each pixel can be moved to the position of the reference point. This step is based on the previous position information to ensure that the control point is accurately positioned at the reference point.

[0081] The output document image after shape restoration has corrected the 3D deformation and restored to a 2D image close to the original flat state, which is convenient for subsequent barcode recognition, text recognition and other further processing.

[0082] This embodiment uses a bilinear interpolation method to perform interpolation processing, and other interpolation methods such as a TPS interpolation method may also be selected according to actual conditions.

[0083] Specifically, the key feature points at specific locations obtained after convolution processing in the convolution layer are defined as control points, and the coordinate positions of the key feature points corresponding to the mapping in the correction template are defined as reference points. The key feature points at specific locations include: the four corner points of the distorted document image, the four corner points and edge position points of the optical mark, barcode and QR code in the distorted document image.

[0084] Furthermore, based on the number of key feature points and the distances between the key feature points and their corresponding mapped coordinate positions in the calibration template, the positioning loss is obtained.

[0085] Positioning loss ( ): This loss function is used to regress the position of the control point to ensure that the predicted control point is as close to the actual control point as possible. The formula is expressed as:

[0086] ,

[0087] in, represents the number of control points, represents the predicted position of the ith control point, Indicates the corresponding real position. h L1 The loss is a loss function that is insensitive to outliers. It uses squared error when the prediction deviation is small and absolute error when the deviation is large, thereby reducing the impact of outliers.

[0088] Based on the number of key feature points and the distance between key feature points and their neighbor key feature points, the neighborhood relationship loss is obtained.

[0089] Neighborhood relation loss ( ): This loss function is designed to simulate the proximity relationship between control points and ensure that the control points maintain the correct relative position relationship locally. It helps to keep the corrected shape of the document more natural and coherent. The formula is:

[0090] ,

[0091] in, represents the number of control points, i represents the sequence number of the control point, represents the target position of the control point, Indicates the actual position of the control point.

[0092] Based on the number of key feature points, the interval of key feature points, and the interval of key feature points in the correction template, the interval loss regression loss is obtained.

[0093] Margin loss regression loss ( ): This loss function is used to regress the interval between reference points to more accurately predict the distance between control points. The formula is expressed as:

[0094] ,

[0095] in, represents the number of control points, is equal to 2, d represents the interval between two horizontal or vertical points, and The predicted interval and the true interval are represented separately. Helps ensure that the document's corrected grid is more uniform and regular.

[0096] Based on the number of key feature points, the gradient direction of the key feature points, and the gradient direction of the key feature points corresponding to the mapping coordinate position in the correction template, the gradient direction loss of the key feature points is obtained.

[0097] The application scenario of this embodiment mainly involves the deformation restoration of key information in documents such as barcodes and QR codes. Considering their inherent texture rules, the grayscale gradient direction at the edge should be biased towards horizontal or vertical. Therefore, in the restored document image, the gradient direction of the control point should be consistent with that of the reference point in the undeformed image. Based on this, a gradient direction loss function of the control point is designed ( ), the formula is:

[0098] ,

[0099] in, represents the number of control points, represents the gradient direction of the i-th control point, It represents the gradient direction of the corresponding reference point. This loss function keeps the predicted transformation direction of the edge control points of high grayscale gradient areas, such as barcodes, QR codes, and markers, consistent with the actual gradient direction.

[0100] The final loss function is a linear combination of the above losses. The positioning loss, neighborhood relationship loss, interval loss regression loss and key feature point gradient direction loss are linearly weighted to obtain the loss function. , , is with , as well as The relevant weight is expressed as:

[0101] ,

[0102] The loss function used in this embodiment takes into account the edge gradient direction of the control point and the edge gradient direction of the reference point, and can maintain the original texture rules of the document image edge to a large extent, that is, straight or horizontal lines. It not only improves the document deformation recovery performance, but also performs well in maintaining the edges of key information such as barcodes and QR codes, ensuring the subsequent decoding and recognition of key information such as barcodes or QR codes.

[0103] The present invention also provides a 3D document key information deformation recovery system, comprising:

[0104] Shallow feature extraction module: used to perform multi-layer convolution processing, batch normalization and nonlinear processing on the document image to be processed in sequence, extract shallow features of the document image, perform shallow feature mapping on it, and obtain a channel feature map, wherein each shallow feature corresponds to a channel;

[0105] Deep feature extraction module: used to perform first channel attention mechanism processing and first spatial attention mechanism processing on the channel feature map in sequence to obtain a feature map after the first feature is enhanced;

[0106] Extracting deep features from the feature map after the first feature enhancement using a plurality of inverted residual blocks to obtain a deep feature map, wherein each inverted residual block includes, in a processing order, an extended convolution layer, a depth-separable convolution layer, and a compressed convolution layer;

[0107] Dilated convolution processing module: used to process the deep feature map in turn with the second channel attention mechanism and the second spatial attention mechanism to obtain a feature map after the second feature is enhanced;

[0108] The feature map after the second feature enhancement is sequentially subjected to a plurality of dilated convolution layers with specific dilation rates for multi-layer dilated convolution processing, wherein the multi-layer dilated convolution processing is performed based on the dilation rate set in each dilated convolution layer and the weight of the convolution kernel at a specific position of the feature map after the second feature enhancement, to obtain a plurality of dilated convolution features, and the plurality of dilated convolution features are subjected to feature fusion processing to obtain a fused feature;

[0109] Document correction module: used to input the fusion features into the fully connected layer and the convolution layer respectively, obtain the predicted spatial information distribution after the fully connected layer is processed, and obtain the correction template based on the spatial information distribution; obtain the key feature points at a specific position after the convolution layer is processed; based on the coordinate positions of the correction template and the key feature points, obtain the position mapping relationship between the correction template and the key feature points,

[0110] Based on the position mapping relationship, each pixel in the document image to be processed is interpolated to obtain the position correspondence between all pixels and their corresponding mapping coordinates in the correction template.

[0111] Based on the position correspondence, each pixel is moved to its corresponding mapping coordinate position in the correction template to obtain the document image after deformation restoration.

[0112] In addition, the present invention also provides a 3D document key information deformation recovery device, including a processor and a memory, wherein the processor implements the 3D document key information deformation recovery method as described above when executing a computer program stored in the memory.

[0113] Finally, the present invention provides a computer-readable storage medium storing a computer program, which, when executed by a processor, can implement the above-mentioned method for restoring deformation of key information of a 3D document.

[0114] Experimental Section

[0115] The present invention proposes a 3D document key information deformation recovery method that performs well in image similarity measurement (MS-SSMI), local deformation (LD), character edit distance (ED), character error rate (CER), and memory usage (Para), especially in the two indicators of MS-SSMI and LD. The larger the value of MS-SSMI, the better the performance; the smaller the value of LD, ED, EDR, and Para, the better the performance. Compared with other methods, the 3D document key information deformation recovery method used in the present invention is particularly significant in performance improvement, as shown in Table 1.

[0116] Table 1 Performance comparison of different deformation recovery methods on different indicators

[0117]

[0118] Table 1 shows that the method for restoring the deformation of key information in 3D documents proposed by the present invention has significant advantages in maintaining image quality and restoring deformation, and can more accurately complete the deformation restoration of key information, especially in local details. In terms of the amount of memory parameters occupied, the number of parameters of the present invention is 15.1M. Compared with other methods (such as 13.25M of DDCP and 24.8M of DocGeoNet), the method of the present invention achieves higher performance while maintaining lightweight, reflecting high efficiency and economy, especially in resource-constrained environments. It has obvious application advantages. At the same time, the performance of the present invention in ED and CER is also very outstanding. These two indicators are directly related to the accuracy and availability of the key information image after deformation restoration. This means that when the present invention restores the deformed image of key information, the overall information loss of the identified or decoded character string is less, the accuracy of the restored local details is relatively high, and it is significantly better than other methods.

[0119] In addition, in order to better compare the test results, a corresponding visual effect comparison was also performed. Since the key information of the present invention is mainly aimed at the key information in the answer sheet image, the deformation recovery effects of key information such as optical marks, barcodes, and QR codes are compared respectively, and for unified testing, the input is a deformed image with background information. The 3D document key information deformation recovery method proposed by the present invention shows good deformation recovery quality, especially in terms of local texture information and complete edge extraction. Among them, DewarpNet, DocTr and Textin are not very effective in removing the background edges of images, and large background edges remain.

[0120] The method for restoring the deformation of key information in a 3D document proposed in the present invention can pay more attention to the key information area by adding an attention mechanism and a multi-scale dilated convolution module. At the same time, it increases the perception of global deformation and local deformation. Therefore, both the global environmental edge filtering and the recovery of local deformation have better visual performance. In addition, since other methods do not have the rules of edge information in the key image, the edges of the restored images are distorted to a certain extent. In the method for restoring the deformation of key information in a 3D document proposed in the present invention, since this key rule is paid attention to and thanks to the designed gradient direction loss function, the original texture rules of the edges on the true image can be maintained to a large extent in edge recovery, that is, straight or horizontal lines. This is crucial for key information such as barcodes or QR codes, and greatly affects the final decoding and recognition effects. The specific experimental results are as follows. Figure 2 , Figure 3 shown.

[0121] Among them, the experimental results on the same test set are as follows Figure 2 As shown. In this embodiment, the channel feature map is processed by the first channel attention mechanism and the first space attention mechanism in sequence, and the depth feature map is processed by the second channel attention mechanism and the second space attention mechanism in sequence. At the same time, the feature map after the second feature enhancement is processed by multi-layer dilation convolution, which significantly enhances the perception of local and global deformation of the document and makes the restored image clearer. In addition, the loss function designed in this embodiment effectively retains the texture of the edge of key information, which is crucial for the accurate recognition of barcodes and QR codes. Therefore, this embodiment performs well in document deformation recovery and minimizes the loss of original image information.

[0122] The deformation recovery test was conducted on the key information such as optical marks, barcodes, and QR codes in the answer sheet images. Figure 3 The results show that this embodiment is superior to other methods in visual effect. Compared with DewarpNet, DocTr and Textin, this embodiment is more effective in background edge removal, especially in local texture information and complete edge extraction.

Claims

1. A method for restoring deformation of key information of a 3D document, characterized in that: The following steps are involved: S1, performing multi-layer convolution processing, batch normalization and nonlinear processing on the document image to be processed in sequence, extracting shallow features of the document image, performing shallow feature mapping on the shallow features to obtain a channel feature map, wherein each shallow feature corresponds to a channel; S2, performing the first channel attention mechanism processing and the first spatial attention mechanism processing on the channel feature map in sequence to obtain a feature map after the first feature is enhanced; Extracting deep features from the feature map after the first feature enhancement using a plurality of inverted residual blocks to obtain a deep feature map, wherein each inverted residual block includes, in a processing order, an extended convolution layer, a depth-separable convolution layer, and a compressed convolution layer; S3, performing the second channel attention mechanism processing and the second spatial attention mechanism processing on the depth feature map in sequence to obtain a feature map after the second feature is enhanced; The feature map after the second feature enhancement is sequentially subjected to a plurality of dilated convolution layers with specific dilation rates for multi-layer dilated convolution processing, wherein the multi-layer dilated convolution processing is performed based on the dilation rate set in each dilated convolution layer and the weight of the convolution kernel at a specific position of the feature map after the second feature enhancement, to obtain a plurality of dilated convolution features, and the plurality of dilated convolution features are subjected to feature fusion processing to obtain a fused feature; S4, input the fusion features into the fully connected layer and the convolution layer respectively, obtain the predicted spatial information distribution after the fully connected layer is processed, and obtain the correction template based on the spatial information distribution; obtain the key feature points at specific positions after the convolution layer is processed; based on the coordinate positions of the correction template and the key feature points, obtain the position mapping relationship between the correction template and the key feature points, Based on the position mapping relationship, each pixel in the document image to be processed is interpolated to obtain the position correspondence between all pixels and their corresponding mapping coordinates in the correction template. Based on the position correspondence, each pixel is moved to its corresponding mapping coordinate position in the correction template to obtain the document image after deformation restoration.

2. The method for restoring 3D document key information deformation according to claim 1, characterized in that: The loss function used in S4 is specifically: Based on the number of key feature points and the distance between the key feature points and their corresponding mapped coordinate positions in the calibration template, the positioning loss is obtained; Based on the number of key feature points and the distance between key feature points and their neighbor key feature points, the neighborhood relationship loss is obtained; Based on the number of key feature points, the interval of key feature points, and the interval of key feature points in the correction template, the interval loss regression loss is obtained; Based on the number of key feature points, the gradient direction of the key feature points, and the gradient direction of the key feature points corresponding to the mapping coordinate position in the correction template, the gradient direction loss of the key feature points is obtained; The positioning loss, neighborhood relationship loss, interval loss regression loss and key feature point gradient direction loss are linearly weighted to obtain the loss function.

3. The method for restoring 3D document key information deformation according to claim 1, characterized in that: Each inverted residual block in S2 includes, in processing order, an extended convolution layer, a depthwise separable convolution layer, and a compressed convolution layer, where: The extended convolution layer uses a 1×1 convolution kernel to expand the number of channels of the feature map after the first feature enhancement to obtain a feature map after channel expansion; The feature map after channel expansion is processed by a depth-separable convolution layer using a 3×3 convolution kernel to perform depth convolution on each channel, extract the depth features, and obtain the feature map after depth convolution; The feature map after the deep convolution is compressed using a 1×1 convolution kernel in the compressed convolution layer to output the final deep feature map.

4. The method for restoring 3D document key information deformation according to claim 1, characterized in that: In S1, the document image to be processed is subjected to multi-layer convolution processing in sequence, specifically, two 3×3 convolution layers with a step size of 2 are used to perform convolution processing on the distorted document image.

5. The method for restoring 3D document key information deformation according to claim 1, characterized in that: In S2, the channel feature map is processed by the first channel attention mechanism and the first spatial attention mechanism in sequence to obtain a feature map after the first feature enhancement, specifically: The channel feature map is pooled, transformed and activated to obtain the attention weight of each channel. The attention weight is multiplied by the channel feature map to obtain the feature map after the channel attention mechanism is applied. The feature map processed by the channel attention mechanism is transformed into a spatial feature to obtain a spatial attention map. The spatial attention map and the feature map processed by the channel attention mechanism are element-wise multiplied to obtain a feature map after the first feature is enhanced.

6. The method for restoring 3D document key information deformation according to claim 1, characterized in that: In S3, the depth feature map is processed by the second channel attention mechanism and the second spatial attention mechanism in sequence to obtain a feature map after the second feature enhancement, specifically: The deep feature map is pooled, transformed, and activated to obtain the attention weight of each channel. The attention weight is multiplied by the deep feature map by channel to obtain the feature map processed by the channel attention mechanism. The feature map processed by the channel attention mechanism is transformed into a spatial feature to obtain a spatial attention map. The spatial attention map and the feature map processed by the channel attention mechanism are element-wise multiplied to obtain a feature map after the second feature is enhanced.

7. The method for restoring 3D document key information deformation according to claim 1, characterized in that: The key feature points at the specific positions in S4 include: four corner points of the distorted document image, four corner points and edge position points of the optical mark, barcode and two-dimensional code in the distorted document image.

8. A 3D document key information deformation recovery system, characterized in that: include: Shallow feature extraction module: used to perform multi-layer convolution processing, batch normalization and nonlinear processing on the document image to be processed in sequence, extract shallow features of the document image, perform shallow feature mapping on it, and obtain a channel feature map, wherein each shallow feature corresponds to a channel; Deep feature extraction module: used to perform first channel attention mechanism processing and first spatial attention mechanism processing on the channel feature map in sequence to obtain a feature map after the first feature is enhanced; Extracting deep features from the feature map after the first feature enhancement using a plurality of inverted residual blocks to obtain a deep feature map, wherein each inverted residual block includes, in a processing order, an extended convolution layer, a depth-separable convolution layer, and a compressed convolution layer; Dilated convolution processing module: used to process the deep feature map in turn with the second channel attention mechanism and the second spatial attention mechanism to obtain a feature map after the second feature is enhanced; The feature map after the second feature enhancement is sequentially subjected to a plurality of dilated convolution layers with specific dilation rates for multi-layer dilated convolution processing, wherein the multi-layer dilated convolution processing is performed based on the dilation rate set in each dilated convolution layer and the weight of the convolution kernel at a specific position of the feature map after the second feature enhancement, to obtain a plurality of dilated convolution features, and the plurality of dilated convolution features are subjected to feature fusion processing to obtain a fused feature; Document correction module: The fused features are input into the fully connected layer and the convolution layer respectively. After the fully connected layer is processed, the predicted spatial information distribution is obtained, and the correction template is obtained based on the spatial information distribution; after the convolution layer is processed, the key feature points at a specific position are obtained; based on the coordinate positions of the correction template and the key feature points, the position mapping relationship between the correction template and the key feature points is obtained. Based on the position mapping relationship, each pixel in the document image to be processed is interpolated to obtain the position correspondence between all pixels and their corresponding mapping coordinates in the correction template. Based on the position correspondence, each pixel is moved to its corresponding mapping coordinate position in the correction template to obtain the document image after deformation restoration.

9. A 3D document key information deformation recovery device, characterized in that: The method comprises a processor and a memory, wherein the processor implements the 3D document key information deformation recovery method as described in any one of claims 1 to 7 when executing the computer program stored in the memory.

10. A computer-readable storage medium, characterized in that: Used to store a computer program, wherein when the computer program is executed by a processor, the method for restoring deformation of key information of a 3D document according to any one of claims 1 to 7 is implemented.

Images