A method and system for rectifying a distorted document image based on multi-task guided multi-view feature fusion

Abstract

The application discloses a kind of based on multi-task guide's multi-view feature fusion's distortion document image rectification method and system, comprising: obtaining distorted document image, and distorted document image is input to visual encoder and obtains distorted document image visual feature;Distorted document image visual feature is input two coordinate branches respectively to calculate the world coordinate of distorted image, UV coordinate mapping, and obtain coordinate branch view feature;Distorted document image visual feature is input two plane branches respectively to calculate foreground mask and text line mask, and obtain plane branch view feature;Coordinate branch view feature and plane branch view feature are input multi-view fusion module, and obtain two-dimensional deformation field coordinates;The pixel position of the distortion document image is mapped using two-dimensional deformation field coordinates, and the rectification of distortion document image is realized.The application is suitable for the rectification of most distortion document images with complex background, improves visual effect and text recognition accuracy.

Description

Technical Field

[0001] This invention relates to the field of distorted document image correction technology, and in particular to a method and system for distorted document image correction based on multi-task guided multi-view feature fusion. Background Technology

[0002] Document image understanding is an important research area in computer vision. With the widespread use of mobile devices, people can more easily use smartphones or portable cameras to take pictures to obtain electronic documents. However, during the process of taking pictures with mobile devices, due to issues such as the position of these devices, lighting conditions, paper placement angle, and distortion, the captured images are distorted to varying degrees. Existing text recognition and document understanding methods cannot accurately extract the structured data (text content, key field extraction, table structure, document layout analysis, etc.) from these images.

[0003] Image correction can bring more aesthetically pleasing visualizations, more accurate text recognition results, and more complete understanding of document content to distorted document images. In recent years, the development of deep learning technology has promoted the development of distorted document image correction tasks. However, most existing distorted document image correction methods are designed for single tasks and single views, and cannot effectively focus on the global and local features of distorted document images. For example, the patent application number CN202211637317.7 (patent title: Direction Correction Method, Device, Medium and Equipment for Distorted Text Lines) only focuses on the curve features of the text line region. This method ignores the multi-granularity view features of the distorted document image, such as world coordinates, foreground, and document boundaries.

[0004] Therefore, in order to address the above problems, how to provide a method and system for correcting distorted document images based on multi-task guided multi-view feature fusion has become a key research task for those skilled in the art. Summary of the Invention

[0005] The technical problem to be solved by the present invention is to provide a method and system for correcting distorted document images based on multi-task guided multi-view fusion, which addresses the above-mentioned deficiencies of the prior art and aims to solve the problem that the existing technology cannot effectively fuse the multi-view features of distorted documents, thereby improving the global and local correction effect of document images.

[0006] The technical solution adopted by the present invention to solve this problem is as follows:

[0007] In a first aspect, the present invention provides a method for correcting distorted document images based on multi-task guided multi-view feature fusion, comprising the following steps:

[0008] Step S1: Obtain a distorted document image and input the distorted document image into a visual encoder to obtain distorted document image visual features corresponding to the document image. The distorted document image visual features are used to characterize the visual content information of the document image.

[0009] Step S2: Input the visual features of the distorted document image into the coordinate branch to obtain the coordinate branch view features, which are used to characterize the spatial features of the document image;

[0010] Step S3: Input the visual features of the distorted document image into the planar branch to obtain the planar branch view features. The planar branch view features are used to characterize the global and local information on the two-dimensional plane of the document image.

[0011] Step S4: Input the coordinate branch view features and planar branch view features into the multi-view fusion module to obtain two-dimensional deformation field coordinates, which are the pixel position mappings between the distorted document image and the scanned image;

[0012] Step S5: The pixel positions of the distorted document image are mapped using two-dimensional deformation field coordinates to correct the distorted document image.

[0013] In one implementation, step S2, the method for inputting the distorted document image visual features into the coordinate branch to obtain the coordinate branch view features includes:

[0014] The visual features of the distorted document image are input into the world coordinate branch of the attention module to obtain the world coordinate view features, where the world coordinates are the true coordinates of the distorted document image in the three-dimensional world.

[0015] The world coordinate view features are input into the upsampling module to calculate the world coordinates of the distorted document image;

[0016] The distorted document image visual features are input into the UV coordinate branch of the attention module to obtain UV coordinate view features;

[0017] The UV coordinate view features are input into the upsampling module to calculate the UV coordinate mapping of the distorted document image. The UV coordinate mapping is a coordinate mapping from the scanned image to the three-dimensional distorted image.

[0018] The coordinate branch view feature is obtained by splicing the world coordinate view feature and the UV coordinate view feature.

[0019] In one implementation, step S3, the method for inputting the distorted document image visual features into the planar branch to obtain the planar branch view features includes:

[0020] The distorted document image visual features are input into the foreground branch of the attention module to obtain foreground view features;

[0021] The foreground view features are input into the upsampling module to obtain the foreground mask of the distorted document image, and the foreground mask represents the global information of the distorted document image;

[0022] The distorted document image visual features are input into the text line branch of the attention module to obtain the text line view features;

[0023] The text line view features are input into the upsampling module to obtain the text line mask of the distorted document image, and the text line mask represents the local information of the distorted document image;

[0024] The foreground view feature and the text line view feature are concatenated to obtain the planar branch view feature.

[0025] In one implementation, step S4, where the coordinate branch view features and planar branch view features are input into a multi-view feature fusion module to obtain two-dimensional deformation field coordinates, includes the following method:

[0026] The coordinate branch view features are used as queries, and the planar branch view features are used as keys. These are input into the cross-attention layer to obtain the coordinate branch attention features.

[0027] The planar branch view features are used as queries, and the coordinate branch view features are used as keys. These are input into the cross-attention layer to obtain the planar branch attention features.

[0028] The coordinate branch attention features and the planar branch attention features are concatenated and input into the feedforward layer to obtain multi-view fusion features;

[0029] The multi-view fusion features are input into the grid coordinate sampling module to obtain two-dimensional deformation field coordinates.

[0030] In a second aspect, the present invention provides a distorted document image correction system based on multi-view feature fusion guided by multi-task, comprising: a visual feature extraction module, a multi-view feature extraction module, a multi-view feature fusion module, a grid coordinate sampling module, and a distortion correction module;

[0031] The multi-view feature extraction module is used to automatically extract multi-view features of distorted document images from their visual features.

[0032] The multi-view feature fusion module is used to fuse multi-view features of a distorted document image to obtain multi-view fused features.

[0033] The grid coordinate sampling module is used to learn two-dimensional deformation field coordinates from multi-view fusion features, representing the pixel position mapping between the distorted document image and the scanned image;

[0034] The distortion correction module uses two-dimensional deformation field coordinate mapping to map the pixel positions of distorted document images, thereby correcting the distorted document images.

[0035] In one implementation, the multi-view features of the distorted document image include features of the distorted document image such as world coordinates, UV coordinates, foreground mask, and text line mask.

[0036] The distorted document image correction method based on multi-view feature fusion guided by multi-task of the present invention can be deployed on various platforms or devices, supports the correction of distorted document images with different backgrounds in real scenes, and achieves efficient and aesthetically pleasing correction results.

[0037] Thirdly, the present invention also discloses a terminal device, the terminal device comprising a memory, a processor, and a program stored in the memory and executable on the processor for a distorted document image correction method based on multi-task guided multi-view feature fusion. When the processor executes the program for the distorted document image correction method, it implements the steps of the distorted document image correction method based on multi-task guided multi-view feature fusion as described in any of the preceding claims.

[0038] Fourthly, the present invention also discloses a computer-readable storage medium storing a program for a distorted document image correction method based on multi-task guided multi-view feature fusion, wherein when the program for the distorted document image correction method based on multi-task guided multi-view feature fusion is executed by a processor, the program implements the steps of the distorted document image correction method based on multi-task guided multi-view feature fusion as described in any of the preceding claims.

[0039] The beneficial effects of this invention are:

[0040] Compared with existing technologies, firstly, this invention enables end-to-end training and inference between multiple tasks; secondly, this invention supports the correction of document images with varying degrees of deformation and is compatible with the correction of document images with complex backgrounds in real-world scenarios; finally, this invention integrates multiple views of distorted document images and designs an effective fusion module to enhance the interaction between multiple views, which helps to perceive the document distortion trend and improve the correction effect. Attached Figure Description

[0041] Figure 1 This is a flowchart of a distorted document image correction method based on multi-view feature fusion guided by multi-task according to the present invention.

[0042] Figure 2This is a schematic diagram of the network structure according to an embodiment of the present invention.

[0043] Figure 3 This is a schematic diagram of the multi-view fusion module according to an embodiment of the present invention. Detailed Implementation

[0044] To more clearly and explicitly demonstrate the objectives, technical solutions, and advantages of this invention, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that these descriptions are merely exemplary and not intended to limit the scope of the invention.

[0045] Because existing technologies cannot effectively capture the distortion trend of document images when correcting distorted documents, and lack the ability to explore the commonalities and complementarities among various view features of distorted images, the correction effect on document images is poor.

[0046] To address the problems of existing technologies, this embodiment provides a method for correcting distorted document images based on multi-task guided multi-view feature fusion. This method can improve the correction effect of distorted documents. Specifically, firstly, a distorted document image is acquired and input into a visual encoder to obtain distorted document image visual features corresponding to the document image. These visual features characterize the visual content information of the document image. Secondly, the distorted document image visual features are input into a coordinate branch to obtain coordinate branch view features, which characterize the spatial features of the document image. The distorted document image visual features are then input into a planar branch to obtain planar branch view features, which characterize the global and local information on the two-dimensional plane of the document image. Thirdly, the coordinate branch view features and planar branch view features are input into a multi-view fusion module to obtain two-dimensional deformation field coordinates, which map the pixel positions between the distorted document image and the scanned image. Finally, the pixel positions of the distorted document image are mapped using the two-dimensional deformation field coordinates to achieve the correction of the distorted document image. Therefore, this invention enhances the model's understanding of document features by extracting features from multiple views of the input distorted document image. Then, by fusing the features from multiple views, it further perceives the distortion characteristics of the distorted document image, achieving feature complementarity between multiple views and multi-granularity features. This invention can significantly improve the correction effect of distorted document images and the accuracy of text recognition.

[0047] In one implementation, step S2, the method for inputting the distorted document image visual features into the coordinate branch to obtain the coordinate branch view features includes:

[0048] The visual features of the distorted document image are input into the world coordinate branch of the attention module to obtain the world coordinate view features, where the world coordinates are the true coordinates of the distorted document image in the three-dimensional world.

[0049] The world coordinate view features are input into the upsampling module to calculate the world coordinates of the distorted document image;

[0050] The distorted document image visual features are input into the UV coordinate branch of the attention module to obtain UV coordinate view features;

[0051] The UV coordinate view features are input into the upsampling module to calculate the UV coordinate mapping of the distorted document image. The UV coordinate mapping is a coordinate mapping from the scanned image to the three-dimensional distorted image.

[0052] The coordinate branch view feature is obtained by splicing the world coordinate view feature and the UV coordinate view feature.

[0053] In one implementation, step S3, the method for inputting the distorted document image visual features into the planar branch to obtain the planar branch view features includes:

[0054] The distorted document image visual features are input into the foreground branch of the attention module to obtain foreground view features;

[0055] The foreground view features are input into the upsampling module to obtain the foreground mask of the distorted document image, and the foreground mask represents the global information of the distorted document image;

[0056] The distorted document image visual features are input into the text line branch of the attention module to obtain the text line view features;

[0057] The text line view features are input into the upsampling module to obtain the text line mask of the distorted document image, and the text line mask represents the local information of the distorted document image;

[0058] The foreground view feature and the text line view feature are concatenated to obtain the planar branch view feature.

[0059] In one implementation, step S4, where the coordinate branch view features and planar branch view features are input into a multi-view feature fusion module to obtain two-dimensional deformation field coordinates, includes the following method:

[0060] The coordinate branch view features are used as queries, and the planar branch view features are used as keys. These are input into the cross-attention layer to obtain the coordinate branch attention features.

[0061] The planar branch view features are used as queries, and the coordinate branch view features are used as keys. These are input into the cross-attention layer to obtain the planar branch attention features.

[0062] The coordinate branch attention features and the planar branch attention features are concatenated and input into the feedforward layer to obtain multi-view fusion features;

[0063] The multi-view fusion features are input into the grid coordinate sampling module to obtain two-dimensional deformation field coordinates.

[0064] The present invention also provides a distorted document image correction system based on multi-view feature fusion guided by multi-task, comprising: a visual feature extraction module, a multi-view feature extraction module, a multi-view feature fusion module, a grid coordinate sampling module, and a distortion correction module;

[0065] The multi-view feature extraction module is used to automatically extract multi-view features of the distorted document image from its visual features, including world coordinates, UV coordinates, foreground mask, and text line mask, etc.

[0066] The multi-view feature fusion module is used to fuse view features such as world coordinates, UV coordinates, foreground mask, and text line mask to obtain multi-view fusion features.

[0067] The grid coordinate sampling module is used to learn two-dimensional deformation field coordinates from multi-view fusion features, representing the pixel position mapping between the distorted document image and the scanned image;

[0068] The distortion correction module uses two-dimensional deformation field coordinate mapping to map the pixel positions of distorted document images, thereby correcting the distorted document images.

[0069] The present invention also discloses a terminal device, the terminal device including a memory, a processor, and a program for a distorted document image correction method based on multi-view feature fusion guided by multi-task, stored in the memory and executable on the processor. When the processor executes the program for the distorted document image correction method, it implements the steps of the distorted document image correction method based on multi-view feature fusion guided by multi-task as described in any of the preceding claims.

[0070] The present invention also discloses a computer-readable storage medium storing a program for a distorted document image correction method based on multi-task guided multi-view feature fusion. When the program for the distorted document image correction method based on multi-task guided multi-view feature fusion is executed by a processor, it implements the steps of the distorted document image correction method based on multi-task guided multi-view feature fusion as described in any of the preceding claims.

[0071] by Figure 1For example, this embodiment provides a method for correcting distorted document images based on multi-view feature fusion guided by multi-task, including the following steps:

[0072] Step S1: Obtain a distorted document image and input the distorted document image into a visual encoder to obtain distorted document image visual features corresponding to the document image. The distorted document image visual features are used to characterize the visual content information of the document image.

[0073] In this embodiment, the distorted document image is any image containing text or charts, and it can include any background scene. In practice, the distorted document image is first acquired and then input into a visual encoder to extract its visual features. These visual features characterize the visual content information of the document image. The image encoding model used in this embodiment consists of multiple residual modules containing convolutional and pooling layers. By extracting features from the distorted document image, general visual features can be obtained, enabling better prediction of the multi-view features of the distorted document image.

[0074] Step S2: Input the visual features of the distorted document image into the coordinate branch to obtain the coordinate branch view features, which are used to characterize the spatial features of the document image.

[0075] In this embodiment, the coordinate branch is divided into two sub-branches, which predict the world coordinates and UV coordinates of the distorted document image, respectively, and obtain the view features of the world coordinates and UV coordinates. The network structure of each sub-branch includes a multi-layer self-attention module and an upsampling module, such as... Figure 2 As shown, the self-attention module consists of a multi-head self-attention mechanism, a normalization layer, and a feedforward layer to obtain view features in world coordinates and UV coordinates; the upsampling module contains an 8x upsampling layer and a convolutional layer to obtain world coordinates and UV coordinates, respectively. The network structures of the two sub-branches are identical.

[0076] Step S3: Input the visual features of the distorted document image into the planar branch to obtain the planar branch view features. The planar branch view features are used to characterize the global and local information on the two-dimensional plane of the document image.

[0077] In this embodiment, the network structure of the planar branch is exactly the same as that of the coordinate branch in step S2.

[0078] Step S4: Input the coordinate branch view features and planar branch view features into the multi-view fusion module to obtain two-dimensional deformation field coordinates, which are the pixel position mappings between the distorted document image and the scanned image.

[0079] The multi-view fusion module used in this embodiment is a dual-path cross-attention fusion network. The multi-view fusion module consists of multiple layers, such as... Figure 3 As shown, firstly, the coordinate branch view feature is used as the query and the planar branch view feature as the key, and input into the cross-attention layer to obtain the coordinate branch attention feature; simultaneously, the planar branch view feature is used as the query and the coordinate branch view feature as the key, and input into the cross-attention layer to obtain the planar branch attention feature; further, the coordinate branch attention feature and the planar branch attention feature are concatenated and input into the feedforward layer to obtain the multi-view fusion feature; then, the multi-view fusion feature is separated in the channel dimension and used as the input to the next layer of the multi-view fusion module; the output of the last layer is concatenated in the channel dimension to obtain the multi-view fusion feature; the multi-view fusion feature is input into the mesh sampling module to obtain the two-dimensional deformation field coordinates.

[0080] Step S5: The pixel positions of the distorted document image are mapped using two-dimensional deformation field coordinates to correct the distorted document image.

[0081] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for rectifying a warped document image based on multi-task guided multi-view feature fusion, characterized in that, The method comprises the following steps: Step S1, obtaining a distorted document image and inputting the distorted document image into a visual encoder to obtain a distorted document image visual feature corresponding to the document image; Step S2, inputting the distorted document image visual feature into a coordinate branch to obtain a coordinate branch view feature; Step S3, inputting the distorted document image visual feature into a plane branch to obtain a plane branch view feature; Step S4, inputting the coordinate branch view feature and the plane branch view feature into a multi-view fusion module to obtain a two-dimensional deformation field coordinate; Step S5, mapping pixel positions of the distorted document image by using the two-dimensional deformation field coordinate to correct the distorted document image; In step S2, the method for inputting the distorted document image visual feature into the coordinate branch to obtain the coordinate branch view feature comprises: inputting the distorted document image visual feature into a world coordinate branch of a self-attention module to obtain a world coordinate view feature; inputting the world coordinate view feature into an upsampling module to calculate world coordinates of the distorted document image; inputting the distorted document image visual feature into a UV coordinate branch of the self-attention module to obtain a UV coordinate view feature; inputting the UV coordinate view feature into the upsampling module to calculate UV coordinate mapping of the distorted document image; splicing the world coordinate view feature and the UV coordinate view feature to obtain the coordinate branch view feature; In step S3, the method for inputting the distorted document image visual feature into the plane branch to obtain the plane branch view feature comprises: inputting the distorted document image visual feature into a foreground branch of the self-attention module to obtain a foreground view feature; inputting the foreground view feature into the upsampling module to obtain a foreground mask of the distorted document image; inputting the distorted document image visual feature into a text line branch of the self-attention module to obtain a text line view feature; inputting the text line view feature into the upsampling module to obtain a text line mask of the distorted document image; splicing the foreground view feature and the text line view feature to obtain the plane branch view feature. 2.The method of claim 1, wherein, In step S4, the method for inputting the coordinate branch view feature and the plane branch view feature into the multi-view feature fusion module to obtain the two-dimensional deformation field coordinate comprises: inputting the coordinate branch view feature as a query and the plane branch view feature as a key value into a cross-attention layer to obtain a coordinate branch attention feature; inputting the plane branch view feature as a query and the coordinate branch view feature as a key value into the cross-attention layer to obtain a plane branch attention feature; splicing the coordinate branch attention feature and the plane branch attention feature to input into a feedforward layer to obtain a multi-view fusion feature; inputting the multi-view fusion feature into a grid coordinate sampling module to obtain the two-dimensional deformation field coordinate.

3. A system for rectifying a warped document image based on multi-task guided multi-view feature fusion, the system comprising: The method comprises: a visual feature extraction module configured to extract a visual feature of a distorted document image, and the visual feature represents image visual content information of the document image; a multi-view feature extraction module configured to automatically extract a multi-view feature of the distorted document image from the visual feature of the distorted document image. The distorted document image visual features are input into a coordinate branch to obtain coordinate branch view features, and the method comprises: The distorted document image visual features are input into a world coordinate branch of a self-attention module to obtain world coordinate view features; The world coordinate view features are input into an upsampling module to calculate the world coordinates of the distorted document image; The distorted document image visual features are input into a UV coordinate branch of the self-attention module to obtain UV coordinate view features; The UV coordinate view features are input into the upsampling module to calculate the UV coordinate mapping of the distorted document image; The world coordinate view features and the UV coordinate view features are spliced to obtain the coordinate branch view features; The distorted document image visual features are input into a plane branch to obtain plane branch view features, and the method comprises: The distorted document image visual features are input into a foreground branch of the self-attention module to obtain foreground view features; The foreground view features are input into the upsampling module to obtain the foreground mask of the distorted document image; The distorted document image visual features are input into a text line branch of the self-attention module to obtain text line view features; The text line view features are input into the upsampling module to obtain the text line mask of the distorted document image; The foreground view features and the text line view features are spliced to obtain the plane branch view features; A multi-view feature fusion module is configured to fuse the multi-view features of the distorted document image to obtain multi-view fusion features; A grid coordinate sampling module is configured to learn two-dimensional deformation field coordinates from the multi-view fusion features, which represent the pixel position mapping between the distorted document image and the scanned image; A distortion correction module is configured to map the pixel positions of the distorted document image by using the two-dimensional deformation field coordinates, thereby correcting the distorted document image.

4. A terminal device, characterized by comprising: The terminal device comprises a memory, a processor, and a program of the distorted document image correction method based on multi-task guided multi-view feature fusion stored in the memory and executable on the processor, and when the processor executes the program of the distorted document image correction method, the steps of the distorted document image correction method based on multi-task guided multi-view feature fusion are implemented.

5. A computer readable storage medium, characterized in that, The computer readable storage medium stores a program of the distorted document image correction method based on multi-task guided multi-view feature fusion, and when the processor executes the program of the distorted document image correction method based on multi-task guided multi-view feature fusion, the steps of the distorted document image correction method based on multi-task guided multi-view feature fusion are implemented.

Images