Method, system, device and medium for annotating photographed document image data

Abstract

The present invention discloses a method, system, device, and medium for annotating photographed document image data, wherein the method comprises: obtaining a first document image; printing the first document image into a paper document; photographing the paper document to obtain a second document image; registering and aligning the first document image and the second document image to obtain an offset field; obtaining annotation information of the second document image based on the offset field and the first document image; and / or sampling the second document image based on the offset field to obtain a third document image aligned at the pixel level with the first document image; and / or forming the offset field and the second document image into document image correction task annotation data. The present invention maps existing annotated electronically generated documents and performs offset sampling on photographed documents through registration and alignment, thereby obtaining annotation data for multiple tasks at a relatively low cost. The present invention can be widely applied in the fields of pattern recognition and artificial intelligence technology.

Description

Technical Field

[0001] The present invention relates to the field of pattern recognition and artificial intelligence technology, and in particular to a method, system, device and medium for annotating photographed document image data. Background Art

[0002] Most existing document analysis and recognition systems only focus on scanned documents or electronically generated documents, but with the popularity of mobile cameras on mobile devices, document images now increasingly appear in the form of photographs. Unlike scanned or electronically generated documents, which are easy to label, it is more difficult to obtain labels for photographed document images. The reasons are as follows: the labels of electronically generated document images can be obtained by combining their source code files with some simple rules, but this method is not applicable to photographed document images; photographed document images contain geometric deformations and require more intensive labeling, while scanned documents and electronically generated documents have more regular content and more sparse labeling; some photographed document image tasks (such as document image correction and stable image enhancement) require pixel-level labeling, which is very difficult to label. The above difficulties have led to the current lack of labeled datasets for photographed document images. Summary of the Invention

[0003] In order to at least partially solve one of the technical problems existing in the prior art, the present invention aims to provide a method, system, device and medium for annotating photographed document image data.

[0004] The technical solution adopted in the present invention is:

[0005] A method for annotating photographed document image data, comprising the following steps:

[0006] Acquire a first document image that does not include geometric deformation degradation and illumination degradation;

[0007] printing the first document image into a paper document;

[0008] photographing the paper document to obtain a second document image;

[0009] Registering and aligning the first document image and the second document image to obtain an offset field;

[0010] Acquire annotation information of the second document image according to the offset field and the first document image; wherein the first document image carries annotation information;

[0011] and / or,

[0012] Sampling the second document image according to the offset field to obtain a third document image aligned with the first document image at the pixel level; wherein the third document image carries shadow information;

[0013] and / or,

[0014] The offset field and the second document image constitute document image correction task annotation data.

[0015] Furthermore, photographing the paper document to obtain a second document image includes:

[0016] After geometric deformation processing is performed on the paper document, photographing is performed to obtain the second document image; and / or,

[0017] The paper document is photographed at various shooting angles or under various lighting environments to obtain the second document image.

[0018] Furthermore, the offset field is a matrix with 2 channels, which is used to specify a 2-dimensional vector for each pixel position on the image to be offset, representing the direction and distance that the pixel should be offset;

[0019] The sampling process of the offset is expressed as follows:

[0020] I3=I1(x+f(x))

[0021] Among them, x is the pixel position, f is the offset field, and the final result I3 is the resulting image after sampling offset.

[0022] Furthermore, the acquiring annotation information of the second document image according to the offset field and the first document image includes:

[0023] offsetting the annotation information in the first document image according to the offset field, and mapping the offsetted annotation information into the second document image;

[0024] The second document image and the mapped annotation information constitute annotation data for the photographed document image layout analysis task, layout segmentation task, and table detection task.

[0025] Furthermore, in the document image layout analysis task, the annotation information of the second document image is obtained by:

[0026] Given the first document image, the annotation format is:

[0027] {X 1 ,X 2 ,…,X N}

[0028]

[0029] There are N labeled instances in total, and the labeled instances are X n Contains detection box annotations And the category label cn ,in and They are the coordinates of the upper left corner and the lower right corner respectively;

[0030] For the labeled instance X n , whose denser annotation is represented as:

[0031]

[0032]

[0033]

[0034]

[0035] Where S is a predefined constant;

[0036] Based on the offset field f, the annotation point (x, y) on the first document image is mapped to (x′, y′) on the second document image:

[0037] x′=x+f x (x,y)

[0038] y′=y+f y (x,y)

[0039] Among them, f x 、f y Respectively represent the offset values of the offset field in the x-direction and y-direction;

[0040] The annotation information of the second document image is obtained as follows:

[0041] {X 1 ′,X 2 ′,…,X N ′}

[0042]

[0043]

[0044]

[0045]

[0046] Finally, the annotation {X 1 ′,X 2 ′,…,X N ′}.

[0047] Furthermore, in the layout segmentation task, the annotation information of the second document image is obtained by:

[0048] Given the first document image, the annotation format is:

[0049] {X 1 ,X 2 ,…,X N}

[0050] X n =(m n ,c n )

[0051] Where m n is a binary mask with the same resolution as the first document image, c n is the category corresponding to the mask image;

[0052] Based on the offset field f, the binary mask m on the first document image is n Mapped onto the second document image:

[0053] m n ′=m n (i+f(i))

[0054] Where i is the pixel position;

[0055] The annotation information of the second document image is obtained as follows:

[0056] {X 1 ′,X 2 ′,…,X N ′}

[0057] X n ′=(m n ′,c n )

[0058] Finally, the annotation {X 1 ′,X 2 ′,…,X N ′}.

[0059] Furthermore, obtaining a third document image aligned with the first document image at the pixel level includes:

[0060] The first document image and the third document image constitute annotation data for a photographed document image enhancement task.

[0061] Another technical solution adopted in the present invention is:

[0062] A photographed document image data annotation system, comprising:

[0063] An image annotation module, configured to obtain a first document image free of geometric deformation degradation and illumination degradation;

[0064] An image printing module, configured to print the first document image into a paper document;

[0065] An image capturing module, configured to capture the paper document to obtain a second document image;

[0066] An image alignment module, configured to register and align the first document image and the second document image to obtain an offset field;

[0067] A data construction module, configured to obtain annotation information of the second document image according to the offset field and the first document image; wherein the first document image carries the annotation information;

[0068] and / or,

[0069] Sampling the second document image according to the offset field to obtain a third document image aligned with the first document image at the pixel level; wherein the third document image carries shadow information;

[0070] and / or,

[0071] The offset field and the second document image constitute document image correction task annotation data.

[0072] Another technical solution adopted in the present invention is:

[0073] A device for annotating image data of a photographed document, comprising:

[0074] at least one processor;

[0075] at least one memory for storing at least one program;

[0076] When the at least one program is executed by the at least one processor, the at least one processor implements the above method.

[0077] Another technical solution adopted in the present invention is:

[0078] A computer-readable storage medium stores a program executable by a processor, wherein the program executable by the processor is used to perform the method described above when executed by the processor.

[0079] The present invention provides the following beneficial effects: by mapping existing annotated electronically generated documents through registration and alignment, and performing offset sampling on photographed documents, it can cost-effectively obtain annotated data for tasks such as layout analysis, layout segmentation, table detection, image enhancement, and image correction of photographed document images. Compared to manual annotation directly on photographed document images, the present annotation method can significantly reduce the labor and time required for annotation. BRIEF DESCRIPTION OF THE DRAWINGS

[0080] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following introduction is made to the drawings of the embodiments of the present invention or the related technical solutions in the prior art. It should be understood that the drawings introduced below are only for the convenience of clearly describing some embodiments of the technical solutions of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without any creative work.

[0081] Figure 1 This is a first flow chart of a method for annotating photographed document image data in an embodiment of the present invention;

[0082] Figure 2 This is a second flow chart of a method for annotating photographed document image data in an embodiment of the present invention;

[0083] Figure 3 The part of the photographed document image layout analysis task annotation data obtained by annotation in an embodiment of the present invention;

[0084] Figure 4 This is the annotation data of part of the photographed document image enhancement task obtained by annotation in the embodiment of the present invention. DETAILED DESCRIPTION

[0085] The embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention and are not to be construed as limiting the present invention. The step numbers in the following embodiments are provided for ease of explanation only and do not limit the order of the steps. The order of execution of the steps in the embodiments can be adaptively adjusted according to the understanding of those skilled in the art.

[0086] In the description of the present invention, it should be understood that descriptions involving orientations, such as up, down, front, back, left, right, etc., indicating orientations or positional relationships, are based on the orientations or positional relationships shown in the accompanying drawings. They are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation. Therefore, they cannot be understood as limitations on the present invention.

[0087] In the description of the present invention, "several" means one or more, "many" means more than two, "greater than," "less than," and "exceed" are understood to exclude the number itself, while "above," "below," and "within" are understood to include the number itself. The use of "first" and "second" in the description is solely for the purpose of distinguishing technical features and should not be construed as indicating or implying relative importance, implicitly specifying the number of the indicated technical features, or implicitly specifying the order of the indicated technical features.

[0088] In the description of the present invention, unless otherwise clearly defined, terms such as setting, installing, and connecting should be understood in a broad sense, and technicians in the relevant technical field can reasonably determine the specific meanings of the above terms in the present invention based on the specific content of the technical solution.

[0089] Example 1

[0090] like Figure 1 As shown, this embodiment provides a method for annotating photographed document image data, comprising the following steps:

[0091] S101: Acquire a first document image with annotation information, wherein the first document image is an image without geometric deformation degradation or illumination degradation.

[0092] The first document image of step S101 can be collected from a large number of existing electronically generated documents or scanned document data sets, which already contain annotations for tasks such as layout analysis, layout segmentation, and table detection; it can also be obtained by using PDF conversion, flatbed scanner scanning, Internet collection, screenshots, etc., among which PDF document conversion and Internet collection can be automatically executed through web crawlers, etc., so that a large number of target documents can be obtained at low cost.

[0093] S102: Print the first document image into a paper document.

[0094] See also Figure 3 After printing the paper document in step S102, various geometric deformations such as twisting, folding, and wrinkling can be manually applied to the paper to increase data diversity and difficulty, so as to better simulate various geometric deformations of photographed document images in real scenes.

[0095] S103: Photograph the paper document to obtain a second document image.

[0096] See also Figure 3 In step S103, when obtaining the second document image, a variety of shooting angles, a variety of natural and artificial lighting environments, shadow occlusion, etc. can be used to make the second document image more diverse and better simulate the various shooting angles, lighting and shadow changes of the document image photographed in the real scene. Finally, the shooting is performed using various mobile devices with cameras, which can be mobile phones, tablet computers, cameras, etc.

[0097] S104: Register and align the first document image and the second document image to obtain an offset field.

[0098] When performing registration and alignment in step S4, a deep learning-based registration method, such as existing GLU-Net, DGC-Net, etc., can be used for registration, where the input is the first and second document images, and the output is the offset field.

[0099] S105: Acquire annotation information of the second document image according to the offset field and the first document image.

[0100] The annotation information in the first document image is offset according to the offset field, and the offset annotation information is mapped to the second document image; the second document image and the mapped annotation information constitute the annotation data for the photographed document image layout analysis task, layout segmentation task, and table detection task.

[0101] Example 2

[0102] like Figure 2 As shown, this embodiment provides a method for annotating photographed document image data, comprising the following steps:

[0103] S201: Acquire a first document image that does not contain geometric deformation degradation and illumination degradation.

[0104] S202: Print the first document image into a paper document.

[0105] S203: Photograph the paper document to obtain a second document image.

[0106] S204: Register and align the first document image and the second document image to obtain an offset field.

[0107] S205 . Sample the second document image according to the offset field to obtain a third document image aligned with the first document image at the pixel level; wherein the third document image carries shadow information.

[0108] See also Figure 4The offset field is used to sample the second document image to obtain a third document image. The third document image is aligned with the first document image to form data for the photographic document image enhancement task. Additionally, the obtained offset field and the second document image can form data for the photographic document image correction task. Figure 4 The annotated image in is the first document image, and the original image is the third document image.

[0109] The specific implementation of the above steps S201-S204 may refer to steps S101-S104 in the first embodiment.

[0110] Example 3

[0111] This embodiment provides a method for annotating photographed document image data, comprising the following steps:

[0112] S301: Acquire a first document image that does not contain geometric deformation degradation and illumination degradation.

[0113] S302: Print the first document image into a paper document.

[0114] S303: Photograph the paper document to obtain a second document image.

[0115] S304: Register and align the first document image and the second document image to obtain an offset field.

[0116] S305: Construct document image correction task annotation data from the offset field and the second document image.

[0117] As an optional implementation, the offset field is a matrix with a resolution equal to the size of the image to be offset and a channel count of 2. It specifies a 2D vector for each pixel position in the image to be offset, representing the direction and distance that the pixel should be offset. The offset sampling process is expressed as follows:

[0118] I3=I1(x+f(x))

[0119] Where x is the pixel position, f is the offset field, and the final result I3 is the resulting image after sampling offset.

[0120] As an optional implementation, in the above steps S104, S204 and S304, the first document image and the second document image are input into a trained neural network model to output an offset field. The following is a detailed description using DGC-Net as an example:

[0121] The first and second document images are scaled to a uniform resolution (240×240) and pixel values are normalized to the range [0,1]. The two images are fed into the VGG16 network to extract multi-scale deep features at resolutions of [15×15, 30×30, 60×60, 120×120, 240×240]. The similarity of each pixel of the level 0 deep feature (the minimum resolution feature) is calculated as follows: Where <,> represents the scalar product. Input to the convolutional neural network, and the regression output is the 0th level offset field w 0 After the 0th level offset field is upsampled by 2 times Level 1 deep features of the first document image To perform an offset: Where x represents the pixel position.

[0122] The first level depth feature of the first document image after the shift Level 1 deep features of the second document image After upsampling by 2 times, the 0th level offset field is spliced together in the channel dimension and input into the convolutional neural network, and the regression output is the 1st level offset field w 1 . After the 1st level offset field is sampled twice Level 2 deep features for the first document image To perform an offset: Where x represents the pixel position.

[0123] The second-level depth features of the first document image after the shift Level 2 deep features of the second document image After upsampling by 2 times, the first-level offset field is spliced together in the channel dimension and input into the convolutional neural network, and the regression output is the second-level offset field w 2 . After the second level offset field is sampled twice Level 3 deep features for the first document image To perform an offset: Where x represents the pixel position.

[0124] The third-level depth feature of the first document image after the shift Level 3 deep features of the second document image After upsampling by 2 times, the second-level offset field is spliced together in the channel dimension and input into the convolutional neural network, and the regression output is the third-level offset field w 3 After the 3rd level offset field is upsampled by 2 times Level 4 deep features of the first document image To perform an offset: Where x represents the pixel position.

[0125] The 4th level depth feature of the first document image after shifting Level 4 deep features of the second document image After upsampling by 2 times, the third-level offset field is spliced together in the channel dimension and input into the convolutional neural network, and the regression output is the fourth-level offset field w 4 . Level 4 offset field w 4 The desired final offset field can be obtained by upsampling to the original resolution of the first document image.

[0126] The above tasks are explained in detail below with reference to specific embodiments.

[0127] (1) Document image layout analysis task

[0128] Given a first document image, it needs to contain annotations for the layout analysis task. Its usual annotation format is:

[0129] {X 1 ,X 2 ,…,X N}

[0130]

[0131] There are N labeled instances in total, instance X n Contains detection box annotations And the category label c n ,in and They are the coordinates of the upper left corner and the lower right corner respectively.

[0132] Considering the deformation of the photographed document, the detection box annotation needs to be more dense, and only rectangular box annotation is not enough. Here we first convert the annotation box on the first document image into a more dense annotation:

[0133] For instance X n , whose denser annotation can be expressed as:

[0134]

[0135]

[0136]

[0137]

[0138] Where S is a predefined constant, which is set to 50 here. As a result, the detection box annotation points are expanded from the original two points to a more dense 4*S points.

[0139] Print the first document image and photograph it to obtain a second document image. After registering the first and second document images, we obtain the offset field f from the first to the second document image. The offset field f specifies a two-dimensional vector for each pixel position in the first document image, representing the direction and distance that pixel should be offset. That is, given a first document image I1 and a second document image I2, we can obtain a third document image I3 that is aligned with the second document image I2:

[0140] I3=I1(i+f(i))

[0141] Where i is the pixel position.

[0142] Based on the offset field f, the annotation point (x, y) on the first document image can be mapped to (x′, y′) on the second document image:

[0143] x ′ =x+f x (x,y)

[0144] y′=y+f y (x,y)

[0145] where f x f y Represents the offset values in the x and y directions of the offset field respectively.

[0146] Thus, the annotation of the second document image can be obtained:

[0147] {X 1′ ,X 2′ ,…,X N′}

[0148]

[0149]

[0150]

[0151]

[0152] In summary, we have obtained the annotations for the layout analysis task of the photographed document image (ie, the second document image I2) {X 1′ ,X 2′ ,…,X N′}.

[0153] (2) Document image layout segmentation task

[0154] Given a first document image, it needs to contain annotations for the layout segmentation task. Its usual annotation format is:

[0155] {X1 ,X 2 ,…,X N}

[0156] X n =(m n ,c n )

[0157] m n is a binary mask with the same resolution as the first document image, c n is the category corresponding to the mask image.

[0158] Print the first document image and photograph it to obtain a second document image. After registering the first and second document images, we obtain the offset field f from the first to the second document image. The offset field f specifies a two-dimensional vector for each pixel position in the first document image, representing the direction and distance that pixel should be offset. That is, given a first document image I1 and a second document image I2, we can obtain a third document image I3 that is aligned with the second document image I2:

[0159] I3=I1(i+f(i))

[0160] Where i is the pixel position.

[0161] Based on the offset field f, the binary mask m on the first document image can be n Mapped onto the second document image:

[0162] m n ′=m n (i+f(i))

[0163] Where i is the pixel position.

[0164] Thus, the annotation of the second document image can be obtained:

[0165] {X 1′ ,X 2 ′,…,X N′}

[0166] X n ′=(m n ′,c n )

[0167] In summary, we have obtained the annotations for the layout segmentation task of the photographed document image (i.e., the second document image I2) {X 1′ ,X 2′ ,…,X N′}.

[0168] (3) Table detection task

[0169] Similar to the document image layout analysis task, but the annotation instance here becomes a table, category c n There is only one type of table. Similarly, we can also get the annotation of the table detection task of the photographed document image (i.e. the second document image I2) {X 1′ ,X 2′ ,…,X N′}.

[0170] (4) Image enhancement

[0171] Print the first document image and photograph it to obtain a second document image. After registering the first and second document images, we obtain the offset field f from the first to the second document image. The offset field f specifies a two-dimensional vector for each pixel position in the first document image, representing the direction and distance that pixel should be offset. That is, given a first document image I1 and a second document image I2, we can obtain a third document image I3 that is aligned with the second document image I2:

[0172] I3=I1(i+f(i))

[0173] Where i is the pixel position.

[0174] In summary, we obtain the annotation I1 of the image enhancement task of the photographed document image (ie, the third document image I3).

[0175] (5) Image correction

[0176] Print the first document image and photograph it to obtain a second document image. After registering the first and second document images, we obtain the offset field f from the first to the second document image. The offset field f specifies a two-dimensional vector for each pixel position in the first document image, representing the direction and distance that pixel should be offset. That is, given a first document image I1 and a second document image I2, we can obtain a third document image I3 that is aligned with the second document image I2:

[0177] I3=I1(i+f(i))

[0178] Where i is the pixel position.

[0179] In summary, we obtain the annotation f of the image correction task of the photographed document image (ie, the second document image I2).

[0180] This embodiment also provides a photographed document image data annotation system, including:

[0181] An image annotation module, configured to obtain a first document image free of geometric deformation degradation and illumination degradation;

[0182] An image printing module, configured to print the first document image into a paper document;

[0183] An image capturing module, configured to capture the paper document to obtain a second document image;

[0184] An image alignment module, configured to register and align the first document image and the second document image to obtain an offset field;

[0185] A data construction module, configured to obtain annotation information of the second document image according to the offset field and the first document image; wherein the first document image carries the annotation information;

[0186] and / or,

[0187] Sampling the second document image according to the offset field to obtain a third document image aligned with the first document image at the pixel level; wherein the third document image carries shadow information;

[0188] and / or,

[0189] The offset field and the second document image constitute document image correction task annotation data.

[0190] A photographed document image data annotation system of this embodiment can execute a photographed document image data annotation method provided by the method embodiment of the present invention, can execute any combination of implementation steps of the method embodiment, and has the corresponding functions and beneficial effects of the method.

[0191] This embodiment also provides a device for annotating photographed document image data, comprising:

[0192] at least one processor;

[0193] at least one memory for storing at least one program;

[0194] When the at least one program is executed by the at least one processor, the at least one processor implements Figure 1 and / or Figure 2 The method shown.

[0195] A photographed document image data annotation device of this embodiment can execute a photographed document image data annotation method provided by the method embodiment of the present invention, can execute any combination of implementation steps of the method embodiment, and has the corresponding functions and beneficial effects of the method.

[0196] The present application also discloses a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device can read the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device performs Figure 1 and / or Figure 2 The method shown.

[0197] This embodiment also provides a storage medium that stores instructions or programs that can execute a method for annotating photographed document image data provided by an embodiment of the method of the present invention. When the instructions or program are run, any combination of implementation steps of the method embodiment can be executed, and the corresponding functions and beneficial effects of the method can be obtained.

[0198] In some optional embodiments, the function / operation mentioned in the block diagram may not occur in the order mentioned in the operation diagram. For example, depending on the function / operation involved, the two boxes shown in succession can actually be executed substantially simultaneously or the boxes can sometimes be executed in reverse order. In addition, the embodiment presented and described in the flow chart of the present invention is provided in an exemplary manner for the purpose of providing a more comprehensive understanding of the technology. The disclosed method is not limited to the operation and logic flow presented herein. Optional embodiments are contemplated in which the order of the various operations is changed and the sub-operations described as a part of a larger operation are performed independently.

[0199] Furthermore, although the present invention is described in the context of functional modules, it should be understood that, unless otherwise indicated, one or more of the functions and / or features described may be integrated into a single physical device and / or software module, or one or more functions and / or features may be implemented in separate physical devices or software modules. It will also be understood that a detailed discussion of the actual implementation of each module is not necessary for understanding the present invention. More specifically, given the properties, functions, and internal relationships of the various functional modules in the devices disclosed herein, the actual implementation of the module will be understood within the ordinary skill of an engineer. Therefore, a person skilled in the art using ordinary skill will be able to implement the present invention set forth in the claims without undue experimentation. It will also be understood that the specific concepts disclosed are merely illustrative and are not intended to limit the scope of the present invention, which is determined by the full scope of the appended claims and their equivalents.

[0200] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0201] The logic and / or steps represented in the flowcharts or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing the logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (e.g., a computer-based system, a system including a processor, or other system that can fetch and execute instructions from an instruction execution system, apparatus, or device). For purposes of this specification, a "computer-readable medium" can be any device that can contain, store, communicate, propagate, or transport a program for use by, or in conjunction with, an instruction execution system, apparatus, or device.

[0202] More specific examples (a non-exhaustive list) of computer-readable media include the following: an electrical connection with one or more wires (electronic devices), a portable computer disk cartridge (magnetic devices), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), a fiber optic device, and a portable compact disc read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, deciphering, or processing in another suitable manner as necessary, and then stored in a computer memory.

[0203] It should be understood that various parts of the present invention can be implemented using hardware, software, firmware, or a combination thereof. In the above-described embodiments, multiple steps or methods can be implemented using software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented using hardware, as in another embodiment, any one of the following technologies known in the art or a combination thereof can be used: a discrete logic circuit having a logic gate circuit for implementing a logic function on a data signal, an application-specific integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0204] In the above description of this specification, reference to the terms "one embodiment / example," "another embodiment / example," or "certain embodiments / examples" means that the specific features, structures, materials, or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples.

[0205] While embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions, and variations may be made to the embodiments without departing from the principles and spirit of the invention, and that the scope of the invention is defined by the claims and their equivalents.

[0206] The above is a specific description of the preferred implementation of the present invention, but the present invention is not limited to the above embodiments. Those skilled in the art can make various equivalent modifications or substitutions without violating the spirit of the present invention. These equivalent modifications or substitutions are all included in the scope defined by the claims of this application.

Claims

1. A method for annotating photographed document image data, characterized in that: The following steps are involved: Acquire a first document image that does not include geometric deformation degradation and illumination degradation; printing the first document image into a paper document; photographing the paper document to obtain a second document image; Registering and aligning the first document image and the second document image to obtain an offset field; Acquire annotation information of the second document image according to the offset field and the first document image; wherein the first document image carries annotation information; and / or, Sampling the second document image according to the offset field to obtain a third document image aligned with the first document image at the pixel level; wherein the third document image carries shadow information; and / or, The offset field and the second document image constitute document image correction task annotation data; The acquiring annotation information of the second document image according to the offset field and the first document image includes: offsetting the annotation information in the first document image according to the offset field, and mapping the offsetted annotation information into the second document image; The second document image and the mapped annotation information constitute annotation data for the photographed document image layout analysis task, layout segmentation task, and table detection task; In the document image layout analysis task, the annotation information of the second document image is obtained by: Given the first document image, the annotation format is: {X 1 ,X 2 ,…,X N } There are N labeled instances in total, and the labeled instances are X n Contains detection box annotations And the category label c n ,in and They are the coordinates of the upper left corner and the lower right corner respectively; For the labeled instance X n , whose denser annotation is represented as: Where S is a predefined constant; Based on the offset field f, the annotation point (x, y) on the first document image is mapped to (x′, y′) on the second document image: x ′ =x+f x (x,y) y′=y+f y (x,y) Among them, f x 、f y Respectively represent the offset values of the offset field in the x-direction and y-direction; The annotation information of the second document image is obtained as follows: {X 1′ ,X 2′ ,…,X N′ } Finally, the annotation {X 1′ ,X 2′ ,…,X N′ }.

2. A method for labeling photographed document image data according to claim 1, characterized in that: The photographing of the paper document to obtain a second document image includes: After geometric deformation processing is performed on the paper document, the paper document is photographed to obtain the second document image; and / or the paper document is photographed at multiple shooting angles or under multiple lighting environments to obtain the second document image.

3. The method for labeling photographed document image data according to claim 1, characterized in that: The offset field is a matrix with 2 channels, which is used to specify a 2-dimensional vector for each pixel position on the image to be offset, representing the direction and distance that the pixel should be offset; The sampling process of the offset is expressed as follows: I3=I1(x+f(x)) Among them, x is the pixel position, f is the offset field, and the final result I3 is the resulting image after sampling offset.

4. The method for labeling photographed document image data according to claim 1, wherein: In the layout segmentation task, the annotation information of the second document image is obtained by: Given the first document image, the annotation format is: {X 1 ,X 2 ,…,X N } X n =(m n ,c n ) Where m n is a binary mask with the same resolution as the first document image, c n is the category corresponding to the mask image; Based on the offset field f, the binary mask m on the first document image is n Mapped onto the second document image: m n ′=m n (i+f(i)) Where i is the pixel position; The annotation information of the second document image is obtained as follows: {X 1′ ,X 2 ′,…,X N′ } X n ′=(m n ′,c n ) Finally, the annotation of the layout segmentation task of the second document image I2 is obtained {X 1′ ,X 2′ ,…,X N′ }.

5. The method for labeling photographed document image data according to claim 1, characterized in that: The obtaining of a third document image aligned with the first document image at the pixel level includes: The first document image and the third document image constitute annotation data for a photographed document image enhancement task.

6. A photographed document image data annotation system, applied to the photographed document image data annotation method according to any one of claims 1 to 5, characterized in that: include: An image annotation module, configured to obtain a first document image free of geometric deformation degradation and illumination degradation; An image printing module, configured to print the first document image into a paper document; An image capturing module, configured to capture the paper document to obtain a second document image; An image alignment module, configured to register and align the first document image and the second document image to obtain an offset field; A data construction module, configured to obtain annotation information of the second document image according to the offset field and the first document image; wherein the first document image carries the annotation information; and / or, Sampling the second document image according to the offset field to obtain a third document image aligned with the first document image at the pixel level; wherein the third document image carries shadow information; and / or, The offset field and the second document image constitute document image correction task annotation data.

7. A device for annotating photographed document image data, characterized in that: include: at least one processor; at least one memory for storing at least one program; When the at least one program is executed by the at least one processor, the at least one processor implements the method according to any one of claims 1 to 5.

8. A computer-readable storage medium storing a program executable by a processor, characterized in that: The processor-executable program is used to perform the method according to any one of claims 1 to 5 when executed by the processor.

Images