Document rotation direction identification method, system and equipment and storage medium

Abstract

The invention discloses a document rotation direction identification method, system and device and a storage medium, and the method comprises the steps: carrying out the equal-proportion zooming of a to-be-predicted document image according to a preset size, and obtaining a target document image; performing image block cutting on the target document image by adopting each preset cutting scale to obtain a target image block corresponding to each preset cutting scale; according to the input resolution of the multi-task convolutional neural network model, zooming the target image block corresponding to each preset cutting scale, and inputting a zooming result into the multi-task convolutional neural network model to obtain a classification result; and voting based on the classification result, and determining a direction classification result of the to-be-predicted document image according to the voting result. According to the method, the classification result is obtained through multi-scale image processing and the multi-task convolutional neural network model, and finally voting is performed based on the classification result, so that compared with the prior art, the accuracy of document rotation direction identification is effectively improved.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a document rotation direction recognition method, system, device and storage medium. Background Art

[0002] With the acceleration of digital transformation, a large number of documents need to be converted from physical form to electronic form for storage, retrieval and analysis. In this process, due to different shooting angles and scanning devices, the document image may be rotated, making it difficult to read the document and extract information. In order to improve the readability of the document and the accuracy of information extraction, it is very necessary to accurately classify the rotation direction of the document.

[0003] In recent years, artificial intelligence and image processing technologies have developed rapidly. Some researchers have studied the problem of document rotation direction and proposed some solutions, such as using edge detection and corner detection in traditional image processing methods to estimate the direction of the document, or using convolutional neural networks to directly identify the direction of the document. However, in actual application, the above methods not only have the problem of high computational time consumption, but also have the problems of insufficient generalization ability and poor anti-interference ability, which leads to low accuracy in document rotation direction recognition.

[0004] Therefore, there is an urgent need for a document rotation direction recognition method that can effectively improve the accuracy of document rotation direction recognition. Summary of the invention

[0005] The main purpose of the present invention is to provide a document rotation direction recognition method, system, device and storage medium, aiming to solve the technical problem of low accuracy in document rotation direction recognition in the prior art.

[0006] To achieve the above object, the present invention provides a method for identifying a document rotation direction, the method comprising the following steps:

[0007] Proportionally scaling the predicted document image according to a preset size to obtain a target document image;

[0008] Using each preset cropping scale to crop the target document image, respectively, to obtain the target image block corresponding to each preset cropping scale;

[0009] According to the input resolution of the multi-task convolutional neural network model, scaling the target image blocks corresponding to each of the preset cropping scales, and inputting the scaling results into the multi-task convolutional neural network model to obtain a classification result;

[0010] Voting is performed based on the classification results, and the direction classification result of the document image to be predicted is determined according to the voting results.

[0011] Optionally, before the step of scaling the document image to be predicted according to a preset size to obtain a target document image, the step further includes:

[0012] Acquire image blocks corresponding to a plurality of preset cropping scales, and generate corresponding region two-category labels and document rotation direction four-category labels based on the image blocks corresponding to the preset cropping scales;

[0013] The image blocks corresponding to the preset cropping scales, the region binary classification labels corresponding to the image blocks, and the document rotation direction four-classification labels are used as training data sets;

[0014] Constructing a convolutional neural network model, wherein the convolutional neural network model includes two classification heads, and the two classification heads are used to output a region classification result and a document rotation direction classification result respectively;

[0015] The convolutional neural network model is trained based on the training data set and the preset loss function, and the parameters of the convolutional neural network model are optimized according to the training results to obtain a multi-task convolutional neural network model.

[0016] Optionally, the step of acquiring image blocks corresponding to a plurality of preset cropping scales, and generating corresponding region two-category labels and document rotation direction four-category labels based on the image blocks corresponding to the preset cropping scales, comprises:

[0017] Collecting document images, and correcting the rotation direction of each of the document images to obtain a corrected document image;

[0018] Proportionally scaling the corrected document image according to a preset size, and annotating a text area on the proportionally scaled document image to obtain an annotated document image;

[0019] Determine a plurality of preset cropping scales, and respectively use the preset cropping scales to crop image blocks of the annotated document image to obtain image blocks corresponding to the preset cropping scales;

[0020] Based on the image blocks corresponding to the preset cropping scales, corresponding region two-category labels and document rotation direction four-category labels are generated.

[0021] Optionally, the step of generating corresponding region two-category labels and document rotation direction four-category labels based on the image blocks corresponding to the preset cropping scales includes:

[0022] Determine the type of each image block based on the area of ​​the text region in the image block corresponding to each preset cropping scale, and obtain a type determination result;

[0023] Generate a regional binary classification label corresponding to each of the image blocks according to the type judgment result;

[0024] Generate four-category labels of document rotation directions corresponding to each image block according to the correction standard of the corrected document image.

[0025] Optionally, the step of scaling the target image blocks corresponding to the preset cropping scales according to the input resolution of the multi-task convolutional neural network model, and inputting the scaling results into the multi-task convolutional neural network model to obtain the classification results includes:

[0026] According to the input resolution of the multi-task convolutional neural network model, scaling the target image blocks corresponding to each of the preset cropping scales to obtain a scaling result;

[0027] Inputting the scaling result into the multi-task convolutional neural network model to obtain the region binary classification result of each target image block and the document rotation direction four-classification result;

[0028] The region binary classification result of each target image block and the document rotation direction quadrature classification result are used as the classification result of each target image block.

[0029] Optionally, the step of voting based on the classification result and determining the direction classification result of the document image to be predicted according to the voting result includes:

[0030] Filtering the background image blocks in the target image blocks based on the regional binary classification results of each of the target image blocks to obtain filtered target image blocks;

[0031] Voting on the classification results corresponding to the filtered target image blocks to obtain a first voting result, and determining an initial classification result corresponding to each of the preset cropping scales based on the first voting result;

[0032] Voting is performed on the initial classification results corresponding to the preset cropping scales to obtain a second voting result, and the direction classification result of the document image to be predicted is determined based on the second voting result.

[0033] Optionally, after the step of using each preset cropping scale to crop the target document image into image blocks respectively to obtain the target image blocks corresponding to each preset cropping scale, the step further includes:

[0034] According to a preset downsampling ratio, the target image blocks corresponding to the preset cropping scales are downsampled at intervals to obtain the sampled target image blocks corresponding to the preset cropping scales.

[0035] In addition, to achieve the above object, the present invention also proposes a document rotation direction recognition system, the system comprising:

[0036] An image scaling module is used to scale the predicted document image in proportion to a preset size to obtain a target document image;

[0037] An image cropping module, used to crop the target document image into image blocks using each preset cropping scale to obtain target image blocks corresponding to each preset cropping scale;

[0038] A model reasoning module, used to scale the target image blocks corresponding to each of the preset cropping scales according to the input resolution of the multi-task convolutional neural network model, and input the scaling results into the multi-task convolutional neural network model to obtain a classification result;

[0039] The result voting module is used to vote based on the classification result and determine the direction classification result of the document image to be predicted according to the voting result.

[0040] In addition, to achieve the above-mentioned purpose, the present invention also proposes a document rotation direction recognition device, which includes: a memory, a processor, and a document rotation direction recognition program stored in the memory and executable on the processor, wherein the document rotation direction recognition program is configured to implement the steps of the document rotation direction recognition method described above.

[0041] In addition, to achieve the above-mentioned purpose, the present invention further proposes a storage medium, on which a document rotation direction recognition program is stored, and when the document rotation direction recognition program is executed by a processor, the steps of the document rotation direction recognition method described above are implemented.

[0042] The present invention discloses scaling a document image to be predicted in proportion to a preset size to obtain a target document image; using each preset cropping scale to crop the target document image in image blocks to obtain target image blocks corresponding to each preset cropping scale; scaling the target image blocks corresponding to each preset cropping scale according to the input resolution of a multi-task convolutional neural network model, and inputting the scaling result into the multi-task convolutional neural network model to obtain a classification result; voting based on the classification result, and determining the direction classification result of the document image to be predicted according to the voting result. Since the present invention uses each preset cropping scale to crop the target document image in image blocks, and then obtains the classification result through the multi-task convolutional neural network model, and finally voting based on the classification result to determine the direction classification result of the document image to be predicted, compared with the prior art, the present invention effectively improves the accuracy of document rotation direction recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 It is a flowchart of a first embodiment of a method for identifying a document rotation direction according to the present invention;

[0044] Figure 2 It is a flow chart of a second embodiment of the method for identifying the document rotation direction of the present invention;

[0045] Figure 3 This is a structural example diagram of a convolutional neural network model in the document rotation direction recognition method of the present invention;

[0046] Figure 4 It is a schematic flow chart of a third embodiment of a method for identifying a document rotation direction according to the present invention;

[0047] Figure 5 It is a structural block diagram of a first embodiment of a document rotation direction recognition system of the present invention;

[0048] Figure 6 It is a structural schematic diagram of a document rotation direction recognition device in a hardware operating environment involved in an embodiment of the present invention.

[0049] The realization of the purpose, functional features and advantages of the present invention will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0050] It should be understood that the specific embodiments described herein are only used to explain the present invention, and are not used to limit the present invention.

[0051] The embodiment of the present invention provides a method for identifying the rotation direction of a document. Figure 1 , Figure 1 FIG. 4 is a flow chart of a first embodiment of a method for identifying a document rotation direction according to the present invention.

[0052] In this embodiment, the document rotation direction identification method includes steps S10 to S40:

[0053] Step S10: scaling the document image to be predicted according to a preset size to obtain a target document image.

[0054] It should be noted that the execution subject of this embodiment can be a computer server device with data processing, network communication and program running functions used in the document rotation direction recognition scenario, such as a server, a tablet computer, a smart phone, a smart watch, etc., or an electronic device capable of realizing the above functions, a document rotation direction recognition device, etc. The following takes the document rotation direction recognition device as an example to illustrate this embodiment and the following embodiments.

[0055] It should be understood that proportional scaling means that when adjusting the size of an image, graphic or object, the ratio of its width and height remains unchanged, so as to ensure that the shape of the image or object will not be deformed or distorted after scaling. The above-mentioned preset size can be a custom setting, and this embodiment does not limit this.

[0056] For example, assuming the preset size is 896×896, the document image to be predicted is scaled to 896×896 in proportion to the aspect ratio. First, the longer side is scaled to 896, and the shorter side is scaled to 896 in proportion, and then filled with 0 pixels, so that the size of the document image to be predicted becomes 896×896, and the target document image is obtained.

[0057] Step S20: using each preset cropping scale to crop the target document image respectively to obtain the target image block corresponding to each preset cropping scale.

[0058] It should be understood that the above-mentioned preset cropping scales may be customized by a system user, and this embodiment does not limit this.

[0059] For example, three preset cropping scales of 56, 112, and 224 are used to crop image blocks of the target document image. The specific cropping method can be sliding window cropping from left to right and from top to bottom. There is no overlap between the image blocks. After cropping, the 56 scale can obtain (896÷56)×(896÷56)=256 target image blocks. Similarly, 112 and 224 scales are cropped to obtain 64 and 16 target image blocks respectively.

[0060] It should be noted that the multi-scale image processing method can effectively improve the generalization performance and anti-interference ability of subsequent document rotation direction recognition, thereby improving the accuracy of document rotation direction recognition.

[0061] In a specific implementation, in order to reduce the amount of calculation and the time spent on calculation, after the step of using each preset cropping scale to perform image block cropping on the target document image respectively and obtaining the target image blocks corresponding to each preset cropping scale, it also includes: performing interval downsampling on the target image blocks corresponding to each preset cropping scale according to a preset downsampling ratio to obtain the target image blocks corresponding to each preset cropping scale after sampling.

[0062] It should be understood that the above-mentioned preset downsampling ratio may be a user-defined setting of the system, and this embodiment does not impose any limitation on this.

[0063] For example, for interval downsampling, a 50% downsampling ratio is used to downsample the image blocks at equal intervals. The sampling order is from left to right, and the image blocks in each scale are sorted from top to bottom. The 1st, 3rd, 5th... image blocks are retained at equal intervals, and the 2nd, 4th, 6th... image blocks are discarded. Finally, 128, 32, and 8 target image blocks are obtained at the three preset cropping scales of 56, 112, and 224, respectively, thereby reducing the amount of calculation by 50%.

[0064] Step S30: According to the input resolution of the multi-task convolutional neural network model, the target image blocks corresponding to each of the preset cropping scales are scaled, and the scaling results are input into the multi-task convolutional neural network model to obtain classification results.

[0065] It should be noted that the above-mentioned multi-task convolutional neural network model includes two classification heads, and the two classification heads are used to output region classification results and document rotation direction classification results respectively.

[0066] It should be explained that the above-mentioned area classification results include the target image block being the background or foreground; the document rotation direction classification results include the target image block being a 0-degree image, a counterclockwise rotated 90-degree image, a counterclockwise rotated 180-degree image, or a counterclockwise rotated 270-degree image.

[0067] It should be noted that in this embodiment, the target image block can be distinguished as a background image block or a foreground image block by whether the area of ​​the text region in the target image block exceeds 10%. For example, when the area of ​​the text region in the target image block exceeds 10%, the target image block is a foreground image block, otherwise it is a background image block.

[0068] In a specific implementation, the target image blocks corresponding to each preset cropping scale can be scaled according to the input resolution of the multi-task convolutional neural network model to obtain the scaling results; the scaling results are input into the multi-task convolutional neural network model to obtain the regional binary classification results (i.e., regional classification results) and the document rotation direction four-classification results (i.e., document rotation direction classification results) of each target image block; the regional binary classification results and the document rotation direction four-classification results of each target image block are used as the classification results of each target image block.

[0069] For example, if the input resolution of the task convolutional neural network model is 56×56, 168 (128+32+8) target image blocks are scaled to 56×56 and assembled into a batch of data, which are input into the multi-task convolutional neural network model for inference. For the preset cropping scale of 56, 128 region classification results and 128 document rotation direction classification results can be obtained, for the preset cropping scale of 112, 32 region classification results and 32 document rotation direction classification results can be obtained, and for the preset cropping scale of 224, 8 region classification results and 8 document rotation direction classification results can be obtained.

[0070] Step S40: Voting is performed based on the classification results, and the direction classification result of the document image to be predicted is determined according to the voting results.

[0071] It should be noted that, the classification results corresponding to the target image block can be voted to obtain a first voting result, and the initial classification results corresponding to each of the preset cropping scales can be determined based on the first voting result; the initial classification results corresponding to each of the preset cropping scales can be voted to obtain a second voting result, and the directional classification result of the document image to be predicted can be determined based on the second voting result.

[0072] For example, multiple classification results obtained within the three preset cropping scales of 56, 112, and 224 are voted within the scale to obtain the document rotation direction results corresponding to the three scales; then the document rotation direction results corresponding to the three scales are voted, and finally the document rotation direction results with more votes and higher probability are output to determine the direction classification result of the document image to be predicted.

[0073] The present embodiment discloses scaling the document image to be predicted in proportion to a preset size to obtain a target document image; using each preset cropping scale to crop the target document image in image blocks to obtain target image blocks corresponding to each preset cropping scale; scaling the target image blocks corresponding to each preset cropping scale according to the input resolution of the multi-task convolutional neural network model, and inputting the scaling result into the multi-task convolutional neural network model to obtain a classification result; voting based on the classification result, and determining the direction classification result of the document image to be predicted based on the voting result. Since the present embodiment uses each preset cropping scale to crop the target document image in image blocks, and then obtains the classification result through the multi-task convolutional neural network model, and finally voting based on the classification result to determine the direction classification result of the document image to be predicted, compared with the prior art, the present embodiment effectively improves the accuracy of document rotation direction recognition.

[0074] refer to Figure 2 , Figure 2 FIG. 4 is a flow chart of a second embodiment of a method for identifying a document rotation direction according to the present invention.

[0075] Based on the first embodiment described above, in this embodiment, before step S10, steps S01 to S04 are included:

[0076] Step S01: acquiring image blocks corresponding to a plurality of preset cropping scales, and generating corresponding region two-category labels and document rotation direction four-category labels based on the image blocks corresponding to the preset cropping scales.

[0077] It should be understood that the above-mentioned two-category region label is a label including the region classification result, and the four-category document rotation direction label is a label including the document rotation direction classification result.

[0078] In a specific implementation, document images can be collected, and the rotation direction of each document image can be corrected to obtain a corrected document image; the corrected document image can be scaled proportionally according to a preset size, and the proportionally scaled document image can be annotated with text areas to obtain annotated document images; multiple preset cropping scales can be determined, and each of the preset cropping scales can be used to crop image blocks of the annotated document image to obtain image blocks corresponding to each of the preset cropping scales; based on the image blocks corresponding to each of the preset cropping scales, corresponding area two-category labels and document rotation direction four-category labels can be generated.

[0079] It should be noted that if the rotation direction of each of the document images is corrected and uniformly corrected to 0 degrees to obtain the corrected document image, the image blocks corresponding to each of the preset cropping scales generate corresponding four-category labels of the document rotation direction, all of which indicate that the image blocks corresponding to each of the preset cropping scales are 0-degree images.

[0080] It should be explained that the step of generating corresponding regional two-category labels and document rotation direction four-category labels based on the image blocks corresponding to each of the preset cropping scales includes: judging the type of each of the image blocks based on the area of ​​the text area in the image blocks corresponding to each of the preset cropping scales to obtain a type judgment result; generating a regional two-category label corresponding to each of the image blocks according to the type judgment result; and generating a document rotation direction four-category label corresponding to each of the image blocks according to the correction standard of the corrected document image.

[0081] It should be understood that the above type determination result includes whether the type of the image block is a background image or a foreground image.

[0082] Step S02: using the image blocks corresponding to the preset cropping scales, the region two-category labels corresponding to the image blocks, and the document rotation direction four-category labels as training data sets.

[0083] Step S03: construct a convolutional neural network model, wherein the convolutional neural network model includes two classification heads, and the two classification heads are used to output region classification results and document rotation direction classification results respectively.

[0084] For example, refer to Figure 3 , Figure 3This is a structural example diagram of the convolutional neural network model in the document rotation direction recognition method of the present invention. In the figure, each cuboid represents a layer in the network, and the labeled number represents the size of the layer. The following is a specific description of each part of the figure: 28×28×3: This is the input layer, indicating that the size of the input image is 28×28 pixels, and 3 represents the three RGB color channels. 24×24×8: This is the first convolution layer. After the convolution operation, the image size becomes 24×24, and the number of channels increases to 8. 24×24×8: This is the first pooling layer, usually a maximum pooling layer, the size remains unchanged, but the number of channels may remain unchanged or change. 12×12×8: In the second convolution layer, the image size is halved to 12×12, and the number of channels may remain unchanged or change. 10×10×16: The second pooling layer, the size becomes 10×10, and the number of channels increases to 16. 10×10×16: The third convolutional layer, the size remains unchanged, and the number of channels increases to 16. 4×4×16: The third pooling layer, the size is halved to 4×4, and the number of channels may remain unchanged. 1×1×10: It is a fully connected layer. 1×1×2: A fully connected layer that flattens the feature map and connects it to two output nodes, which can be used for two-classification tasks. 1×1×4: A fully connected layer that flattens the feature map and connects it to four output nodes, which can be used for four-classification tasks. This embodiment and the following embodiments are described with an input resolution of 56×56 for the multi-task convolutional neural network model.

[0085] Step S04: training the convolutional neural network model based on the training data set and the preset loss function, and optimizing the parameters of the convolutional neural network model according to the training results to obtain a multi-task convolutional neural network model.

[0086] It should be noted that before model training, it is also necessary to uniformly scale the image blocks corresponding to the preset cropping scales in the training data set to 56×56 size according to the input resolution of the convolutional neural network model to adapt to the input resolution requirements of the convolutional neural network model, while reducing the amount of calculation.

[0087] In the specific implementation, the cross entropy loss function is used for loss calculation for both the region binary classification task and the document rotation direction four-classification task. In particular, for the loss calculation of the document rotation direction four-classification task, a filtering operation is required, that is, the document direction four-classification loss calculation is performed only on the foreground image blocks, and the background image blocks are ignored, thereby improving the calculation efficiency.

[0088] It should be explained that the above-mentioned preset loss function includes a region two-classification loss function and a document rotation direction four-classification loss function.

[0089] The regional binary classification loss function is:

[0090]

[0091] Where N is the number of samples and C is the number of categories. ij is the jth element (0 or 1) in the true label vector of the i-th sample. ij is the model's predicted probability that the i-th sample belongs to the j-th category.

[0092] The four-category loss function for document rotation direction is:

[0093]

[0094] Where M is the number of foreground samples, F is the foreground sample index set, and C is the number of categories. ij is the jth element (0 or 1) in the true label vector of the i-th sample. ij is the model's predicted probability that the i-th sample belongs to the j-th category.

[0095] The default loss function is:

[0096] L total =L 2_cls +L 4_cls .

[0097] The present embodiment discloses obtaining image blocks corresponding to a plurality of preset cropping scales, and generating corresponding regional two-classification labels and document rotation direction four-classification labels based on the image blocks corresponding to each of the preset cropping scales; using the image blocks corresponding to each of the preset cropping scales and the regional two-classification labels and document rotation direction four-classification labels corresponding to each of the image blocks as training data sets; constructing a convolutional neural network model, the convolutional neural network model including two classification heads, the two classification heads being used to output regional classification results and document rotation direction classification results respectively; training the convolutional neural network model based on the training data set and a preset loss function, and optimizing the parameters of the convolutional neural network model according to the training results to obtain a multi-task convolutional neural network model. The present embodiment obtains a multi-task convolutional neural network model by constructing a convolutional neural network model and a training data set, and training the convolutional neural network model by the training data set and a preset loss function. Compared with the prior art, the present embodiment effectively improves the accuracy of the multi-task convolutional neural network model in identifying the document rotation direction.

[0098] refer to Figure 4 , Figure 4 FIG. 4 is a flow chart of a third embodiment of a method for identifying a document rotation direction according to the present invention.

[0099] Based on the above embodiments, in this embodiment, step S40 includes steps S401 to S402:

[0100] Step S401: filtering background image blocks in the target image blocks based on the regional binary classification results of each target image block to obtain a filtered target image block.

[0101] Step S402: voting on the classification results corresponding to the filtered target image blocks to obtain a first voting result, and determining initial classification results corresponding to each of the preset cropping scales based on the first voting result.

[0102] Step S403: voting on the initial classification results corresponding to the preset cropping scales to obtain a second voting result, and determining the direction classification result of the document image to be predicted based on the second voting result.

[0103] It should be noted that before voting, the background image blocks in the target image blocks are filtered based on the regional binary classification results of each target image block to obtain the filtered target image blocks, which can effectively ensure the accuracy of document rotation direction classification and further enhance the generalization performance and anti-interference ability of document rotation direction recognition.

[0104] This embodiment discloses filtering the background image blocks in the target image blocks based on the regional binary classification results of each target image block to obtain the filtered target image blocks; voting on the classification results corresponding to the filtered target image blocks to obtain a first voting result, and determining the initial classification results corresponding to each preset cropping scale based on the first voting result; voting on the initial classification results corresponding to each preset cropping scale to obtain a second voting result, and determining the direction classification result of the document image to be predicted based on the second voting result. Compared with the prior art, this embodiment further enhances the generalization performance and anti-interference ability of document rotation direction recognition.

[0105] In addition, an embodiment of the present invention further provides a storage medium, on which a document rotation direction recognition program is stored. When the document rotation direction recognition program is executed by a processor, the steps of the document rotation direction recognition method described above are implemented.

[0106] Reference Figure 5 , Figure 5 It is a structural block diagram of the first embodiment of the document rotation direction recognition system of the present invention.

[0107] like Figure 5 As shown, the document rotation direction recognition system proposed in the embodiment of the present invention includes: an image scaling module 501, an image cropping module 502, a model reasoning module 503 and a result voting module 504.

[0108] The image scaling module 501 is used to scale the document image to be predicted in proportion to a preset size to obtain a target document image.

[0109] The image cropping module 502 is used to crop the target document image with each preset cropping scale to obtain the target image block corresponding to each preset cropping scale.

[0110] The model inference module 503 is used to scale the target image blocks corresponding to each preset cropping scale according to the input resolution of the multi-task convolutional neural network model, and input the scaling result into the multi-task convolutional neural network model to obtain a classification result.

[0111] The result voting module 504 is used to vote based on the classification result, and determine the direction classification result of the document image to be predicted according to the voting result.

[0112] The model inference module 503 is also used to scale the target image blocks corresponding to each preset cropping scale according to the input resolution of the multi-task convolutional neural network model to obtain a scaling result; input the scaling result into the multi-task convolutional neural network model to obtain a regional binary classification result and a document rotation direction four-classification result of each target image block; and use the regional binary classification result and the document rotation direction four-classification result of each target image block as the classification result of each target image block.

[0113] The image cropping module 502 is further configured to perform interval down-sampling on the target image blocks corresponding to the preset cropping scales according to a preset down-sampling ratio, so as to obtain the sampled target image blocks corresponding to the preset cropping scales.

[0114] The embodiment of the system discloses scaling the document image to be predicted in proportion to a preset size to obtain a target document image; using each preset cropping scale to crop the image block of the target document image to obtain the target image block corresponding to each preset cropping scale; scaling the target image block corresponding to each preset cropping scale according to the input resolution of the multi-task convolutional neural network model, and inputting the scaling result into the multi-task convolutional neural network model to obtain a classification result; voting based on the classification result, and determining the direction classification result of the document image to be predicted according to the voting result. Since the embodiment of the system uses each preset cropping scale to crop the image block of the target document image, and then obtains the classification result through the multi-task convolutional neural network model, and finally voting based on the classification result to determine the direction classification result of the document image to be predicted, compared with the prior art, the embodiment of the system effectively improves the accuracy of document rotation direction recognition.

[0115] Based on the first embodiment of the document rotation direction recognition system of the present invention, a second embodiment of the document rotation direction recognition system of the present invention is proposed.

[0116] In this embodiment, the image scaling module 501 is also used to obtain image blocks corresponding to multiple preset cropping scales, and generate corresponding regional two-category labels and document rotation direction four-category labels based on the image blocks corresponding to each preset cropping scale; use the image blocks corresponding to each preset cropping scale and the regional two-category labels and document rotation direction four-category labels corresponding to each image block as training data sets; construct a convolutional neural network model, the convolutional neural network model includes two classification heads, and the two classification heads are used to output regional classification results and document rotation direction classification results, respectively; train the convolutional neural network model based on the training data set and a preset loss function, and optimize the parameters of the convolutional neural network model according to the training results to obtain a multi-task convolutional neural network model.

[0117] The image scaling module 501 is also used to collect document images, and correct the rotation direction of each of the document images to obtain a corrected document image; scale the corrected document image proportionally according to a preset size, and annotate the proportionally scaled document image with text areas to obtain annotated document images; determine multiple preset cropping scales, and use each of the preset cropping scales to crop image blocks of the annotated document image to obtain image blocks corresponding to each of the preset cropping scales; and generate corresponding area two-category labels and document rotation direction four-category labels based on the image blocks corresponding to each of the preset cropping scales.

[0118] The image scaling module 501 is further used to determine the type of each image block based on the area of ​​the text region in the image block corresponding to each preset cropping scale to obtain a type determination result; generate a two-category label of the region corresponding to each image block according to the type determination result; and generate a four-category label of the document rotation direction corresponding to each image block according to the correction standard of the corrected document image.

[0119] Other embodiments or specific implementations of the document rotation direction recognition system of the present invention may refer to the above-mentioned method embodiments, which will not be described in detail here.

[0120] The present application provides a document rotation direction recognition device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the document rotation direction recognition method in the above-mentioned embodiment one.

[0121] Reference below Figure 6, which shows a schematic diagram of the structure of a document rotation direction recognition device suitable for implementing the embodiment of the present application. The document rotation direction recognition device in the embodiment of the present application may include but is not limited to mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Descriptions), PMPs (Portable Media Players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 6 The document rotation direction identification device shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0122] like Figure 6 As shown, the document rotation direction recognition device may include a processing device 1001 (e.g., a central processing unit, a graphics processor, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM: Read Only Memory) 1002 or a program loaded from a storage device 1003 to a random access memory (RAM: Random Access Memory) 1004. In RAM1004, various programs and data required for the operation of the document rotation direction recognition device are also stored. The processing device 1001, ROM1002, and RAM1004 are connected to each other through a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Generally, the following systems can be connected to the I / O interface 1006: an input device 1007 including, for example, a touch screen, a touch pad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; an output device 1008 including, for example, a liquid crystal display (LCD: Liquid Crystal Display), a speaker, a vibrator, etc.; a storage device 1003 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 1009. The communication device 1009 can allow the document rotation direction recognition device to communicate with other devices wirelessly or by wire to exchange data. Although the document rotation direction recognition device with various systems is shown in the figure, it should be understood that it is not required to implement or have all the systems shown. More or fewer systems can be implemented or have alternatively.

[0123] In particular, according to the embodiments disclosed in the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present application include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device, or installed from a storage device 1003, or installed from a ROM 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the method of the embodiment disclosed in the present application are executed.

[0124] The document rotation direction recognition device provided by the present application adopts the document rotation direction recognition method in the above embodiment, which can solve the technical problem of low accuracy of document rotation direction recognition in the prior art. Compared with the prior art, the beneficial effects of the document rotation direction recognition device provided by the present application are the same as the beneficial effects of the document rotation direction recognition method provided by the above embodiment, and other technical features in the document rotation direction recognition device are the same as the features disclosed in the method of the previous embodiment, which will not be repeated here.

[0125] It should be understood that the various parts disclosed in this application can be implemented by hardware, software, firmware or a combination thereof. In the description of the above embodiments, specific features, structures, materials or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0126] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0127] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or system including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or system. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or system including the element.

[0128] The serial numbers of the above embodiments of the present invention are only for description and do not represent the advantages or disadvantages of the embodiments.

[0129] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course can also be implemented by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as a read-only memory / random access memory, a magnetic disk, or an optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.) to execute the methods described in each embodiment of the present invention.

[0130] The above are only preferred embodiments of the present invention, and are not intended to limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A method for identifying a document rotation direction, characterized in that: The method comprises: Proportionally scaling the predicted document image according to a preset size to obtain a target document image; Using each preset cropping scale to crop the target document image, respectively, to obtain the target image block corresponding to each preset cropping scale; According to the input resolution of the multi-task convolutional neural network model, scaling the target image blocks corresponding to each of the preset cropping scales, and inputting the scaling results into the multi-task convolutional neural network model to obtain a classification result; Voting is performed based on the classification results, and the direction classification result of the document image to be predicted is determined according to the voting results.

2. The document rotation direction recognition method according to claim 1, characterized in that: Before the step of scaling the predicted document image in proportion to the preset size to obtain the target document image, the method further includes: Acquire image blocks corresponding to a plurality of preset cropping scales, and generate corresponding region two-category labels and document rotation direction four-category labels based on the image blocks corresponding to the preset cropping scales; The image blocks corresponding to the preset cropping scales, the region binary classification labels corresponding to the image blocks, and the document rotation direction four-classification labels are used as training data sets; Constructing a convolutional neural network model, wherein the convolutional neural network model includes two classification heads, and the two classification heads are used to output a region classification result and a document rotation direction classification result respectively; The convolutional neural network model is trained based on the training data set and the preset loss function, and the parameters of the convolutional neural network model are optimized according to the training results to obtain a multi-task convolutional neural network model.

3. The document rotation direction recognition method according to claim 2, characterized in that: The step of obtaining image blocks corresponding to a plurality of preset cropping scales, and generating corresponding region two-category labels and document rotation direction four-category labels based on the image blocks corresponding to the preset cropping scales, comprises: Collecting document images, and correcting the rotation direction of each of the document images to obtain a corrected document image; Proportionally scaling the corrected document image according to a preset size, and annotating a text area on the proportionally scaled document image to obtain an annotated document image; Determine a plurality of preset cropping scales, and respectively use the preset cropping scales to crop image blocks of the annotated document image to obtain image blocks corresponding to the preset cropping scales; Based on the image blocks corresponding to the preset cropping scales, corresponding region two-category labels and document rotation direction four-category labels are generated.

4. The document rotation direction recognition method according to claim 3, characterized in that: The step of generating corresponding region two-category labels and document rotation direction four-category labels based on the image blocks corresponding to the preset cropping scales comprises: Determine the type of each image block based on the area of ​​the text region in the image block corresponding to each preset cropping scale, and obtain a type determination result; Generate a regional binary classification label corresponding to each of the image blocks according to the type judgment result; Generate four-category labels of document rotation directions corresponding to each image block according to the correction standard of the corrected document image.

5. The document rotation direction recognition method according to claim 1, characterized in that: The step of scaling the target image blocks corresponding to the preset cropping scales according to the input resolution of the multi-task convolutional neural network model, and inputting the scaling results into the multi-task convolutional neural network model to obtain the classification results includes: According to the input resolution of the multi-task convolutional neural network model, scaling the target image blocks corresponding to each of the preset cropping scales to obtain a scaling result; Inputting the scaling result into the multi-task convolutional neural network model to obtain the region binary classification result of each target image block and the document rotation direction four-classification result; The region binary classification result of each target image block and the document rotation direction quadrature classification result are used as the classification result of each target image block.

6. The document rotation direction recognition method according to claim 5, characterized in that: The step of voting based on the classification result and determining the direction classification result of the document image to be predicted according to the voting result includes: Filtering the background image blocks in the target image blocks based on the regional binary classification results of each of the target image blocks to obtain filtered target image blocks; Voting on the classification results corresponding to the filtered target image blocks to obtain a first voting result, and determining an initial classification result corresponding to each of the preset cropping scales based on the first voting result; Voting is performed on the initial classification results corresponding to the preset cropping scales to obtain a second voting result, and the direction classification result of the document image to be predicted is determined based on the second voting result.

7. The document rotation direction recognition method according to any one of claims 1 to 6, characterized in that: After the step of using each preset cropping scale to crop the target document image into image blocks respectively to obtain the target image blocks corresponding to each preset cropping scale, the method further includes: According to a preset downsampling ratio, the target image blocks corresponding to the preset cropping scales are downsampled at intervals to obtain the sampled target image blocks corresponding to the preset cropping scales.

8. A document rotation direction recognition system, characterized in that: The system comprises: An image scaling module is used to scale the predicted document image in proportion to a preset size to obtain a target document image; An image cropping module, used to crop the target document image into image blocks using each preset cropping scale to obtain target image blocks corresponding to each preset cropping scale; A model reasoning module, used to scale the target image blocks corresponding to each of the preset cropping scales according to the input resolution of the multi-task convolutional neural network model, and input the scaling results into the multi-task convolutional neural network model to obtain a classification result; The result voting module is used to vote based on the classification result and determine the direction classification result of the document image to be predicted according to the voting result.

9. A document rotation direction recognition device, characterized in that: The device comprises: a memory, a processor, and a document rotation direction recognition program stored in the memory and executable on the processor, wherein the document rotation direction recognition program is configured to implement the steps of the document rotation direction recognition method according to any one of claims 1 to 7.

10. A storage medium, characterized in that: The storage medium stores a document rotation direction recognition program, and when the document rotation direction recognition program is executed by the processor, the steps of the document rotation direction recognition method according to any one of claims 1 to 7 are implemented.

Images