Document image processing method and device, medium, equipment and model training method

Abstract

The present disclosure relates to a document image processing method, device, medium, equipment and model training method. The processing method comprises: performing image segmentation on the document image through a hue saturation value color model to generate a three-channel space image, filtering a background area of the document image to generate a binary edge image, splicing and fusing the three-channel space image and the binary edge image to generate target input data, performing flip recognition on the target input data according to an image flip recognition model, determining a target flip category of the document image, and performing reverse flip on the document image according to the target flip category to obtain a target image. Thus, the text features in the document image are recognized through the image flip recognition model, and the document image is flipped according to the flip angle, so that the uploaded document image is not disturbed by the shooting angle, and the user can intuitively view the information in the document image.

Description

Technical Field

[0001] This disclosure relates to the field of image processing, and more specifically, to a method, apparatus, medium, device, and model training method for processing document images. Background Technology

[0002] In existing online education scenarios, teachers and students frequently use mobile phones, tablets, and other camera devices to photograph and upload teaching materials such as books, test papers, and workbooks for subsequent assignment submission and note archiving. However, in practice, improper camera angles often result in uploaded images being flipped, causing difficulties in grading assignments and organizing archives. Summary of the Invention

[0003] To overcome the problems existing in related technologies, this disclosure provides a method, apparatus, medium, device, and model training method for processing document images, which solves the technical problem that uploaded images are flipped due to non-standard shooting angles, making them inconvenient to view.

[0004] According to a first aspect of the present disclosure, a method for processing document images is provided, comprising:

[0005] The document image is segmented using a hue saturation value color model to generate a three-channel spatial image.

[0006] The background region of the document image is filtered to generate a binary edge image;

[0007] The three-channel spatial image and the binary edge image are stitched and fused together to generate target input data;

[0008] The target input data is flipped using an image flipping recognition model to determine the target flipping category of the document image.

[0009] The document image is flipped in reverse according to the target flip category to obtain the target image.

[0010] Optionally, the image flipping recognition model includes a feature extraction network and a convolutional neural network. The step of performing flipping recognition on the target input data based on the image flipping recognition model to determine the target flipping category of the document image includes:

[0011] The feature extraction network extracts features from the target input data to generate a three-dimensional feature image.

[0012] The confidence scores of the three-dimensional feature images belonging to multiple image flipping categories are obtained based on the convolutional neural network, and the image flipping category with the highest confidence score is determined as the target flipping category.

[0013] Optionally, the step of extracting features from the target input data through the feature extraction network to generate a three-dimensional feature image includes:

[0014] The feature extraction network identifies the text size of the target input data.

[0015] The first and second convolutional kernels of the feature extraction network are determined based on the text size.

[0016] The input data is used to extract features based on the first-layer convolutional kernel and the second-layer convolutional kernel to generate the three-dimensional feature image.

[0017] Optionally, the step of obtaining confidence scores of the three-dimensional feature image belonging to multiple image flipping categories based on the convolutional neural network, and determining the image flipping category with the highest confidence score as the target flipping category, includes:

[0018] The three-dimensional feature map is reduced in dimensionality using the convolutional neural network to generate a one-dimensional feature vector.

[0019] The one-dimensional feature vector is linked with a preset fully connected network to generate multiple confidence scores belonging to multiple image flipping categories, wherein the preset fully connected network is set according to the multiple image flipping categories;

[0020] The multiple confidence scores are compared, and the image flip category with the highest confidence score is determined as the target flip category.

[0021] Optionally, linking the one-dimensional feature vector with a preset fully connected network to generate multiple confidence scores belonging to multiple image flipping categories includes:

[0022] The one-dimensional feature vector is classified according to the preset fully connected network to generate the multiple image flipping categories;

[0023] Confidence scores are calculated for the multiple image flipping categories to generate the multiple confidence scores.

[0024] According to a second aspect of the present disclosure, a training method for an image flipping recognition model is provided, applied to the image flipping recognition model described in any one of the first aspects above, the method comprising:

[0025] The sample image is cropped according to a random cropping algorithm to generate a preset first number of cropped sub-images;

[0026] Based on whether the cropped sub-image contains text information, the cropped sub-image is filtered to obtain a valid image, wherein the valid image contains text information;

[0027] The valid image is mirrored according to a preset flipping rule to generate a flipped image and a preset flipping label.

[0028] The image flipping recognition model is used to identify the flipped image and determine the target flipped label.

[0029] If the target flip label is inconsistent with the preset flip label, the image flip recognition model is trained based on the preset flip label.

[0030] Optionally, the step of filtering the cropped sub-images based on whether they contain text information to obtain valid images includes:

[0031] The text region of any cropped sub-image is located using a text detection algorithm, and a first position record code is generated.

[0032] If the total value of the first position recording code is greater than a preset pixel threshold, and the total area of ​​the text region is greater than a preset proportion of the cropping area, then the cropped sub-image is determined to be the valid image.

[0033] Optionally, the method includes:

[0034] If the number of valid images is less than a preset second number, the number of valid images is expanded to generate the preset second number of valid images.

[0035] Optionally, expanding the valid images to generate the preset second number of valid images includes:

[0036] A preset style image is randomly pasted into the effective image to generate an initial augmented image;

[0037] Based on the text detection algorithm, the text regions other than the preset style image in the initial augmented image are located, and a second position record code is generated;

[0038] If the total value of the second position recording code is greater than the preset pixel threshold, and the total area of ​​the text region is greater than the cropping area of ​​the preset ratio, the initial augmented image is determined to be the valid image.

[0039] According to a third aspect of the present disclosure, a document image processing apparatus is provided, the apparatus comprising:

[0040] The first generation module is used to perform image segmentation on the document image using a hue saturation value color model to generate a three-channel spatial image.

[0041] The second generation module is used to filter the background area of ​​the document image and generate a binary edge image;

[0042] The third generation module is used to stitch and fuse the three-channel spatial image and the binary edge image to generate target input data;

[0043] The determination module is used to perform flip recognition on the target input data according to the image flip recognition model, and determine the target flip category of the document image;

[0044] The flipping module is used to flip the document image in reverse according to the target flipping category to obtain the target image.

[0045] According to a fourth aspect of the present disclosure, a training apparatus for an image flipping recognition model is provided, comprising:

[0046] The fourth generation module is used to crop the sample image according to the random cropping algorithm and generate a preset first number of cropped sub-images;

[0047] The filtering module is used to filter cropped sub-images based on whether they contain text information, and obtain valid images, wherein the valid images contain text information.

[0048] The fifth generation module is used to mirror and flip valid images according to preset flipping rules, generating flipped images and preset flip labels;

[0049] The recognition module is used to identify flipped images using an image flipping recognition model and determine the target flipped label.

[0050] The execution module is used to train the image flipping recognition model based on the preset flipping label when the target flipping label is inconsistent with the preset flipping label.

[0051] According to a fifth aspect of the present disclosure, a computer-readable storage medium is provided having a computer program stored thereon that, when executed by a processor, implements the steps of the method described in any one of the first aspects.

[0052] According to a sixth aspect of the present disclosure, an electronic device is provided, comprising:

[0053] A memory on which computer programs are stored;

[0054] A processor for executing the computer program in the memory to implement the steps of the method of any one of the first aspects.

[0055] The above technical solution involves segmenting the document image using a hue and saturation value color model to generate a three-channel spatial image. The background area of ​​the document image is filtered to generate a binary edge image. The three-channel spatial image and the binary edge image are then stitched together to generate target input data. An image flipping recognition model is used to identify the flipping category of the target input data, determining the flipping type of the document image. Based on this category, the document image is flipped in reverse to obtain the target image. This method also identifies text features in the document image, determines the flipping category, and flips the image in reverse to obtain a correctly identifiable target image. This ensures that uploaded document images are not affected by the shooting angle, allowing users to intuitively view the information within the document image.

[0056] Other features and advantages of this disclosure will be described in detail in the following detailed description section. Attached Figure Description

[0057] The accompanying drawings are provided to further illustrate the present disclosure and form part of the specification. They are used together with the following detailed description to explain the present disclosure, but do not constitute a limitation thereof. In the drawings:

[0058] Figure 1 This is a flowchart illustrating a document image processing method according to an exemplary embodiment.

[0059] Figure 2 This is an example diagram illustrating a three-channel image according to an exemplary embodiment.

[0060] Figure 3 This is an example diagram of a binary edge image according to an exemplary embodiment.

[0061] Figure 4 This is a flowchart illustrating a method for confirming an image flipping category according to an exemplary embodiment.

[0062] Figure 5 This is an example diagram illustrating a first-layer convolutional kernel according to an exemplary embodiment.

[0063] Figure 6 This is an example diagram illustrating a sublayer convolutional kernel according to an exemplary embodiment.

[0064] Figure 7 This is a flowchart illustrating another method for confirming the flipping category of a target image according to an exemplary embodiment.

[0065] Figure 8 This is a flowchart illustrating a training method for an image flipping recognition model according to an exemplary embodiment. Figure 9This is a block diagram illustrating a document image processing apparatus according to an exemplary embodiment.

[0066] Figure 10 This is a block diagram illustrating a training apparatus for an image flipping recognition model according to an exemplary embodiment.

[0067] Figure 11 This is a block diagram illustrating an electronic device according to an exemplary embodiment. Detailed Implementation

[0068] The specific embodiments of this disclosure will be described in detail below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are for illustration and explanation only and are not intended to limit this disclosure.

[0069] It should be noted that all actions involving the acquisition of signals, information, or data in this disclosure are carried out in compliance with the relevant data protection laws and policies of the country where the location is situated, and with authorization from the owner of the relevant device.

[0070] As the internet increasingly integrates with educational settings, teachers and students often use their phones to photograph exam papers for submission and grading to facilitate information transfer. They also frequently use their phones to photograph key points in notes or data to preserve important information. However, when taking photos with a phone, users often struggle to maintain a uniform, upright orientation, resulting in tilted document images. This often leads to uploaded documents being flipped at 90°, 180°, 270°, or other angles, ultimately rendering them unreadable.

[0071] In existing technologies, mobile phones generate images containing EXIF ​​(Exchangeable Image File Format) information when taking photos. EXIF ​​information records the attributes and shooting data of the digital photo, which can be used for angle correction of document images. However, EXIF ​​information is easily lost during image uploading and parsing, and image correction based on EXIF ​​information requires the precondition of shooting in the correct orientation; if this precondition is not met, angle recognition cannot be performed. For document images with diverse styles, angle recognition using EXIF ​​information suffers from poor robustness, leading to significant difficulties in document image recognition. Training a recognition model by classifying sample images and then using the trained model to recognize document images is also problematic. When the sample set is limited and cannot comprehensively cover all types of document images, the resulting recognition model has poor robustness and is prone to incorrect angle recognition. Angle correction based on prior information about document styles requires ensuring the consistency of the position of specific style content within the document image. It also necessitates the use of a corresponding object detection model to detect this style content and determine the flip angle of the document image. However, object detection models cannot achieve high accuracy in detecting specific style targets, significantly limiting the accuracy of this recognition method. Furthermore, when prior information changes, both the angle correction rules and the object detection algorithm need to be redesigned, resulting in high maintenance costs.

[0072] In view of this, embodiments of this disclosure propose a method for processing document images. Figure 1 This is a flowchart illustrating a document image processing method according to an exemplary embodiment, see [link to flowchart]. Figure 1 The processing method includes:

[0073] Step S11: Perform image segmentation on the document image using the hue saturation value color model to generate a three-channel spatial image.

[0074] It should be noted that the document images in this embodiment mainly refer to document images generated when users take photos of book documents (including textbooks, notebooks, test papers, and other teaching materials) using mobile devices such as smartphones in an educational context. Typically, these document images contain a large amount of standardized or handwritten text information. This embodiment focuses on the text information in the document image, confirming the flip angle of the document image, and requires feature extraction of the text information. Since the text information in the document image is not uniform and is affected by the background color and text, different feature extraction methods are needed for different types of text information. The document image is segmented using a hue / saturation value color model to generate a three-channel spatial image. For example, the hue / saturation value color model can be the HSV color model, where H represents hue, S represents saturation, and V represents brightness; the HSV color model can filter the document image in three dimensions to generate an HSV three-channel image. For example, Figure 2 This is an example diagram illustrating a three-channel image according to an exemplary embodiment; see also... Figure 2 Figure (a) shows the initial document image; Figure (b) shows the channel sub-image generated after filtering the document image using hue as the color filter information in HSV space; Figure (c) shows the channel sub-image generated after filtering the document image using saturation as the color filter information in HSV space; and Figure (d) shows the channel sub-image generated after filtering the document image using brightness as the filter information. In HSV space, the document image is filtered using various dimensions as filter color information to separate the background pattern and text area of ​​the document image, extract the text area, and generate a three-dimensional, three-channel spatial image.

[0075] Step S12: Filter the background area of ​​the document image to generate a binary edge image.

[0076] It is understandable that the document images are obtained by photographing textbook texts, which contain watermarks and background interference. When extracting features from the document images, it is necessary to filter out the corresponding interference information. In this embodiment, an edge detection operator can be introduced to perform edge detection on the document images. Figure 3 This is an example diagram of a binary edge image according to an exemplary embodiment, see [link to example diagram]. Figure 3 By exploiting the different frequency responses between text and background regions in a document image, the background in the document image is filtered out. Specifically, the background region of the document image lacks high-frequency information, so edge detection filters out this region. For text regions in the document image, edge detection generates high-frequency responses, thus determining the location of the text regions and extracting features from them to generate a binary edge image.

[0077] Step S13: The three-channel spatial image and the binary edge image are stitched together and fused to generate the target input data.

[0078] Understandably, after obtaining the three-channel spatial image and the binary edge image through the above steps, since the spatial image and the edge image have the same image attributes, a fusion algorithm can be used to fuse the three-channel spatial image and the binary edge image to generate target input data for feature extraction. This target input data is 4-channel data, used to represent the textual information features of the document image in four different dimensions.

[0079] Step S14: Perform flip recognition on the target input data according to the image flip recognition model to determine the target flip category of the document image.

[0080] After filtering the text information of the document image through the above steps, a filtered image with 4-channel attributes is obtained. This filtered image is then used as the target input data and fed into the image flipping recognition model. The image flipping recognition model performs flipping recognition on the target input data to determine the flipping angle of the text information in the target input data, thereby determining the target flipping category of the document image. It should be noted that, typically, after a user obtains a document image by photographing it with a mobile device, the flipping category of the document image can include: horizontal axis mirror flip (the pixel values ​​in the vertical direction are symmetrically swapped vertically); vertical axis mirror flip (the pixel values ​​in the horizontal direction are symmetrically swapped horizontally); counterclockwise rotation of 0-180° (the document image is rotated counterclockwise around the image center by a certain angle); and clockwise rotation of 0-180° (the document image is rotated clockwise around the image center by a certain angle). The image flipping recognition model is used to extract features from the text information in the input data and perform flipping recognition on the extracted text information to determine the target flipping category of the document image.

[0081] Figure 4 This is a flowchart illustrating a method for determining an image flipping category according to an exemplary embodiment, see [link to flowchart]. Figure 4 The image flipping recognition model in the above embodiments includes a feature extraction network and a convolutional neural network. Step S14 includes:

[0082] Step S141: Extract features from the target input data using a feature extraction network to generate a three-dimensional feature image.

[0083] Step S142: Obtain the confidence scores of the three-dimensional feature images belonging to multiple image flipping categories based on the convolutional neural network, and determine the image flipping category with the highest confidence score as the target flipping category.

[0084] It is understood that the image flipping recognition model in this embodiment includes a neural network learning algorithm, which may include a feature extraction network and a convolutional neural network. The feature extraction network in the image flipping recognition model extracts text features from the input data to obtain a three-dimensional feature image of size hs*ws*c (where hs represents the height of the feature image, ws represents the width of the feature image, and c represents the number of channels in the feature image). The convolutional neural network extracts features from the three-dimensional feature image, determines the confidence score of the three-dimensional feature image belonging to multiple image flipping categories, and determines the image flipping category with the highest confidence score as the target flipping category for the document image. For example, the image flipping categories may include five categories: not flipped, horizontal axis mirror flip, vertical axis mirror flip, counterclockwise flip, and clockwise flip. The convolutional neural network scores the confidence of the three-dimensional feature image belonging to the five flipping categories, obtaining multiple confidence scores, and determines the image flipping category with the highest confidence frequency as the target flipping category.

[0085] Optionally, step S141 above includes:

[0086] The feature extraction network identifies the text size of the target input data.

[0087] The first and second convolutional kernels of the feature extraction network are determined based on the text size.

[0088] The input data is used to extract features based on the first-layer convolutional kernel and the second-layer convolutional kernel to generate a three-dimensional feature image.

[0089] Understandably, text sizes vary across different books and documents. To ensure the feature extraction network accurately extracts text information from the target input data, the sizes of the first and second layer convolutional kernels are set differently for different text types. Typically, text sizes in books and documents are uniform. Therefore, for documents with smaller text sizes, a smaller kernel can be used in the first layer to extract text features; conversely, for documents with larger text sizes, a larger kernel can be used. For example... Figure 5 This is an example diagram illustrating a first-layer convolutional kernel according to an exemplary embodiment; see also... Figure 5 After determining the size of the text in the target input data, a 3*3 convolution kernel or a 5*5 convolution kernel can be determined based on the text size.

[0090] It should be noted that, typically, books and documents are written from top to bottom and from left to right. Therefore, considering the characteristics of books and documents, in this embodiment, the secondary convolutional kernels in the feature extraction network can be set to elongated kernels based on the text size of the target input data. This increases the kernel size while maintaining the downsampling of the convolutional image, thus improving the feature extraction network's performance in extracting features from text regions. For example... Figure 6 This is an example diagram illustrating a sub-convolutional kernel according to an exemplary embodiment, see [link to example diagram]. Figure 6 After determining the text size in the target input data to be 1*1 through the above steps, the secondary convolutional kernel can be set to a kernel with a width of 1 unit and a length of 3 units, or it can be set to a kernel with a size of 1*5. Features are extracted from the text information in the input data through the primary and secondary convolutional kernels to generate a 3D feature image corresponding to the document image.

[0091] Figure 7 This is a flowchart illustrating another method for confirming the flipping category of a target image according to an exemplary embodiment, see [link to flowchart]. Figure 7 The above step S142 includes:

[0092] Step S1421: Reduce the dimensionality of the three-dimensional feature map using a convolutional neural network to generate a one-dimensional feature vector.

[0093] Step S1422: Link the one-dimensional feature vector with a preset fully connected network to generate multiple confidence scores belonging to multiple image flipping categories. The preset fully connected network is set according to multiple image flipping categories.

[0094] Step S1423: Compare multiple confidence scores and determine the image flipping category with the highest confidence score as the target flipping category.

[0095] It is understandable that after generating a 3D feature image through the above steps, the 3D feature image is dimensionality reduced to generate a 1D feature vector. For example, in this embodiment, a flattening method can be used to flatten the 3D feature image to obtain a 1D feature vector of length (hs*ws*c). Based on the image flipping category determined in the above embodiment as 5, a preset fully connected network with 5 nodes (attributes) is set. The preset fully connected network is linked with the 1D feature vector to obtain multiple confidence scores of the input data belonging to different image flipping categories. The image flipping category with the highest confidence score is determined as the target flipping category.

[0096] Optionally, step S1422 above includes:

[0097] The one-dimensional feature vector is classified according to a pre-defined fully connected network to generate multiple image flipping categories.

[0098] Confidence scores are calculated for multiple image flipping categories to generate multiple confidence ratings.

[0099] It is understood that in this embodiment, the preset fully connected network can be set according to the image flipping category, and the corresponding number of nodes are set in the preset fully connected network according to the number of image flipping categories. The one-dimensional feature vector is classified according to the preset fully connected network to determine multiple image flipping categories corresponding to the one-dimensional feature vector. Softmax (logistic regression function) is used to calculate the confidence of each image flipping category to generate multiple confidence scores under different categories.

[0100] Step S15: Based on the target flip category, flip the document image in reverse to obtain the target image.

[0101] It is understandable that after obtaining the target flip category through the above steps, the document image is flipped in reverse according to the target flip category to obtain the target image. For example, if the above steps determine that the document image's image flip category is horizontal mirror flip, then according to the relevant code instructions, the document image can be mirror flipped once according to the rule of horizontal axis mirror flipping to obtain the target image. For example, using the Python programming language, the code `img=img[∷-1,∶,∶]` can be used to achieve a horizontal mirror flip of the document image to obtain the target image.

[0102] The above technical solution involves segmenting the document image using a hue and saturation value color model to generate a three-channel spatial image. The background area of ​​the document image is filtered to generate a binary edge image. The three-channel spatial image and the binary edge image are then stitched together to generate target input data. An image flipping recognition model is used to identify the flipping category of the target input data, determining the flipping type of the document image. Based on this category, the document image is flipped in reverse to obtain the target image. This method also identifies text features in the document image, determines the flipping category, and flips the image in reverse to obtain a correctly identifiable target image. This ensures that uploaded document images are not affected by the shooting angle, allowing users to intuitively view the information within the document image.

[0103] Figure 8 This is a flowchart illustrating a training method for an image flipping recognition model according to an exemplary embodiment. See also... Figure 8 The training method includes:

[0104] Step S21: Crop the sample image according to the random cropping algorithm to generate a preset first number of cropped sub-images.

[0105] It is understood that in this embodiment, the image flipping recognition model in the above embodiment is trained based on sample images from the database. However, the sample images in the database are document images uploaded by users via mobile devices, and the number of samples is relatively small. Therefore, to improve the robustness of the image flipping recognition model, it is necessary to crop the sample images to generate a preset number of sample images. In this embodiment, the sample images in the database are cropped according to a random cropping algorithm to generate a preset first number of cropped sub-images. For example, the random cropping algorithm can randomly determine the cropping start point, cropping length, and cropping width of the sample image based on its size, thereby generating the preset first number of cropped sub-images. For example, the random cropping algorithm can be set using the following pseudocode:

[0106] Input: image (the origin image)

[0107] Output: the list of final image (including some cut image from originimage)

[0108] # ----------------------------------------------------------------

[0109] Begin:

[0110] w, h = image.shape# the shape of ori image

[0111] cut_images = list() # to save final image

[0112] for i begin 0 to 10:

[0113] fw = random(0, w) # the width of final image

[0114] fh = random(0, h) # the height of final image

[0115] begin_x = random(0, w-fw) # the begin x location of final image

[0116] begin_y = random(0, h-fh)

[0117] cut_image = image[begin_y:begin_y+fh, begin_x:begin_x+fw]

[0118] if cal_valid(cut_image): # assert cut_image has text information.

[0119] cut_images.append(cut_image) # if valid, save cut image

[0120] End

[0121] # -------------------------------------------------------------------

[0122] It should be noted that the sample images in this embodiment include multiple images. Under normal circumstances, the number of cropped sub-images corresponding to each sample image can be determined by a preset first number and the sample images, that is, the number of times random cropping needs to be performed based on the same sample image.

[0123] Step S22: Based on whether the cropped sub-images contain text information, filter the cropped sub-images to obtain valid images, wherein the valid images contain text information.

[0124] It is understood that this embodiment primarily determines the flip angle of the document image by extracting and recognizing the text information from the document image. The document image may contain text regions as well as image regions. The cropped sub-images generated after randomly cropping the document image may include invalid images containing only images and no text information, as well as valid images containing text information. Invalid images cannot be used to train the image recognition model. Therefore, this embodiment determines whether a cropped sub-image is valid or invalid by confirming whether it contains text information, and then selects the valid images for model training.

[0125] Optionally, step S22 above includes:

[0126] The text region of any cropped sub-image is located using a text detection algorithm, and a first position record code is generated.

[0127] If the total value of the first position recording code is greater than the preset pixel threshold and the total area of ​​the text region is greater than the cropping area of ​​the preset ratio, the cropped sub-image is determined to be a valid image.

[0128] It is understood that in this embodiment, text regions in the cropped sub-image can be located using text detection algorithms (e.g., DBNet, ESAT, PANet, etc.) to generate a first position recording code for recording the position of the text region in the cropped sub-image. For example, the first position recording code is a mask code, which is used to record the position of the text region in the cropped sub-image. Typically, the mask code value corresponding to the region containing text information is 1, and the mask code value corresponding to the region not containing text information is 0. The validity of a cropped sub-image is determined by checking whether the total value of the mask codes in the cropped sub-image exceeds a preset pixel threshold and whether the total area of ​​the corresponding text regions exceeds a preset proportion of the cropped area. For example, if the preset pixel threshold is 30 and the preset proportion is 1 / 3, and the total mask value of the cropped sub-image is determined to be 50 using the first position record code, then the cropped sub-image is considered valid if the total area of ​​the regions with mask codes of 1 is greater than 1 / 3 of the total area of ​​the cropped sub-image. Conversely, if neither the total mask value is greater than 30 nor the area of ​​regions with mask values ​​of 1 is less than 1 / 3 of the total area of ​​the cropped sub-image, then the cropped sub-image is considered invalid. By filtering the cropped sub-images using these steps, valid images are identified.

[0129] Optionally, the above training method also includes:

[0130] If the number of valid images is less than a preset second number, the number of valid images is expanded to generate a preset second number of valid images.

[0131] Understandably, to improve the robustness of the image flipping recognition model, it is necessary to expand the number of samples to enhance its generalization ability. After determining a certain number of valid images through the above steps, it is necessary to judge whether the number of valid images exceeds a second preset number. When the number of valid images reaches the preset second preset number, the image flipping recognition model can be directly trained based on these valid images. When the number of valid images does not reach the preset second preset number, it is necessary to expand the valid images based on the existing valid images to generate the preset second preset number of valid images. For example, the valid images can be expanded by pasting images or by further cropping.

[0132] Alternatively, the effective image can be augmented in the following ways:

[0133] The preset style image is randomly pasted into the valid image to generate the initial augmented image.

[0134] The text detection algorithm is used to locate the text regions in the initial augmented image other than the preset style image, and a second position record code is generated.

[0135] If the total value of the second position recording code is greater than the preset pixel threshold and the total area of ​​the text region is greater than the cropping area of ​​the preset ratio, the initial augmented image is determined to be a valid image.

[0136] It is understood that in this embodiment, an image augmentation method is used to augment the valid image determined in the above steps. A preset style image stored in the database is randomly pasted into the valid image to generate an initial augmented image. Based on the method for determining the valid image in the above steps, it is determined whether the initial augmented image generated after removing the preset style image portion is a valid image, thereby determining whether the image augmentation is successful. If the image augmentation is successful, the initial augmented image is determined to be a valid image; if the image augmentation is unsuccessful, based on the valid image, the image augmentation continues according to the above image augmentation method until a valid image is generated. The text regions in the initial augmented image, after removing the preset style image region, are located, and a second position record code is generated. The validity of the text regions in the initial augmented image other than the preset style image is determined based on the second position record code. When the total value of the second position record code is greater than a preset pixel threshold, and the total area of ​​the text regions is greater than a preset proportion of the cropped area area, the initial augmented image is determined to be a valid image.

[0137] Step S23: Mirror the valid image according to the preset flipping rules to generate a flipped image and a preset flipped label.

[0138] Valid images are flipped according to preset flipping rules to generate flipped images and corresponding preset flipping labels. These preset flipping rules can include: horizontal axis mirror flip (swapping pixel values ​​vertically symmetrically); vertical axis mirror flip (swapping pixel values ​​horizontally symmetrically); 90° counterclockwise rotation (rotating the image 90° counterclockwise around its center); and 90° clockwise rotation (rotating the image 90° clockwise around its center). It should be noted that to improve the robustness of the image flipping recognition model, each valid image is mirror-flipped in a preset direction, generating four mirror-flipped images. A corresponding preset flipping label is then generated based on the flipping type of each image.

[0139] Step S24: The flipped image is identified using an image flipping recognition model to determine the target flipped label.

[0140] Step S25: If the target flip label is inconsistent with the preset flip label, train the image flip recognition model according to the preset flip label.

[0141] Understandably, the valid images and flipped images generated through the above steps are used as a training set to train the image flipping recognition model, determining the target flipping label for each image. The target flipping label is then compared with a preset flipping label to determine if they match. If they do not match, the image flipping recognition model is trained based on the preset flipping label, correcting the neural network algorithm of the model and enabling it to accurately identify the mirror flipping category of document images.

[0142] The above-described technical method involves cropping sample images using a random cropping algorithm to generate a predetermined number of cropped sub-images. These sub-images are then filtered based on whether they contain text information to obtain valid images containing text. These valid images are then mirror-flipped according to a predetermined flipping rule, generating flipped images and predetermined flipping labels. An image flipping recognition model identifies the flipped images and determines the target flipping label. If the target flipping label does not match the predetermined flipping label, the image flipping recognition model is trained based on the predetermined flipping label. This approach increases the richness of the sample through random cropping and ensures that all training samples are valid. Flipping these valid samples trains the image flipping recognition model, improving its robustness and generalization ability, and enabling it to accurately determine the image flipping angle.

[0143] Figure 9 This is a block diagram illustrating a document image processing apparatus according to an exemplary embodiment, see below. Figure 9 The processing device 100 includes a first generation module 110, a second generation module 120, a third generation module 130, a determination module 140, and a flipping module 150.

[0144] The first generation module 110 is used to perform image segmentation on the document image using a hue saturation value color model to generate a three-channel spatial image.

[0145] The second generation module 120 is used to filter the background area of ​​the document image and generate a binary edge image.

[0146] The third generation module 130 is used to stitch and fuse the three-channel spatial image and the binary edge image to generate target input data.

[0147] The determination module 140 is used to perform flip recognition on the target input data according to the image flip recognition model to determine the target flip category of the document image.

[0148] The flip module 150 is used to flip the document image in reverse according to the target flip category to obtain the target image.

[0149] Optionally, the image flipping recognition model includes a feature extraction network and a convolutional neural network, and the determining module 140 includes:

[0150] The generation submodule is used to extract features from the target input data through a feature extraction network and generate a 3D feature image.

[0151] The determination submodule is used to obtain the confidence scores of the three-dimensional feature images belonging to multiple image flipping categories based on the convolutional neural network, and to determine the image flipping category with the highest confidence score as the target flipping category.

[0152] Optionally, this generation submodule is also used for:

[0153] The feature extraction network identifies the text size of the target input data.

[0154] The first and second convolutional kernels of the feature extraction network are determined based on the text size.

[0155] The input data is used to extract features based on the first-layer convolutional kernel and the second-layer convolutional kernel to generate a three-dimensional feature image.

[0156] Optionally, the determining submodule includes:

[0157] The first generation unit is used to reduce the dimensionality of the three-dimensional feature map using a convolutional neural network to generate a one-dimensional feature vector.

[0158] The second generation unit is used to link the one-dimensional feature vector with a preset fully connected network to generate multiple confidence scores belonging to multiple image flipping categories. The preset fully connected network is set according to multiple image flipping categories.

[0159] The determination unit is used to compare multiple confidence scores and determine the image flip category with the highest confidence score as the target flip category.

[0160] Optionally, the second generating unit can also be used for:

[0161] The one-dimensional feature vector is classified according to a pre-defined fully connected network to generate multiple image flipping categories.

[0162] Confidence scores are calculated for multiple image flipping categories to generate multiple confidence ratings.

[0163] Figure 10 This is a block diagram illustrating a training apparatus for an image flipping recognition model according to an exemplary embodiment. See also... Figure 10 The training device 200 includes: a fourth generation module 210, a screening module 220, a fifth generation module 230, a recognition module 240, and an execution module 250.

[0164] The fourth generation module 210 is used to crop the sample image according to the random cropping algorithm and generate a preset first number of cropped sub-images.

[0165] The filtering module 220 is used to filter the cropped sub-images based on whether they contain text information, and obtain valid images, wherein the valid images contain text information.

[0166] The fifth generation module 230 is used to mirror and flip the valid image according to the preset flipping rules, and generate the flipped image and the preset flip label.

[0167] The recognition module 240 is used to recognize the flipped image through the image flipping recognition model and determine the target flipped label.

[0168] The execution module 250 is used to train the image flipping recognition model according to the preset flipping label when the target flipping label is inconsistent with the preset flipping label.

[0169] Optionally, the filtering module 220 can also be used for:

[0170] The text region of any cropped sub-image is located using a text detection algorithm, and a first position record code is generated.

[0171] If the total value of the first position recording code is greater than the preset pixel threshold and the total area of ​​the text region is greater than the cropping area of ​​the preset ratio, the cropped sub-image is determined to be a valid image.

[0172] Optionally, the training device 200 also includes an expansion module for:

[0173] If the number of valid images is less than a preset second number, the number of valid images is expanded to generate a preset second number of valid images.

[0174] Optionally, the expansion module can also be used for:

[0175] The preset style image is randomly pasted into the valid image to generate the initial augmented image.

[0176] The text detection algorithm is used to locate the text regions in the initial augmented image other than the preset style image, and a second position record code is generated.

[0177] If the total value of the second position recording code is greater than the preset pixel threshold and the total area of ​​the text region is greater than the cropping area of ​​the preset ratio, the initial augmented image is determined to be a valid image.

[0178] Regarding the apparatus in the above embodiments, the specific manner in which each module performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated upon here.

[0179] Figure 11 This is a block diagram illustrating an electronic device 1100 according to an exemplary embodiment. For example, the electronic device 1100 may be provided as a server. (Refer to...) Figure 11 The electronic device 1100 includes a processor 1122, which may be one or more, and a memory 1132 for storing computer programs executable by the processor 1122. The computer program stored in the memory 1132 may include one or more modules, each corresponding to a set of instructions. Furthermore, the processor 1122 may be configured to execute the computer program to perform the aforementioned document image processing method.

[0180] Additionally, the electronic device 1100 may also include a power supply component 1126 and a communication component 1150. The power supply component 1126 can be configured to perform power management of the electronic device 1100, and the communication component 1150 can be configured to enable communication of the electronic device 1100, such as wired or wireless communication. Furthermore, the electronic device 1100 may also include an input / output (I / O) interface 1158. The electronic device 1100 can operate on an operating system, such as Windows Server, stored in memory 1132. TM Mac OS X TM Unix TM Linux TM etc.

[0181] In another exemplary embodiment, a computer-readable storage medium including program instructions is also provided, which, when executed by a processor, implement the steps of the document image processing method described above. For example, the non-transitory computer-readable storage medium may be the memory 1132 including the program instructions described above, which may be executed by the processor 1122 of the electronic device 1100 to complete the document image processing method described above.

[0182] In another exemplary embodiment, a computer program product is also provided, the computer program product comprising a computer program executable by a programmable device, the computer program having a code portion for performing the above-described document image processing method when executed by the programmable device.

[0183] The preferred embodiments of this disclosure have been described in detail above with reference to the accompanying drawings. However, this disclosure is not limited to the specific details of the above embodiments. Within the scope of the technical concept of this disclosure, various simple modifications can be made to the technical solutions of this disclosure, and these simple modifications all fall within the protection scope of this disclosure.

[0184] It should also be noted that the various specific technical features described in the above specific embodiments can be combined in any suitable manner without contradiction.

[0185] Furthermore, various different embodiments of this disclosure can be combined in any way, as long as they do not violate the spirit of this disclosure, they should also be regarded as the content disclosed in this disclosure.

Claims

1. A method for processing document images, characterized in that, include: The document image is segmented using a hue saturation value color model to generate a three-channel spatial image. The background region of the document image is filtered to generate a binary edge image; The three-channel spatial image and the binary edge image are stitched and fused together to generate target input data, which is used to characterize the text information features of the document image in four different dimensions. The target input data is flipped using an image flipping recognition model to determine the target flipping category of the document image. According to the target flip category, the document image is flipped in reverse to obtain the target image; The image flipping recognition model includes a feature extraction network and a convolutional neural network. The step of performing flipping recognition on the target input data based on the image flipping recognition model to determine the target flipping category of the document image includes: The feature extraction network identifies the text size of the target input data. The first and second convolutional kernels of the feature extraction network are determined based on the text size. The size of the first convolutional kernel is positively correlated with the text size, and the second convolutional kernel is a strip-shaped convolutional kernel. Based on the first-layer convolutional kernel and the second-layer convolutional kernel, feature extraction is performed on the input data to generate a three-dimensional feature image; The confidence scores of the three-dimensional feature images belonging to multiple image flipping categories are obtained by using a convolutional neural network, and the image flipping category with the highest confidence score is determined as the target flipping category. The multiple image flipping categories include no flipping, horizontal axis mirror flipping, vertical axis mirror flipping, counterclockwise flipping, and clockwise flipping.

2. The processing method according to claim 1, characterized in that, The step of obtaining confidence scores for the three-dimensional feature image belonging to multiple image flipping categories based on the convolutional neural network, and determining the image flipping category with the highest confidence score as the target flipping category, includes: The three-dimensional feature map is reduced in dimensionality using the convolutional neural network to generate a one-dimensional feature vector. The one-dimensional feature vector is linked with a preset fully connected network to generate multiple confidence scores belonging to multiple image flipping categories, wherein the preset fully connected network is set according to the multiple image flipping categories; The multiple confidence scores are compared, and the image flip category with the highest confidence score is determined as the target flip category.

3. The processing method according to claim 2, characterized in that, The step of linking the one-dimensional feature vector with a preset fully connected network to generate multiple confidence scores belonging to multiple image flipping categories includes: The one-dimensional feature vector is classified according to the preset fully connected network to generate the multiple image flipping categories; Confidence scores are calculated for the multiple image flipping categories to generate the multiple confidence scores.

4. A training method for an image flipping recognition model, characterized in that, The method, applied to the image flipping recognition model according to any one of claims 1-3, comprises: The sample image is cropped according to a random cropping algorithm to generate a preset first number of cropped sub-images; Based on whether the cropped sub-image contains text information, the cropped sub-image is filtered to obtain a valid image, wherein the valid image contains text information; The valid image is mirrored according to a preset flipping rule to generate a flipped image and a preset flipping label. The image flipping recognition model is used to identify the flipped image and determine the target flipped label. If the target flip label is inconsistent with the preset flip label, the image flip recognition model is trained based on the preset flip label.

5. The training method according to claim 4, characterized in that, The step of filtering the cropped sub-images based on whether they contain text information to obtain valid images includes: The text region of any cropped sub-image is located using a text detection algorithm, and a first position record code is generated. If the total value of the first position recording code is greater than a preset pixel threshold, and the total area of ​​the text region is greater than a preset proportion of the cropping area, then the cropped sub-image is determined to be the valid image.

6. The training method according to claim 4, characterized in that, The method includes: If the number of valid images is less than a preset second number, the number of valid images is expanded to generate the preset second number of valid images.

7. The training method according to claim 6, characterized in that, The step of expanding the valid images to generate the preset second number of valid images includes: A preset style image is randomly pasted into the effective image to generate an initial augmented image; Based on the text detection algorithm, the text regions other than the preset style image in the initial augmented image are located, and a second position record code is generated; If the total value of the second position recording code is greater than the preset pixel threshold, and the total area of ​​the text region is greater than the cropping area of ​​the preset ratio, the initial augmented image is determined to be the valid image.

8. A document image processing apparatus, characterized in that, include: The first generation module is used to perform image segmentation on the document image using a hue saturation value color model to generate a three-channel spatial image. The second generation module is used to filter the background area of ​​the document image and generate a binary edge image; The third generation module is used to stitch and fuse the three-channel spatial image and the binary edge image to generate target input data. The target input data is used to characterize the text information features of the document image in four different dimensions. A determination module is used to identify the text size of the target input data based on the feature extraction network; determine the first-layer convolutional kernel and the second-layer convolutional kernel of the feature extraction network based on the text size, wherein the size of the first-layer convolutional kernel is positively correlated with the text size, and the second-layer convolutional kernel is a strip-shaped convolutional kernel; perform feature extraction on the input data based on the first-layer convolutional kernel and the second-layer convolutional kernel to generate a three-dimensional feature image; obtain the confidence scores of the three-dimensional feature image belonging to multiple image flipping categories based on the convolutional neural network, and determine the image flipping category with the highest confidence score as the target flipping category, wherein the multiple image flipping categories include no flipping, horizontal axis mirror flipping, vertical axis mirror flipping, counterclockwise flipping, and clockwise flipping; The flipping module is used to flip the document image in reverse according to the target flipping category to obtain the target image.

9. A training device for an image flipping recognition module, characterized in that, The image flipping recognition model applied to any one of claims 1-3 includes: The fourth generation module is used to crop the sample image according to the random cropping algorithm and generate a preset first number of cropped sub-images; The filtering module is used to filter cropped sub-images based on whether they contain text information, and obtain valid images, wherein the valid images contain text information. The fifth generation module is used to mirror and flip valid images according to preset flipping rules, generating flipped images and preset flip labels; The recognition module is used to identify flipped images using an image flipping recognition model and determine the target flipped label. The execution module is used to train the image flipping recognition model based on the preset flipping label when the target flipping label is inconsistent with the preset flipping label.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the program implements the steps of the method described in any one of claims 1-3.

11. An electronic device, characterized in that, include: A memory on which computer programs are stored; A processor for executing the computer program in the memory to implement the steps of the method according to any one of claims 1-3.

Images