Image processing method and device

By using the first processing model to perform angle correction and the second processing model to perform distortion correction in image processing, the problem that the prior art cannot process deflection and distortion images is solved, and accurate extraction and efficient processing of image information are achieved.

CN113554558BActive Publication Date: 2024-11-01BEIJING KINGSOFT DIGITAL ENTERTAINMENT CO LTD +1
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202010340730.1
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2020-04-26
Publication Date
2024-11-01
Estimated Expiration
2040-04-26

AI Technical Summary

Technical Problem

The prior art cannot effectively extract and correct information when processing unconventional images, especially images with deflection and distortion, resulting in the inability to accurately extract important information in the image.

Method used

An image processing method is adopted, by inputting the original image into the first processing model to obtain the correction angle for rotation correction, and then inputting the intermediate image into the second processing model for distortion correction, and correcting the distortion information using the generative adversarial network model, and finally Get the target image.

Benefits of technology

Accurate angle correction and information correction of deflected and distorted images are achieved, the efficiency and accuracy of image processing are improved, and the accuracy of subsequent layout analysis is ensured.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN113554558B_ABST
    Figure CN113554558B_ABST
Patent Text Reader

Abstract

The present application provides an image processing method and device, the method comprising: receiving an original image; inputting the original image into a first processing model, obtaining a correction angle output by the first processing model, and determining an intermediate image; inputting the intermediate image into a second processing model, obtaining the second processing model for distortion correction processing to determine a target image, thereby rotating the original image using the correction angle output by the first processing model to obtain an intermediate image, without relying on existing complex rules for processing the original image, and accurately correcting the original image that is deviated from the correct position, and then inputting the intermediate image into the second processing model to generate a target image, which can accurately correct the angle of the original image and correct the distortion of information in the intermediate image.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to an image processing method and apparatus, a computing device, and a computer-readable storage medium. Background Art

[0002] In the process of digitizing paper documents, it is necessary to extract important information from document images. Before extracting important information from document images, it is necessary to analyze the layout of the document images. The layout analysis usually includes the detection of text lines, pictures, and tables.

[0003] Before the layout analysis, it is usually necessary to process the document image to make it meet the requirements of the layout analysis. In the prior art, the layout analysis generally adopts the traditional image processing algorithm, such as edge detection, projection and other operations to perform element cutting and other processing on the document image. The specific method includes: first, using the image morphology algorithm to perform edge detection, expansion and corrosion on the original image to obtain an over-segmented image. Then, the edge of the original image is re-extracted using a linear filter, and the fragmented areas are preliminarily merged. Finally, the relative position information of the fragmented areas is used to merge the preliminarily merged areas again.

[0004] The image processing algorithms in the prior art are overly dependent on rules. When processing unconventional images, such as deflected images or images with distortion, the existing image processing algorithms are unable to process unconventional images, resulting in the inability to extract important information from the unconventional images. Summary of the invention

[0005] In view of this, embodiments of the present application provide an image processing method and apparatus, a computing device, and a computer-readable storage medium to address the technical deficiencies in the prior art.

[0006] The present application discloses an image processing method, including:

[0007] receiving the original image;

[0008] Inputting the original image into a first processing model, obtaining a correction angle output by the first processing model, and determining an intermediate image;

[0009] The intermediate image is input into a second processing model, and a distortion correction process is performed on the intermediate image to determine a target image.

[0010] Optionally, the method further includes: analyzing the target image to obtain layout information.

[0011] Optionally, the first processing model is a classification model;

[0012] Obtaining the correction angle output by the first processing model and determining an intermediate image includes:

[0013] Inputting the original image into the classification model, and determining a correction angle of the original image output by the classification model;

[0014] Based on the correction angle of the original image, the original image is rotated to obtain an intermediate image.

[0015] Optionally, determining a correction angle of the original image output by the classification model includes:

[0016] When it is determined that the deflection angles corresponding to at least two pieces of information in the intermediate image are different, the deflection angle corresponding to the information with the largest number of characters is output as the correction angle of the original image according to the number of characters of the at least two pieces of information.

[0017] Optionally, the layout information includes at least one of graphic information, table information, formula information and text information;

[0018] Analyze the target image and obtain layout information, including:

[0019] Detecting graphic information, table information, formula information and text information in the target image by a third processing model;

[0020] Obtaining relative position information and position area size of the graphic information, table information, formula information and text information in the target image;

[0021] The layout information of the target image is determined according to the relative position information and position area size of the graphic information, table information, formula information and text information in the target image.

[0022] Optionally, determining the layout information of the target image according to the relative position information and position area size of the graphic information, table information, formula information and text information in the target image includes:

[0023] Merging adjacent text information to obtain merged text information;

[0024] Based on the relative position information and position area size of the graphic information, table information, formula information and merged text information in the target image, the layout information of the target image is determined.

[0025] Optionally, the intermediate image is input into a second processing model, and a distortion correction process is performed on the intermediate image to determine a target image, including:

[0026] The intermediate image is input into a generative adversarial network model, and the target image output by the generative adversarial network model is obtained by performing distortion correction processing on the intermediate image to determine the target image.

[0027] The present application also discloses an image processing device, including:

[0028] A receiving module, configured to receive an original image;

[0029] A determination module is configured to input the original image into a first processing model, obtain a correction angle output by the first processing model, and determine an intermediate image;

[0030] The acquisition module is configured to input the intermediate image into a second processing model and perform distortion correction processing on the intermediate image to determine a target image.

[0031] Optionally, the device further includes: an analysis module configured to analyze the target image and obtain layout information.

[0032] An embodiment of the present application discloses a computing device, including a memory, a processor, and computer instructions stored in the memory and executable on the processor, wherein the processor implements the steps of the image processing method described above when executing the instructions.

[0033] An embodiment of the present application discloses a computer-readable storage medium storing computer instructions, which implement the steps of the image processing method described above when executed by a processor.

[0034] The present application provides an image processing method and device, which inputs the original image into a first processing model, and uses the correction angle output by the first processing model to rotate the original image to obtain an intermediate image. Without relying on the existing complex rules for processing the original image, the original image that deviates from the correct position can be accurately corrected, and then the intermediate image is input into a second processing model to obtain a target image output by the second processing model. The target image is an image obtained by correcting the distortion of the information in the intermediate image, which solves the problem in the prior art that the distortion of the information in the image cannot be corrected, and can accurately correct the angle of the original image and the distortion of the information in the intermediate image. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 is a schematic diagram of the structure of a computing device according to an embodiment of the present application;

[0036] Figure 2 is a flowchart of an image processing method according to a first embodiment of the present application;

[0037] Figure 3is a flowchart of an image processing method according to a second embodiment of the present application;

[0038] Figure 4 is a flowchart of an image processing method according to a third embodiment of the present application;

[0039] Figure 5a is a schematic diagram of the original image received in this application;

[0040] Figure 5b is a schematic diagram of an intermediate image for performing angle correction on the original image in the present application;

[0041] Figure 5c is a schematic diagram of a target image for which distortion correction is performed on the intermediate image in the present application;

[0042] Figure 5d is a schematic diagram of the position and size of the table information and text information in the target image in this application;

[0043] Figure 5e is a schematic diagram of a target image after merging text information in this application;

[0044] Figure 6 It is a schematic diagram of the structure of the classification model in this application;

[0045] Figure 7 It is a schematic diagram of the structure of the generative adversarial network model in this application;

[0046] Figure 8 It is a schematic diagram of the structure of the YOLOv3 model in this application;

[0047] Fig. 9 It is a schematic diagram of the structure of an image processing device according to an embodiment of the present application. DETAILED DESCRIPTION

[0048] Many specific details are described in the following description to facilitate a full understanding of the present application. However, the present application can be implemented in many other ways than those described herein, and those skilled in the art can make similar generalizations without violating the connotation of the present application, so the present application is not limited by the specific implementation disclosed below.

[0049] The terms used in one or more embodiments of this specification are only for the purpose of describing specific embodiments, and are not intended to limit one or more embodiments of this specification. The singular forms of "a", "said" and "the" used in one or more embodiments of this specification and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used in one or more embodiments of this specification refers to and includes any or all possible combinations of one or more associated listed items.

[0050] It should be understood that although the terms first, second, etc. may be used to describe various information in one or more embodiments of this specification, this information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of one or more embodiments of this specification, the first may also be referred to as the second, and similarly, the second may also be referred to as the first. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0051] First, the terms involved in one or more embodiments of the present invention are explained.

[0052] Original image: An image with deflection and / or distortion problems.

[0053] Intermediate image: The image obtained after angle correction of the original image.

[0054] Target image: The image obtained after the intermediate image is distorted.

[0055] First processing model: Process the original image to obtain the correction angle corresponding to the original image. The first processing model can be a trained deep residual network.

[0056] The second processing model: processes the intermediate image and corrects the distortion of the intermediate image to obtain the target image. The second processing model treats the correction of information distortion in the intermediate image as a generation problem. The second processing model can be a trained generative adversarial network model.

[0057] The third processing model is used to extract layout information from the target image. The third processing model may be a trained target detection model. Specifically, the target detection model may be a third version of the YOLO model.

[0058] Layout information: including graphic information, table information, formula information, text information, etc.

[0059] Deep residual network (ResNet): The deep residual network is easy to optimize and can improve accuracy by increasing the depth. The residual block inside it uses skip connections to alleviate the gradient vanishing problem caused by increasing the depth in the deep neural network.

[0060] Generative Adversarial Networks (GAN): is a deep learning model and one of the most promising methods for unsupervised learning on complex distributions. The model produces fairly good outputs through the mutual game learning of the two modules of the Generative Model and the Discriminative Model in the framework.

[0061] YOLO: It is an end-to-end target detection model that can output categories, confidence levels, and coordinate positions through one model.

[0062] Convolutional layer: extracts key features of the input image through convolution operations.

[0063] Deconvolution layer: It is the inverse operation of the convolution operation, which is the process of converting the input features into an image.

[0064] Pooling layer: It compresses the results of the convolutional layer to obtain more important features, while also effectively controlling overfitting.

[0065] Fully connected layer: Integrates the highly abstract features after multiple convolutions, and then normalizes them to output the formula features corresponding to the formula image.

[0066] In the present application, a method and apparatus for image processing, a computing device, and a computer-readable storage medium are provided, which are described in detail one by one in the following embodiments.

[0067] Figure 1 1 is a block diagram showing a structure of a computing device 100 according to an embodiment of the present specification. The components of the computing device 100 include but are not limited to a memory 110 and a processor 120. The processor 120 is connected to the memory 110 via a bus 130, and a database 150 is used to store data.

[0068] The computing device 100 also includes an access device 140 that enables the computing device 100 to communicate via one or more networks 160. Examples of these networks include a public switched telephone network (PSTN), a local area network (LAN), a wide area network (WAN), a personal area network (PAN), or a combination of communication networks such as the Internet. The access device 140 may include one or more of any type of network interface (e.g., a network interface card (NIC)) whether wired or wireless, such as an IEEE 802.11 wireless local area network (WLAN) wireless interface, a World Wide Interoperability for Microwave Access (Wi-MAX) interface, an Ethernet interface, a universal serial bus (USB) interface, a cellular network interface, a Bluetooth interface, a near field communication (NFC) interface, and the like.

[0069] In one embodiment of the present specification, the above components of the computing device 100 and Figure 1 Other components not shown in the figure may also be connected to each other, for example, via a bus. It should be understood that Figure 1 The computing device structure block diagram shown is only for the purpose of illustration, and is not intended to limit the scope of this specification. Those skilled in the art can add or replace other components as needed.

[0070] The computing device 100 may be any type of stationary or mobile computing device, including a mobile computer or mobile computing device (e.g., a tablet computer, a personal digital assistant, a laptop computer, a notebook computer, a netbook, etc.), a mobile phone (e.g., a smart phone), a wearable computing device (e.g., a smart watch, smart glasses, etc.), or other types of mobile devices, or a stationary computing device such as a desktop computer or PC. The computing device 100 may also be a mobile or stationary server.

[0071] The processor 120 may execute Figure 2 Steps in the method shown. Figure 2 is a schematic flow chart showing an image processing method according to the first embodiment of the present application, including steps 202 to 206 .

[0072] Step 202: Receive the original image.

[0073] The original image is a paper document photographed by a terminal device connected to a camera or a terminal device with a built-in camera, or a picture given by a user is received as the original image.

[0074] The content in the original image may include at least one of picture information, table information, formula information and text information.

[0075] The original image is an unconventional image, that is, the original image is deflected and the content in the original image is distorted and deformed, ensuring that the following steps can be carried out normally to achieve correction of the original image.

[0076] Step 204: input the original image into the first processing model, obtain the correction angle output by the first processing model, and determine the intermediate image.

[0077] Determining the correction angle of the original image is processed as a classification problem, the first processing model can be a classification model, the input to the first processing model is the original image, the output of the first processing model is the rotation angle of the original image, and the rotation angle of the original image is the correction angle. The classification model can be a deep residual network.

[0078] The information in the original image is deflected, or in other words, the content in the original image deviates from the correct position. The original image is input into the classification model to determine the correction angle of the original image output by the classification model.

[0079] The correction angle output by the first processing model is used to rotate the original image to obtain an intermediate image. That is, the deflected content in the original image is rotated to the correct position, thereby correcting the deflection problem of the original image. The original image can be accurately corrected without relying on the existing complex rules for processing the original image, and the processing efficiency of the original image is high.

[0080] Step 206: Input the intermediate image into a second processing model, and perform distortion correction processing on the intermediate image to determine a target image.

[0081] The information in the intermediate image is distorted. The intermediate image is input into a second processing model to obtain a target image output by the second processing model. The target image is the image obtained by correcting the distortion of the information in the intermediate image.

[0082] The existing complex rules for image processing cannot achieve the distortion correction of information in the image. In this step, the target image for distortion correction of the intermediate image output by the second processing model is directly obtained, which can not only accurately correct the distortion of information in the intermediate image, but also has high processing efficiency for the intermediate image.

[0083] Optionally, the image processing method of this embodiment further includes: S208: analyzing the target image to obtain layout information.

[0084] Among them, layout information includes graphic information, table information, formula information and text information.

[0085] Specifically, step S208 includes: analyzing the position and size of the graphic information, table information, formula information and text information in the target image in the target image to improve the accuracy of obtaining the layout information in the target image.

[0086] In this embodiment, the original image is input into the first processing model, and the correction angle output by the first processing model is used to rotate the original image to obtain an intermediate image. Without relying on the existing complex rules for processing the original image, the original image that deviates from the correct position can be accurately corrected. The intermediate image is then input into the second processing model to obtain the target image output by the second processing model. The target image is the image obtained by correcting the distortion of the information in the intermediate image, which solves the problem in the prior art that the information in the image cannot be corrected for distortion. Not only can the angle of the original image and the distortion of the information in the intermediate image be accurately corrected, ensuring that the subsequent layout analysis can be carried out accurately and quickly, but also the efficiency of processing the original image to obtain the target image is high by using the first processing model and the second processing model to process the image.

[0087] Figure 3 3 is a schematic flow chart showing an image processing method according to the second embodiment of the present application, including steps 302 to 314. The first processing model is a classification model, and the second processing model is a generative adversarial network model.

[0088] Step 302: Receive the original image.

[0089] The original image is deflected and the content in the original image is distorted and deformed, ensuring that the following steps can be carried out normally to achieve correction of the original image.

[0090] Step 304: Input the original image into the classification model. When it is determined that the deflection angles corresponding to at least two parts of information in the intermediate image are different, output the angle corresponding to the information with the largest number of characters as the correction angle of the original image based on the number of characters of the at least two parts of information.

[0091] The intermediate image includes at least two parts of information whose corresponding deflection angles are different. For example, the intermediate image includes two parts of information, the clockwise deflection angle of all lines of text information in the upper half of the intermediate image is 20°, and the clockwise deflection angle of the two lines of text information in the lower half is 30°, and the number of characters in all lines of text information in the upper half is more than the number of characters in the two lines of text information in the lower half, then the classification model outputs the angle of 20° corresponding to the text information in all lines of the upper half of the original image as the correction angle.

[0092] By using the angle corresponding to the information with the largest number of characters in the original image output by the classification model as the correction angle, it is ensured that as much information in the intermediate image as possible can be corrected, so that as much layout information in the target image as possible can be obtained in the following steps.

[0093] Step 306: Based on the correction angle of the original image, the original image is rotated to obtain an intermediate image.

[0094] The correction angle output by the classification model is used to rotate the original image to obtain an intermediate image, that is, the deflected content in the original image is rotated to the correct position, so as to correct the deflection problem of the original image. The original image can be accurately corrected without relying on the existing complex rules for processing the original image, and the processing efficiency of the original image is high.

[0095] Step 308: Input the intermediate image into a generative adversarial network model, obtain the target image output by the generative adversarial network model, and perform distortion correction processing on the intermediate image to determine the target image.

[0096] Specifically, the target image is the image after the information in the intermediate image is distorted and corrected, thereby improving the accuracy of the distortion correction of the target image.

[0097] Step 310: Detecting graphic information, table information, formula information and text information in the target image through a third processing model.

[0098] The third processing model may be a target detection model. Specifically, the target detection model may be a third version of the YOLO model, which improves the efficiency of detecting and obtaining layout information in the target image.

[0099] The graphic information is the information in the target image that uses points, lines, symbols, etc. to depict the geometric features, shapes, positions and sizes of objects; the table information is the information formed by drawing grids in the target image according to the required content items and filling in words or numbers in the grids; the formula information is the information formed by formulas expressed in mathematical symbols and having certain relationships between various quantities; the text information refers to a piece of text composed of multiple words or numbers.

[0100] Step 312: Obtain the relative position information and position area size of the graphic information, table information, formula information and text information in the target image.

[0101] For example, the acquired graphic information is located at the top position of the target image and has a length*width of 15cm*5cm, the table information is located at the upper middle position of the target image and has a length*width of 10cm*3cm, the formula information is located at the middle position of the target image and has a length*width of 5cm*1cm, and the text information is located at the lower position of the target image and has a length*width of 15cm*10cm, to ensure that the layout information can be quickly acquired in the following steps.

[0102] Step 314: Determine the layout information in the target image based on the relative position information and position area size of the graphic information, table information, formula information and text information in the target image.

[0103] In this embodiment, the original image is input into the first processing model, and the correction angle output by the first processing model is used to rotate the original image to obtain an intermediate image. Without relying on the existing complex rules for processing the original image, the original image that deviates from the correct position can be accurately corrected. The intermediate image is then input into the second processing model to obtain the target image output by the second processing model. The target image is the image obtained by correcting the distortion of the information in the intermediate image, which solves the problem in the prior art that the information in the image cannot be corrected for distortion. Not only can the angle of the original image and the distortion of the information in the intermediate image be accurately corrected, ensuring that the subsequent layout analysis can be carried out accurately and quickly, but also the efficiency of processing the original image to obtain the target image is high by using the first processing model and the second processing model to process the image.

[0104] Figure 4 is a schematic flow chart showing an image processing method according to the third embodiment of the present application, including steps 402 to 416. Figure 5a to Figure 5e This embodiment is described in further detail.

[0105] Step 402: Receive the original image.

[0106] Figure 5a The received original image is shown. The original image is deflected as a whole and deformed and distorted at the lower right part. Receiving the original image facilitates correction of the original image in the following steps.

[0107] Step 404: Input the original image into the first processing model, input the original image into the first processing model, and determine the correction angle of the original image output by the first processing model.

[0108] Will Figure 5a The original image shown is input into the first processing model, and it is assumed that the correction angle of the original image output by the first processing model is determined to be 20°.

[0109] The first processing model may be a trained classification model. Specifically, the classification model may be a deep residual network.

[0110] Specifically, see Figure 6 As shown, the classification model includes an input layer, an intermediate layer and a fully connected layer. The intermediate layer is the first convolution layer, the second pooling layer, the third to the thirty-second convolution layers, the thirty-third pooling layer, and the thirty-fourth fully connected layer connected in sequence. The residual structure in the classification model is that a short-circuit connection is added between every two layers of the third to the thirty-second convolution layers, that is, the output information of the third convolution layer is directly used as the input of the fifth convolution layer, and the output information of the fifth convolution layer is directly used as the input of the seventh convolution layer, and so on. The convolution layer includes multiple short-circuit connections.

[0111] Figure 5a The original image shown is input to the first convolution layer. The convolution kernel size of the first convolution layer is 3x3, and the number of convolution kernels is 32. The result of convolution calculation of the original image by 32 convolution kernels of size 3x3 is input to the second pooling layer; the second pooling layer compresses the result of the first convolution layer to obtain more important features, and the compressed features are input to the third convolution layer. The third convolution layer convolutions the features output by the second maximum pooling layer by 32 convolution kernels of size 5x5, and inputs the result to the fourth maximum pooling layer. And so on. Finally, the 34th fully connected layer integrates the highly abstract features after multiple convolutions, and then can be normalized to finally output the low-dimensional features corresponding to the original image, and convert the low-dimensional features into the correction angle corresponding to the original image.

[0112] During the classification model training process, the image with deflection defects is used as the sample image, the correct correction angle corresponding to the sample image is used as the supervision information, and the sample image and the correct correction angle corresponding to the sample image are used as the input of the classification model. The classification model is trained until the classification model converges, that is, the loss value output by the classification model is lower than the preset stability threshold, and the training of the classification model is completed.

[0113] The correction angle output by the first processing model is used to rotate the original image to obtain an intermediate image. That is, the deflected content in the original image is rotated to the correct position, thereby correcting the deflection problem of the original image. The original image can be accurately corrected without relying on the existing complex rules for processing the original image, and the processing efficiency of the original image is high.

[0114] Step 406: Based on the correction angle of the original image, the original image is rotated to obtain an intermediate image.

[0115] Based on the correction angle of 20° of the original image, the original image is rotated 20° counterclockwise to obtain Figure 5b The middle image is shown.

[0116] The correction angle output by the first processing model is used to rotate the original image to obtain an intermediate image, that is, the deflected content in the original image is rotated to the correct position, thereby correcting the deflection problem of the original image.

[0117] Step 408: Input the intermediate image into a second processing model, and perform distortion correction processing on the intermediate image to determine a target image.

[0118] Will Figure 5b The intermediate image shown is input into the second processing model to obtain the target image output by the second processing model. Figure 5c The target image shown is an image obtained by correcting the distortion of the lower right portion of the intermediate image.

[0119] The information in the intermediate image is distorted. The intermediate image is input into a second processing model to obtain a target image output by the second processing model. The target image is the image obtained by correcting the distortion of the information in the intermediate image.

[0120] The information distortion correction in the intermediate image is processed as a generation problem, and the second processing model can be a trained generative adversarial network model. The intermediate image is input into the generative adversarial network model to obtain a target image output by the generative adversarial network model, and the target image is the image after the information in the intermediate image is distorted and corrected.

[0121] During the training process of the generative adversarial network model, an image with distortion and deformation defects is used as a sample image, and a correct image corresponding to the sample image is used as supervision information. The sample image and the correct image corresponding to the sample image are input into the generative adversarial network model for training until the generative adversarial network model converges, that is, the loss value output by the generative adversarial network model is lower than a preset stability threshold, and the training of the generative adversarial network model is completed.

[0122] Specifically, Figure 7 As shown in Figure 1, the generative adversarial network model includes a generative model and a discriminative model, and both the generative model and the discriminative model can use convolutional neural networks.

[0123] The generative model includes a fully connected layer, a first deconvolution layer, a second deconvolution layer, a third deconvolution layer and an output layer which are connected in sequence. The fully connected layer highly abstracts the sample image input from the input layer to obtain features. The features output by the fully connected layer are subjected to inverse convolution operations through three deconvolution layers, and the highly abstract features are input to the output layer. The output layer outputs a new sample image, and the new sample image output by the generative model is used as a false sample.

[0124] The discriminant model includes an input layer, a first convolutional layer, a second convolutional layer, a third convolutional layer, a fully connected layer and an output layer which are connected in sequence.

[0125] The new sample image and the correct image output by the generative model are input into the input layer of the discriminative model. The features after three convolution operations are input into the fully connected layer. The fully connected layer integrates the highly abstract features after three convolutions. The normalized features are output through the output layer to output the discriminative results, so as to train the generative adversarial network model.

[0126] After the generative model of the generative adversarial network model collects the potential distribution of the real sample image, the generative model generates a new sample image as a fake sample. The discriminant model is a binary classification model. The real sample image and the fake sample generated by the generative model are input into the discriminant model, and the real sample image and the fake sample generated by the generative model are discriminated by the discriminant model.

[0127] The existing complex rules for image processing cannot achieve the distortion correction of information in the image, while in this step, the target image for distortion correction of the intermediate image output by the second processing model is directly obtained, which can not only accurately correct the distortion of information in the intermediate image, but also has high processing efficiency for the intermediate image.

[0128] Step 410: Detecting graphic information, table information, formula information and text information in the target image through a third processing model.

[0129] The target image is detected by a third processing model, and the position and size of the graphic information, table information, formula information and text information in the target image are obtained. Figure 5c The target image shown is input into the third processing model, and the position and size of the table information and text information output by the third processing model in the target image are obtained, thereby improving the processing efficiency of the layout information in the target image and improving the accuracy of the layout information obtained in the target image.

[0130] The third processing model may be a target detection model. Specifically, the target detection model may be a trained YOLOv3, that is, the third version of YOLO.

[0131] Figure 8 The figure shows a schematic diagram of the structure of YOLOv3, including an input layer, a convolutional layer, a darknet-53 layer, a feature interaction layer, and an output layer.

[0132] The target image is input to the convolution layer through the input layer, and after the convolution operation, the features of 32 channels with a size of 416*416 are output, and the features are input to the darknet-53 layer.

[0133] The darknet-53 layer includes the 0th to 74th layers, a total of 53 convolutional layers, and the rest are convolutional layers with residual structures. The darknet-53 layer is the main network structure extracted by YOLOv3. The structure uses a series of 3*3 and 1*1 convolutional layers to extract features. Among them, the convolutional layer with a residual structure solves the gradient diffusion or gradient explosion phenomenon of the network, changes the layer-by-layer training of the deep neural network to stage-by-stage training, and divides the deep neural network into several sub-segments. Each small segment contains a relatively shallow number of network layers, and then uses a short-circuit connection method to make each small segment trained based on the residual. Each small segment learns a part of the total difference (total loss), and finally reaches an overall smaller loss value. At the same time, it controls the propagation of the gradient well to avoid situations that are not conducive to training, such as gradient disappearance or explosion.

[0134] The feature interaction layer is the convolution layer from the 75th to the 105th layer, which is divided into three scales, namely the minimum scale layer, the medium scale layer and the large scale layer. In each scale, local feature interaction is achieved by means of convolution kernels, which is similar to the fully connected layer, but the local feature interaction between features is achieved by means of convolution kernels (the fully connected layer achieves global feature interaction). Classification and position regression are performed after the output of each scale layer, and finally the output layer outputs the position and size of the graphic information, table information, formula information and text information in the target image.

[0135] The graphic information is the information in the target image that uses points, lines, symbols, etc. to depict the geometric features, shapes, positions and sizes of objects; the table information is the information formed by drawing grids in the target image according to the required content items and filling in words or numbers in the grids; the formula information is the information formed by formulas expressed in mathematical symbols and having certain relationships between various quantities; the text information refers to a piece of text composed of multiple words or numbers.

[0136] Step 412: Obtain the relative position information and position area size of the graphic information, table information, formula information and text information in the target image.

[0137] Figure 5dThe position and size of the output table information and text information in the target image are shown. The table information is located in the middle of the target image and has a length and width of 12cm*5cm. The text information is located in the upper and lower positions of the target image and each line of text information is marked. The length and width of the first line of text information are 11cm*0.6cm, the length and width of the second line of text information are 12cm*0.6cm, and the length and width of the third line of text information are 12cm*0.6cm. The size of the text information is not repeated below.

[0138] The target image is input into the third processing model, and the third processing model is directly obtained to detect the graphic information, table information, formula information and text information in the target image, that is, the graphic information, table information, formula information and text information in the detected target image will be respectively marked with their respective positions and sizes to ensure that the text information can be merged in the following steps.

[0139] Step 414: Merge the adjacent text lines containing the text information to obtain merged text information.

[0140] merge Figure 5d The text lines where the adjacent text information shown in the figure are located obtain the merged text information, Figure 5e The target image after the text information is merged is shown, that is, the text information in the upper part of the target image is merged together, and the text information in the lower part of the target image is merged together.

[0141] The graphic information, table information, formula information and text information in the detected target image will be marked with their respective relative position information and position area size. Simply put, the graphic information, table information, formula information and text information will be framed separately.

[0142] Each line of the text information will be marked separately, which can be simply understood as each line of the text information being framed separately, and the adjacent text lines of the text information being merged to obtain merged text information. After the layout information in the target image is determined in the following steps, the subsequent text recognition of the layout information is facilitated.

[0143] Step 416: Determine the layout information in the target image based on the relative position information and position area size of the graphic information, table information, formula information and merged text information in the target image.

[0144] Based on the position and size of the table information and the merged text information in the target image, the layout information in the target image is determined, which can not only accurately correct the angle of the original image and correct the distortion of the information in the intermediate image, ensuring that the third processing model can accurately and quickly detect the layout of the target image, but also improve the efficiency of processing the original image to obtain the target image, and finally obtain Figure 5e The results of the layout analysis shown ensure that the text recognition of the target image can be accurately performed in the later stage.

[0145] The present application inputs the original image into the first processing model, and uses the correction angle output by the first processing model to rotate the original image to obtain an intermediate image. Without relying on the existing complex rules for processing the original image, the original image that deviates from the correct position can be accurately corrected. The intermediate image is then input into the second processing model to obtain the target image output by the second processing model, and the target image is the image after the information in the intermediate image is distorted and corrected. Finally, the target image is input into the third processing model to obtain the relative position information and position area size of the graphic information, table information, formula information and text information in the target image, so as to ensure that the layout information of the target image can be quickly and accurately obtained, and the text lines where the adjacent text information is located are merged to obtain the merged text information, so as to facilitate the later text recognition of the text information in the layout information. Not only can the angle of the original image be accurately corrected and the distortion of the information in the intermediate image be corrected, but also the third processing model can be ensured to accurately and quickly detect the layout of the target image, and the efficiency of processing the original image to obtain the target image is high.

[0146] Fig. 9 A schematic structural diagram of an image processing device is shown, comprising:

[0147] The receiving module 902 is configured to receive an original image;

[0148] A determination module 904 is configured to input the original image into a first processing model, obtain a correction angle output by the first processing model, and determine an intermediate image;

[0149] The acquisition module 906 is configured to input the intermediate image into a second processing model and perform a distortion correction process on the intermediate image to determine a target image.

[0150] Optionally, the device further includes: an analysis module configured to analyze the target image and obtain layout information.

[0151] The determination module 904 is further configured to: input the original image into a classification model, and determine a correction angle of the original image output by the classification model;

[0152] Based on the correction angle of the original image, the original image is rotated to obtain an intermediate image.

[0153] The determination module 904 is further configured to: when it is determined that the deflection angles corresponding to at least two parts of information in the intermediate image are different, output the angle corresponding to the information with the largest number of characters as the correction angle of the original image according to the number of characters of the at least two parts of information.

[0154] The information in the target image includes at least one of graphic information, table information, formula information and text information;

[0155] The analysis module is further configured to: detect graphic information, table information, formula information and text information in the target image through a third processing model;

[0156] Obtaining the position and size of the graphic information, table information, formula information and text information in the target image;

[0157] The layout information in the target image is determined according to the relative position information and position area size of the graphic information, table information, formula information and text information in the target image.

[0158] The analysis module is further configured to:

[0159] Merging adjacent text information to obtain merged text information;

[0160] Based on the relative position information and position area size of the graphic information, table information, formula information and merged text information in the target image, the layout information in the target image is determined.

[0161] The analysis module is further configured to input the intermediate image into a generative adversarial network model, obtain the target image output by the generative adversarial network model, and perform distortion correction processing on the intermediate image to determine the target image.

[0162] In this embodiment, the original image is input into the first processing model, and the correction angle output by the first processing model is used to rotate the original image to obtain an intermediate image. Without relying on the existing complex rules for processing the original image, the original image that deviates from the correct position can be accurately corrected. The intermediate image is then input into the second processing model to obtain the target image output by the second processing model. The target image is the image obtained by correcting the distortion of the information in the intermediate image, which solves the problem in the prior art that the information in the image cannot be corrected for distortion. Not only can the angle of the original image and the distortion of the information in the intermediate image be accurately corrected, ensuring that the subsequent layout analysis can be carried out accurately and quickly, but also the efficiency of processing the original image to obtain the target image is high by using the first processing model and the second processing model to process the image.

[0163] An embodiment of the present application also provides a computing device, including a memory, a processor, and computer instructions stored in the memory and executable on the processor, wherein the processor implements the steps of the image processing method as described above when executing the instructions.

[0164] An embodiment of the present application further provides a computer-readable storage medium storing computer instructions, which, when executed by a processor, implement the steps of the image processing method as described above.

[0165] The above is a schematic scheme of a computer-readable storage medium of this embodiment. It should be noted that the technical scheme of the storage medium and the technical scheme of the above-mentioned image processing method belong to the same concept, and the details not described in detail in the technical scheme of the storage medium can be referred to the description of the technical scheme of the above-mentioned image processing method.

[0166] The computer instructions include computer program codes, which may be in source code form, object code form, executable files or some intermediate forms, etc. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium, etc. It should be noted that the content contained in the computer-readable medium may be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media do not include electric carrier signals and telecommunication signals.

[0167] It should be noted that, for the above-mentioned method embodiments, for the sake of simplicity of description, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the described action sequence, because according to the present application, certain steps can be performed in other sequences or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present application.

[0168] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0169] The preferred embodiments of the present application disclosed above are only used to help explain the present application. The optional embodiments do not describe all the details in detail, nor do they limit the invention to the specific implementation methods described. Obviously, many modifications and changes can be made according to the content of this specification. This specification selects and specifically describes these embodiments in order to better explain the principles and practical applications of the present application, so that those skilled in the art can understand and use the present application well. The present application is limited only by the claims and their full scope and equivalents.

Claims

1. A method for image processing, characterized in that: include: Receiving an original image, wherein the original image is a document image and includes picture information, table information, formula information and text information, and the original image has information deflection and the image content is distorted; Input the original image into a first processing model, obtain the correction angle output by the first processing model, and determine an intermediate image, wherein the first processing model is a classification model, and when it is determined that the intermediate image has at least two pieces of information having different deflection angles respectively corresponding to them, the correction angle is the deflection angle corresponding to the information having the largest number of characters in the at least two pieces of information; The intermediate image is input into a second processing model, and distortion correction processing is performed on the deflected information in the intermediate image to determine the target image, wherein the second processing model is a generative adversarial network model.

2. The method according to claim 1, characterized in that Also includes: Analyze the target image and obtain layout information.

3. The method according to claim 1, characterized in that The first processing model is a classification model; Obtaining the correction angle output by the first processing model and determining an intermediate image includes: Inputting the original image into the classification model, and determining a correction angle of the original image output by the classification model; Based on the correction angle of the original image, the original image is rotated to obtain an intermediate image.

4. The method according to claim 2, characterized in that: The layout information includes at least one of graphic information, table information, formula information and text information; Analyze the target image and obtain layout information, including: Detecting graphic information, table information, formula information and text information in the target image by a third processing model; Obtaining relative position information and position area size of the graphic information, table information, formula information and text information in the target image; The layout information of the target image is determined according to the relative position information and position area size of the graphic information, table information, formula information and text information in the target image.

5. The method according to claim 4, characterized in that Determining the layout information of the target image according to the relative position information and the size of the position area of ​​the graphic information, table information, formula information and text information in the target image, including: Merging adjacent text information to obtain merged text information; Based on the relative position information and position area size of the graphic information, table information, formula information and merged text information in the target image, the layout information of the target image is determined.

6. The method according to claim 1, characterized in that Inputting the intermediate image into a second processing model, and performing distortion correction processing on the intermediate image to determine a target image, comprising: The intermediate image is input into a generative adversarial network model, and the target image output by the generative adversarial network model is obtained by performing distortion correction processing on the intermediate image to determine the target image.

7. An image processing device, characterized in that: include: A receiving module is configured to receive an original image, wherein the original image is a document image and includes picture information, table information, formula information and text information, and the original image has information deflection and the image content is distorted; a determination module configured to input the original image into a first processing model, obtain a correction angle output by the first processing model, and determine an intermediate image, wherein the first processing model is a classification model, and when it is determined that the intermediate image has at least two pieces of information having different deflection angles respectively corresponding to them, the correction angle is the deflection angle corresponding to the information having the largest number of characters in the at least two pieces of information; The acquisition module is configured to input the intermediate image into a second processing model, and perform distortion correction processing on the deflected information in the intermediate image to determine the target image, wherein the second processing model is a generative adversarial network model.

8. The device according to claim 7, characterized in that The device also includes: The analysis module is configured to analyze the target image and obtain layout information.

9. A computing device comprising a memory, a processor, and computer instructions stored in the memory and executable on the processor, characterized in that: When the processor executes the instructions, the steps of the method according to any one of claims 1 to 6 are implemented.

10. A computer-readable storage medium storing computer instructions, characterized in that: When the instruction is executed by a processor, the steps of the method described in any one of claims 1 to 6 are implemented.