Image correction method and device

By calculating the preliminary and intermediate offset fields of the image and using the remapping algorithm for image correction, the problem of expensive and cumbersome use of the scanner is solved, and the readability of the image and the OCR recognition effect are improved.

CN118887673APending Publication Date: 2024-11-01ZHUHAI KINGSOFT OFFICE SOFTWARE +2
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
CN202410907008.X
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-08
Publication Date
2024-11-01

AI Technical Summary

Technical Problem

Existing scanners are expensive and cumbersome, which limits the application scenarios of image correction, especially when taking documents and documents, resulting in image distortion affecting image readability and OCR recognition effect.

Method used

An image correction method is adopted to calculate the final offset field by obtaining the initial offset field and the intermediate offset field of the image to be corrected, and two corrections are performed using the remapping algorithm to obtain the corrected image, avoiding the use of a scanner.

Benefits of technology

The correction of images is achieved, the readability of images and the OCR recognition effect are improved, and the need to use expensive scanners is avoided.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118887673A_ABST
    Figure CN118887673A_ABST
Patent Text Reader

Abstract

The invention relates to an image correction method and device, and the method comprises the steps: obtaining a to-be-corrected image, and determining a preliminary offset field of the to-be-corrected image; performing first correction on the to-be-corrected image by using the initial offset field to obtain an initial corrected image; determining a middle offset field of the preliminarily corrected image; determining a final offset field of the to-be-corrected image according to the middle offset field and the initial offset field; and performing second correction on the to-be-corrected image by using the final offset field to obtain a corrected image. In this way, the to-be-corrected image is corrected twice, the corrected image is obtained, a scanner can be avoided, the to-be-corrected image can still be corrected, and negative effects on the readability of the image and the recognition effect of OCR are avoided.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to an image correction method and device. Background Art

[0002] At present, when photographing objects such as documents and certificates, they are often disturbed by perspective deformation and geometric deformation, causing the objects such as documents and certificates in the photographed images to appear bent and distorted, which will have a negative impact on the readability of the image and the recognition effect of OCR (Optical Character Recognition).

[0003] To this end, a scanner is used. The scanner uses laser or structured light technology to obtain the three-dimensional structural information of objects such as documents and certificates in the image, and then calculates the distortion parameters through geometric algorithms for correction of objects such as documents and certificates in the image. However, scanners are expensive and cumbersome to use. The bulky machines also limit their usage scenarios. Summary of the invention

[0004] In order to solve the technical problems that the above-mentioned scanners are expensive, cumbersome to use, and bulky machines also limit the use scenarios, the embodiments of the present application provide an image correction method, device, electronic device and storage medium. The specific technical solutions are as follows:

[0005] In a first aspect of an embodiment of the present application, an image correction method is first provided, the method comprising:

[0006] Acquire an image to be corrected, and determine a preliminary offset field of the image to be corrected;

[0007] Performing a first correction on the image to be corrected by using the preliminary offset field to obtain a preliminary corrected image;

[0008] determining an intermediate offset field of the preliminary corrected image;

[0009] Determining a final offset field of the image to be corrected according to the intermediate offset field and the preliminary offset field;

[0010] The final offset field is used to perform a second correction on the image to be corrected to obtain a corrected image.

[0011] In an optional implementation, determining the final offset field of the image to be corrected according to the intermediate offset field and the preliminary offset field includes:

[0012] The intermediate offset field is added to the preliminary offset field to obtain a final offset field of the image to be corrected.

[0013] In an optional embodiment, the preliminary offset field represents a preliminary offset direction and a preliminary offset amount of a corresponding position in the image to be corrected;

[0014] performing a first correction on the image to be corrected by using the preliminary offset field to obtain a preliminary corrected image;

[0015] A remapping algorithm is adopted to perform a first correction on a corresponding position of the image to be corrected with reference to the preliminary offset direction and the preliminary offset amount, so as to obtain a preliminary corrected image.

[0016] In an optional embodiment, the final offset field represents the final offset direction and final offset amount of the corresponding position in the image to be corrected;

[0017] The method of performing a second correction on the image to be corrected by using the final offset field to obtain a corrected image includes:

[0018] A remapping algorithm is adopted to perform a second correction on a corresponding position of the image to be corrected with reference to the final offset direction and the final offset amount to obtain a corrected image.

[0019] In an optional implementation, determining the preliminary offset field of the image to be corrected includes:

[0020] Inputting the image to be corrected into a pre-trained first semantic segmentation network to obtain a preliminary offset field of the image to be corrected;

[0021] The determining of the intermediate offset field of the preliminary corrected image comprises:

[0022] The preliminary corrected image is input into a pre-trained second semantic segmentation network to obtain an intermediate offset field of the preliminary corrected image.

[0023] In an optional embodiment, before executing the method, the method further includes:

[0024] Acquire a sample image to be corrected, input the sample image to be corrected into a first semantic segmentation network, and obtain a predicted preliminary offset field of the sample image to be corrected;

[0025] Performing a first correction on the sample image to be corrected by using the predicted preliminary offset field to obtain a predicted preliminary corrected image;

[0026] Inputting the predicted preliminary corrected image into a second semantic segmentation network to obtain a predicted intermediate offset field of the predicted preliminary corrected image;

[0027] Determining a predicted final offset field of the sample image to be corrected according to the predicted preliminary offset field and the predicted intermediate offset field;

[0028] Using the predicted final offset field, performing a second correction on the sample image to be corrected to obtain a predicted corrected image;

[0029] Determining an image loss between a sample corrected image corresponding to the sample image to be corrected and the predicted corrected image;

[0030] The first semantic segmentation network and the second semantic segmentation network are trained according to the image loss, and the training is stopped when the image loss converges to obtain a pre-trained first semantic segmentation network and a pre-trained second semantic segmentation network.

[0031] In an optional implementation, determining the predicted final offset field of the sample image to be corrected according to the predicted preliminary offset field and the predicted intermediate offset field includes:

[0032] The predicted preliminary offset field is added to the predicted intermediate offset field to obtain a predicted final offset field of the sample image to be corrected.

[0033] In an optional embodiment, the first semantic segmentation network and the second semantic segmentation network are trained according to the image loss, and the training is stopped when the image loss converges to obtain the pre-trained first semantic segmentation network and the pre-trained second semantic segmentation network, including:

[0034] Determine an offset field loss between a sample offset field corresponding to the sample image to be corrected and the predicted preliminary offset field;

[0035] Determining a sample intermediate offset field corresponding to the predicted preliminary corrected image, and determining an intermediate offset field loss between the sample intermediate offset field and the predicted intermediate offset field;

[0036] The first semantic segmentation network and the second semantic segmentation network are trained according to the image loss, the offset field loss and the intermediate offset field loss, and the training is stopped when the image loss, the offset field loss and the intermediate offset field loss converge to obtain a pre-trained first semantic segmentation network and a pre-trained second semantic segmentation network.

[0037] In an optional implementation, determining the sample intermediate offset field corresponding to the predicted preliminary corrected image includes:

[0038] Subtract the predicted preliminary offset field from the sample offset field to obtain a difference offset field;

[0039] The difference offset field is determined as a sample intermediate offset field corresponding to the predicted preliminary corrected image.

[0040] In an optional embodiment, the first semantic segmentation network and the second semantic segmentation network are trained according to the image loss, the offset field loss, and the intermediate offset field loss, and the training is stopped when the image loss, the offset field loss, and the intermediate offset field loss converge to obtain a pre-trained first semantic segmentation network and a pre-trained second semantic segmentation network, including:

[0041] determining a first contour mask of a sample element in a sample correction object of the sample correction image;

[0042] determining a second contour mask of a prediction element in a prediction-corrected object of the prediction-corrected image;

[0043] determining a contour mask loss between the first contour mask and the second contour mask;

[0044] The first semantic segmentation network and the second semantic segmentation network are trained according to the image loss, the offset field loss, the intermediate offset field loss and the contour mask loss, and the training is stopped when the image loss, the offset field loss, the intermediate offset field loss and the contour mask loss converge to obtain a pre-trained first semantic segmentation network and a pre-trained second semantic segmentation network.

[0045] In an optional implementation, the determining a first contour mask of a sample element in a sample correction object of the sample correction image comprises:

[0046] Performing contour detection on the sample corrected image to obtain a first contour image of a sample element in a sample corrected object of the sample corrected image;

[0047] Binarizing the first contour image to obtain a first binary image;

[0048] A dilation operation is performed on the first binary image to obtain a first contour mask of sample elements in the sample correction object of the sample correction image.

[0049] In an optional implementation, the determining a second contour mask of a prediction element in the prediction correction object of the prediction correction image comprises:

[0050] Performing contour detection on the predicted and corrected image to obtain a second contour image of a predicted element in a predicted and corrected object of the predicted and corrected image;

[0051] performing binarization processing on the second contour image to obtain a second binarized image;

[0052] A dilation operation is performed on the second binarized image to obtain a second contour mask of the prediction element in the prediction correction object of the prediction correction image.

[0053] In an optional embodiment, the first semantic segmentation network and the second semantic segmentation network are trained according to the image loss, the offset field loss, the intermediate offset field loss, and the contour mask loss, and the training is stopped when the image loss, the offset field loss, the intermediate offset field loss, and the contour mask loss converge to obtain a pre-trained first semantic segmentation network and a pre-trained second semantic segmentation network, including:

[0054] determining a weighted sum of the image loss, the offset field loss, the intermediate offset field loss, and the contour mask loss;

[0055] The first semantic segmentation network and the second semantic segmentation network are trained according to the weighted sum of losses, and the training is stopped when the weighted sum of losses converges to obtain a pre-trained first semantic segmentation network and a pre-trained second semantic segmentation network.

[0056] In a second aspect of the embodiments of the present application, an image correction device is also provided, the device comprising:

[0057] An image acquisition module, used for acquiring an image to be corrected;

[0058] A preliminary offset field determination module, used to determine a preliminary offset field of the image to be corrected;

[0059] An image preliminary correction module, used for performing a first correction on the image to be corrected using the preliminary offset field to obtain a preliminary corrected image;

[0060] An intermediate offset field determination module, used to determine the intermediate offset field of the preliminary corrected image;

[0061] A final offset field determination module, used to determine a final offset field of the image to be corrected according to the intermediate offset field and the preliminary offset field;

[0062] The final image correction module is used to perform a second correction on the image to be corrected by using the final offset field to obtain a corrected image.

[0063] In an optional implementation, the final offset field determination module is specifically configured to:

[0064] The intermediate offset field is added to the preliminary offset field to obtain a final offset field of the image to be corrected.

[0065] In an optional implementation, the preliminary offset field represents the preliminary offset direction and preliminary offset amount of the corresponding position in the image to be corrected; and the preliminary image correction module is specifically used for:

[0066] A remapping algorithm is adopted to perform a first correction on a corresponding position of the image to be corrected with reference to the preliminary offset direction and the preliminary offset amount, so as to obtain a preliminary corrected image.

[0067] In an optional implementation, the final offset field represents the final offset direction and final offset amount of the corresponding position in the image to be corrected; and the image final correction module is specifically used for:

[0068] A remapping algorithm is adopted to perform a second correction on a corresponding position of the image to be corrected with reference to the final offset direction and the final offset amount to obtain a corrected image.

[0069] In an optional implementation, the preliminary offset field determination module is specifically configured to:

[0070] Inputting the image to be corrected into a pre-trained first semantic segmentation network to obtain a preliminary offset field of the image to be corrected;

[0071] The intermediate offset field determination module is specifically used for:

[0072] The preliminary corrected image is input into a pre-trained second semantic segmentation network to obtain an intermediate offset field of the preliminary corrected image.

[0073] In an optional embodiment, the device further comprises:

[0074] A sample image to be corrected acquisition module is used to acquire a sample image to be corrected;

[0075] A predicted preliminary offset field determination module, used for inputting the sample image to be corrected into a first semantic segmentation network to obtain a predicted preliminary offset field of the sample image to be corrected;

[0076] A sample image to be corrected preliminary correction module, used for performing a first correction on the sample image to be corrected using the predicted preliminary offset field to obtain a predicted preliminary corrected image;

[0077] A predicted intermediate offset field determination module, used for inputting the predicted preliminary corrected image into a second semantic segmentation network to obtain a predicted intermediate offset field of the predicted preliminary corrected image;

[0078] A predicted final offset field determination module, used to determine the predicted final offset field of the sample image to be corrected according to the predicted preliminary offset field and the predicted intermediate offset field;

[0079] A sample to-be-corrected image final correction module, used to perform a second correction on the sample to-be-corrected image using the predicted final offset field to obtain a predicted corrected image;

[0080] An image loss determination module, used to determine the image loss between the sample corrected image corresponding to the sample image to be corrected and the predicted corrected image;

[0081] The network training module is used to train the first semantic segmentation network and the second semantic segmentation network according to the image loss, and stop the training when the image loss converges to obtain the pre-trained first semantic segmentation network and the pre-trained second semantic segmentation network.

[0082] In an optional implementation, the predicted final offset field determination module is specifically used to:

[0083] The predicted preliminary offset field is added to the predicted intermediate offset field to obtain a predicted final offset field of the sample image to be corrected.

[0084] In an optional implementation, the network training module specifically includes:

[0085] An offset field loss determination submodule, used to determine an offset field loss between a sample offset field corresponding to the sample image to be corrected and the predicted preliminary offset field;

[0086] A sample intermediate offset field determination submodule, used to determine the sample intermediate offset field corresponding to the predicted preliminary corrected image;

[0087] An intermediate offset field loss determination submodule, used to determine an intermediate offset field loss between the sample intermediate offset field and the predicted intermediate offset field;

[0088] The network training submodule is used to train the first semantic segmentation network and the second semantic segmentation network according to the image loss, the offset field loss and the intermediate offset field loss, and stop the training when the image loss, the offset field loss and the intermediate offset field loss converge to obtain the pre-trained first semantic segmentation network and the pre-trained second semantic segmentation network.

[0089] In an optional implementation manner, the sample intermediate offset field determination submodule is specifically used to:

[0090] Subtract the predicted preliminary offset field from the sample offset field to obtain a difference offset field;

[0091] The difference offset field is determined as a sample intermediate offset field corresponding to the predicted preliminary corrected image.

[0092] In an optional implementation, the network training submodule specifically includes:

[0093] a first contour mask determining unit, configured to determine a first contour mask of a sample element in a sample correction object of the sample correction image;

[0094] a second contour mask determining unit, configured to determine a second contour mask of a prediction element in a prediction correction object of the prediction correction image;

[0095] a contour mask loss determining unit, configured to determine a contour mask loss between the first contour mask and the second contour mask;

[0096] A network training unit is used to train the first semantic segmentation network and the second semantic segmentation network according to the image loss, the offset field loss, the intermediate offset field loss and the contour mask loss, and stop training when the image loss, the offset field loss, the intermediate offset field loss and the contour mask loss converge, so as to obtain a pre-trained first semantic segmentation network and a pre-trained second semantic segmentation network.

[0097] In an optional implementation, the first contour mask determining unit is specifically configured to:

[0098] Performing contour detection on the sample corrected image to obtain a first contour image of a sample element in a sample corrected object of the sample corrected image;

[0099] Binarizing the first contour image to obtain a first binary image;

[0100] A dilation operation is performed on the first binary image to obtain a first contour mask of sample elements in the sample correction object of the sample correction image.

[0101] In an optional implementation, the second contour mask determining unit is specifically configured to:

[0102] Performing contour detection on the predicted and corrected image to obtain a second contour image of a predicted element in a predicted and corrected object of the predicted and corrected image;

[0103] performing binarization processing on the second contour image to obtain a second binarized image;

[0104] A dilation operation is performed on the second binarized image to obtain a second contour mask of the prediction element in the prediction correction object of the prediction correction image.

[0105] In an optional implementation, the network training unit is specifically used for:

[0106] determining a weighted sum of the image loss, the offset field loss, the intermediate offset field loss, and the contour mask loss;

[0107] The first semantic segmentation network and the second semantic segmentation network are trained according to the weighted sum of losses, and the training is stopped when the weighted sum of losses converges to obtain a pre-trained first semantic segmentation network and a pre-trained second semantic segmentation network.

[0108] In a third aspect of the embodiments of the present application, there is further provided an electronic device, comprising a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus;

[0109] Memory, used to store computer programs;

[0110] The processor is used to implement any image correction method described in the first aspect when executing the program stored in the memory.

[0111] In a fourth aspect of the embodiments of the present application, a storage medium is further provided, wherein instructions are stored in the storage medium, and when the storage medium is run on a computer, the computer executes the image correction method described in any one of the first aspects above.

[0112] In a fifth aspect of the embodiments of the present application, a computer program product comprising instructions is also provided, which, when executed on a computer, enables the computer to execute any of the above-mentioned image correction methods.

[0113] The technical solution provided by the embodiment of the present application obtains an image to be corrected, determines a preliminary offset field of the image to be corrected, performs a first correction on the image to be corrected using the preliminary offset field to obtain a preliminary corrected image, determines an intermediate offset field of the preliminary corrected image, determines a final offset field of the image to be corrected based on the intermediate offset field and the preliminary offset field, and performs a second correction on the image to be corrected using the final offset field to obtain a corrected image.

[0114] By determining a preliminary offset field of an image to be corrected, performing a first correction on the image to be corrected using the preliminary offset field to obtain a preliminary corrected image, determining an intermediate offset field of the preliminary corrected image, determining a final offset field of the image to be corrected based on the intermediate offset field and the preliminary offset field, and performing a second correction on the image to be corrected using the final offset field to obtain a corrected image, the image to be corrected is corrected twice to obtain a corrected image, and the use of a scanner can be avoided. The correction of the image to be corrected can still be achieved, thereby avoiding a negative impact on the readability of the image and the recognition effect of OCR. BRIEF DESCRIPTION OF THE DRAWINGS

[0115] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0116] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0117] One or more embodiments are exemplarily described by pictures in the corresponding drawings, and these exemplified descriptions do not constitute limitations on the embodiments. Elements with the same reference numerals in the drawings represent similar elements, and unless otherwise stated, the figures in the drawings do not constitute proportional limitations.

[0118] Figure 1 A schematic diagram of an implementation process of an image correction method shown in an embodiment of the present application;

[0119] Figure 2 It is a schematic diagram of a document to be corrected in a document image to be corrected in an embodiment of the present application, showing a state where the document to be corrected is bent and distorted;

[0120] Figure 3 A schematic diagram of a preliminary offset field shown in an embodiment of the present application;

[0121] Figure 4 A corrected document image in which the corrected document is flattened is shown in an embodiment of the present application;

[0122] Figure 5 It is a schematic diagram of an implementation process of another image correction method shown in an embodiment of the present application;

[0123] Figure 6 A schematic diagram of an implementation flow of a network training method shown in an embodiment of the present application;

[0124] Figure 7 A schematic diagram of a first contour mask of text and objects in a sample document of a sample document image shown in an embodiment of the present application;

[0125] Figure 8 is a schematic diagram of the structure of an image correction device shown in an embodiment of the present application;

[0126] Fig. 9 It is a schematic diagram of the structure of an electronic device shown in an embodiment of the present application. DETAILED DESCRIPTION

[0127] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0128] The disclosure below provides many different embodiments or examples to realize the different structures of the present application. In order to simplify the disclosure of the present application, the parts and settings of specific examples are described below. Of course, they are only examples, and the purpose is not to limit the present application. In addition, the present application can repeat reference numbers and / or letters in different examples. This repetition is for the purpose of simplification and clarity, and does not itself indicate the relationship between the various embodiments and / or settings discussed.

[0129] like Figure 1 FIG. 1 is a schematic diagram of an implementation flow of an image correction method provided in an embodiment of the present application. The method is applied to an electronic device and may specifically include the following steps:

[0130] S101, acquiring an image to be corrected, and determining a preliminary offset field of the image to be corrected.

[0131] In an embodiment of the present application, when photographing objects such as documents and certificates, they are often disturbed by perspective deformation and geometric deformation, causing the documents, certificates and other objects in the photographed images to appear bent and distorted. Therefore, it is necessary to correct the documents, certificates and other objects in the image so that the documents, certificates and other objects in the image appear flat.

[0132] To this end, the above-mentioned images can be used as images to be corrected, thereby obtaining images to be corrected, wherein the object to be corrected in the images to be corrected is in a bent or distorted state (referring to the object to be corrected having various deformations). For example, a document image to be corrected is obtained, wherein the document to be corrected in the image to be corrected is in a bent or distorted state, and the bent or distorted state is as follows: Figure 2 shown.

[0133] It should be noted that the object to be corrected in the image to be corrected may be, for example, a document or a certificate, or may be other objects such as a book, and the embodiment of the present application does not limit this.

[0134] In addition, for the image to be corrected, a preliminary offset field of the image to be corrected can be determined, wherein the preliminary offset field represents the preliminary offset direction and preliminary offset amount of the corresponding position in the image to be corrected. The preliminary offset field of the image to be corrected can be determined based on a deep learning model. For example, Figure 3 shown.

[0135] It should be noted that the preliminary offset field is an image with direction and magnitude, which is similar to the image gradient. The shape of the preliminary offset field is the same as the image to be corrected. Each arrow in the preliminary offset field is a vector containing information such as direction and magnitude. That is, the preliminary offset field indicates the direction in which the corresponding position in the image to be corrected needs to be initially offset, as well as the preliminary offset amount.

[0136] S102, performing a first correction on the image to be corrected using the preliminary offset field to obtain a preliminary corrected image.

[0137] In the embodiment of the present application, for the preliminary offset field of the image to be corrected, the preliminary offset field can be used to perform a first correction on the image to be corrected to obtain a preliminary corrected image, wherein the preliminary corrected object in the preliminary corrected image presents a preliminary flattened state (meaning that the object to be corrected is initially in a normal state without various deformations).

[0138] It should be noted that in the process of performing the first correction on the image to be corrected using the preliminary offset field, it is necessary to use the remap algorithm to perform the first correction, which means that the remap algorithm is used to perform the first correction on the image to be corrected with reference to the preliminary offset field to obtain a preliminary corrected image. The preliminary offset field represents the preliminary offset direction and preliminary offset amount of the corresponding position in the image to be corrected, and thus the remap algorithm is used to perform the first correction on the corresponding position of the image to be corrected with reference to the preliminary offset direction and preliminary offset amount to obtain a preliminary corrected image.

[0139] It should be noted that for the remap algorithm, the original image will be remapped geometrically according to the specified mapping form. For this reason, for the corresponding position in the image to be corrected, the preliminary offset field contains the preliminary offset direction and preliminary offset of the position, that is, the mapping form of the position is specified, so that the position can be remapped geometrically through the mapping form. This operation is performed for each position in the image to be corrected, and finally a preliminary corrected image can be obtained.

[0140] S103, determining an intermediate offset field of the preliminary corrected image.

[0141] In an embodiment of the present application, for a preliminary corrected image, an intermediate offset field of the preliminary corrected image can be determined. The intermediate offset field can be a residual offset field of the preliminary corrected image, and of course, it can also be other types of offset fields, which are not limited in the embodiment of the present application. The residual offset field is the difference between the real offset field of the image to be corrected and the preliminary offset field. Among them, the residual offset field characterizes the residual offset direction (i.e., the difference between the real offset direction and the preliminary offset direction) and the residual offset (i.e., the difference between the real offset and the preliminary offset) of the corresponding position in the preliminary corrected image, and the residual offset field of the preliminary corrected image can be determined based on a deep learning model.

[0142] It should be noted that the residual offset field is an image with direction and magnitude, which is similar to the image gradient. The shape of the residual offset field is the same as that of the preliminary corrected image. Each arrow in the residual offset field is a vector containing information such as direction and magnitude. That is, the residual offset field represents the direction in which the corresponding position in the preliminary corrected image needs to be offset (that is, the difference between the actual offset direction and the preliminary offset direction), as well as the residual offset (that is, the difference between the actual offset and the preliminary offset).

[0143] S104, determining a final offset field of the image to be corrected according to the intermediate offset field and the preliminary offset field.

[0144] In the embodiment of the present application, for the preliminary offset field of the image to be corrected, and the intermediate (for example, residual) offset field of the preliminary corrected image, the final offset field of the image to be corrected can be determined according to the intermediate (for example, residual) offset field and the preliminary offset field. The final offset field can be approximately regarded as the real offset field of the image to be corrected, but there is a certain error between it and the real offset field of the image to be corrected in the true sense.

[0145] It should be noted that, since it is difficult to directly predict the true offset field of the image to be corrected, most methods currently use iterative methods to gradually approximate the true offset field of the image to be corrected. The embodiment of the present application uses a method of predicting residuals. The first round of iteration predicts the preliminary offset field of the image to be corrected, that is, the approximate offset field of the image to be corrected. The second round of iteration predicts the difference between the true offset field of the image to be corrected and the preliminary offset field of the image to be corrected predicted in the first round, that is, the residual offset field. Then, based on the residual offset field and the preliminary offset field, the final offset field of the image to be corrected is determined. The final offset field can be approximately regarded as the true offset field of the image to be corrected. This method of predicting residuals can better predict the accurate true offset field of the image to be corrected.

[0146] S105, performing a second correction on the image to be corrected using the final offset field to obtain a corrected image.

[0147] In the embodiment of the present application, for the final offset field of the image to be corrected, the final offset field can be used to perform a second correction on the image to be corrected to obtain a final corrected image. In the final corrected image, the corrected object is in a flattened state (referring to the normal state of the object to be corrected without various deformations). The flattened state is as follows: Figure 4 shown.

[0148] It should be noted that in the process of performing the second correction on the image to be corrected using the final offset field, it is necessary to use the remap algorithm to perform the second correction, which means that the remap algorithm is used to perform the second correction on the image to be corrected with reference to the final offset field to obtain the final corrected image. The final offset field represents the final offset direction and the final offset amount of the corresponding position in the image to be corrected, and the remap algorithm is used to perform the second correction on the corresponding position of the image to be corrected with reference to the final offset direction and the final offset amount to obtain the corrected image.

[0149] It should be noted that for the remap algorithm, the original image will be remapped geometrically according to the specified mapping form. For this reason, for the corresponding position in the image to be corrected, the final offset field contains the final offset direction and final offset of the position, that is, the mapping form of the position is specified, so that the position can be remapped geometrically through the mapping form. This operation is performed for each position in the image to be corrected, and finally a corrected image can be obtained.

[0150] In addition, for the final corrected image, the final corrected image can be mirrored and expanded and morphologically processed to cleverly achieve edge loss completion. In addition, the final offset field can also be smoothed, so that the remap algorithm is used to perform a second correction on the image to be corrected with reference to the final offset field after smoothing to obtain the final corrected image, and the image details in the final corrected image are smoother and more natural.

[0151] Through the above description of the technical solution provided in the embodiment of the present application, an image to be corrected is obtained, and a preliminary offset field of the image to be corrected is determined. The preliminary offset field is used to perform a first correction on the image to be corrected to obtain a preliminary corrected image. An intermediate (for example, residual) offset field of the preliminary corrected image is determined. The intermediate (for example, residual) offset field is the difference between a real offset field of the image to be corrected and the preliminary offset field. According to the intermediate (for example, residual) offset field and the preliminary offset field, a final offset field of the image to be corrected is determined. The final offset field is used to perform a second correction on the image to be corrected to obtain a corrected image.

[0152] By determining a preliminary offset field of an image to be corrected, performing a first correction on the image to be corrected using the preliminary offset field to obtain a preliminary corrected image, determining an intermediate (e.g., residual) offset field of the preliminary corrected image, determining a final offset field of the image to be corrected based on the intermediate (e.g., residual) offset field and the preliminary offset field, performing a second correction on the image to be corrected using the final offset field to obtain a corrected image, and thus obtaining a corrected image by correcting the image to be corrected twice, it is possible to avoid using a scanner, and still achieve correction of the image to be corrected, thereby avoiding negative impacts on the readability of the image and the recognition effect of OCR.

[0153] In addition, in an embodiment of the present application, the correction of objects in an image can be achieved based on deep learning. For this purpose, a distortion network is constructed. The distortion network specifically includes a first semantic segmentation network and a second semantic segmentation network. The first semantic segmentation network can process the image to be corrected to obtain a preliminary offset field of the image to be corrected. The second semantic segmentation network can process the preliminary corrected image to obtain an intermediate (e.g., residual) offset field of the preliminary corrected image. According to the intermediate (e.g., residual) offset field and the preliminary offset field, the final offset field of the image to be corrected is determined. The final offset field is used to perform a second correction on the image to be corrected to obtain a corrected image. Based on this, Figure 5 FIG. 1 is a schematic diagram of an implementation flow of another image correction method provided in an embodiment of the present application. The method is applied to an electronic device and may specifically include the following steps:

[0154] S501, obtaining an image to be corrected, and inputting the image to be corrected into a pre-trained first semantic segmentation network to obtain a preliminary offset field of the image to be corrected.

[0155] In an embodiment of the present application, an image to be corrected may be obtained, for example, a document image to be corrected. For the image to be corrected, the image to be corrected may be input into a pre-trained first semantic segmentation network to obtain a preliminary offset field of the image to be corrected output by the pre-trained first semantic segmentation network.

[0156] It should be noted that, for the pre-trained first semantic segmentation network, for example, it can be a semantic segmentation network UNet, which is mainly composed of three parts: a backbone feature extraction network, an enhanced feature extraction network and a prediction network. The backbone feature extraction network performs backbone feature extraction on the image to be corrected, and then the enhanced feature extraction network performs enhanced feature extraction on the extracted backbone features. Finally, the prediction network predicts the enhanced features to obtain a preliminary offset field of the image to be corrected. The embodiment of the present application is not limited to this.

[0157] S502: Perform a first correction on the image to be corrected using the preliminary offset field to obtain a preliminary corrected image.

[0158] In the embodiment of the present application, this step is similar to the above-mentioned step S102, and the embodiment of the present application will not be described one by one here.

[0159] S503: Input the preliminary corrected image into a pre-trained second semantic segmentation network to obtain an intermediate offset field of the preliminary corrected image.

[0160] In an embodiment of the present application, for the preliminary corrected image, the preliminary corrected image can be input into a pre-trained second semantic segmentation network to obtain an intermediate (e.g., residual) offset field of the preliminary corrected image output by the pre-trained second semantic segmentation network, where the intermediate (e.g., residual) offset field is the difference between the true offset field of the image to be corrected and the preliminary offset field.

[0161] It should be noted that, for the pre-trained second semantic segmentation network, the network structure is the same as the pre-trained first semantic segmentation network, and the difference lies in the different network parameters. The pre-trained second semantic segmentation network, for example, can be a semantic segmentation network UNet, which is mainly composed of three parts: a backbone feature extraction network, an enhanced feature extraction network, and a prediction network. The backbone feature extraction network extracts backbone features from the preliminary corrected image, and then the enhanced feature extraction network performs enhanced feature extraction on the extracted backbone features. Finally, the prediction network predicts the enhanced features to obtain an intermediate (e.g., residual) offset field of the preliminary corrected image, which is not limited in the embodiments of the present application.

[0162] S504: Add the intermediate offset field to the preliminary offset field to obtain a final offset field of the image to be corrected.

[0163] In an embodiment of the present application, for a preliminary offset field of an image to be corrected, and an intermediate (e.g., residual) offset field of the preliminary corrected image, the intermediate (e.g., residual) offset field can be added to the preliminary offset field to obtain a final offset field of the image to be corrected. The final offset field can be approximately regarded as the true offset field of the image to be corrected, but there is a certain error between the final offset field and the true offset field of the image to be corrected in the true sense.

[0164] S505: Perform a second correction on the image to be corrected using the final offset field to obtain a corrected image.

[0165] In the embodiment of the present application, this step is similar to the above-mentioned step S105, and the embodiment of the present application will not be described one by one here.

[0166] Through the pre-trained first semantic segmentation network, the preliminary offset field of the image to be corrected is determined, and the preliminary offset field is used to perform a first correction on the image to be corrected to obtain a preliminary corrected image. Through the pre-trained second semantic segmentation network, the intermediate (for example, residual) offset field of the preliminary corrected image is determined, and the intermediate (for example, residual) offset field is added to the preliminary offset field to obtain a final offset field of the image to be corrected. The final offset field is used to perform a second correction on the image to be corrected to obtain a corrected image. In this way, the image to be corrected is corrected twice to obtain a corrected image, which can avoid the use of a scanner and still achieve correction of the image to be corrected, thereby avoiding negative impacts on the readability of the image and the recognition effect of OCR.

[0167] In addition, the pre-trained first semantic segmentation network and the pre-trained second semantic segmentation network need to be trained before the pre-trained first semantic segmentation network can process the image to be corrected to obtain a preliminary offset field of the image to be corrected, and the pre-trained second semantic segmentation network can process the preliminary corrected image to obtain an intermediate (e.g., residual) offset field of the preliminary corrected image. Based on this, Figure 6 FIG. 1 is a schematic diagram of an implementation flow of a network training method provided in an embodiment of the present application. The method is applied to an electronic device and may specifically include the following steps:

[0168] S601, obtaining a sample image to be corrected, inputting the sample image to be corrected into a first semantic segmentation network, and obtaining a predicted preliminary offset field of the sample image to be corrected.

[0169] In an embodiment of the present application, a sample image to be corrected may be obtained, for example, a sample document image to be corrected. For the sample image to be corrected, the sample image to be corrected may be input into a first semantic segmentation network to obtain a predicted preliminary offset field of the sample image to be corrected output by the first semantic segmentation network.

[0170] It should be noted that, for the first semantic segmentation network, for example, it can be a semantic segmentation network UNet, which is mainly composed of three parts: a backbone feature extraction network, an enhanced feature extraction network and a prediction network. The backbone feature extraction network extracts backbone features of the sample image to be corrected, and then the enhanced feature extraction network performs enhanced feature extraction on the extracted backbone features. Finally, the prediction network predicts the enhanced features to obtain a predicted preliminary offset field of the sample image to be corrected. The embodiment of the present application is not limited to this.

[0171] S602, performing a first correction on the sample image to be corrected using the predicted preliminary offset field to obtain a predicted preliminary corrected image.

[0172] In the embodiment of the present application, for the predicted preliminary offset field of the sample image to be corrected, the predicted preliminary offset field can be used to perform a first correction on the sample image to be corrected to obtain a predicted preliminary corrected image. In the predicted preliminary corrected image, the predicted preliminary corrected object presents a preliminary flattened state (meaning that the sample object to be corrected is initially in a normal state without various deformations).

[0173] It should be noted that in the process of performing the first correction on the sample image to be corrected using the predicted preliminary offset field, it is necessary to use the remap algorithm to perform the first correction, which means that the remap algorithm is used to perform the first correction on the sample image to be corrected with reference to the predicted preliminary offset field to obtain the predicted preliminary corrected image. Among them, the predicted preliminary offset field represents the predicted preliminary offset direction and the predicted preliminary offset amount of the corresponding position in the sample image to be corrected, and the remap algorithm is used to perform the first correction on the corresponding position of the sample image to be corrected with reference to the predicted preliminary offset direction and the predicted preliminary offset amount to obtain the predicted preliminary corrected image.

[0174] S603: Input the predicted preliminary corrected image into a second semantic segmentation network to obtain a predicted intermediate offset field of the predicted preliminary corrected image.

[0175] In an embodiment of the present application, for the predicted preliminary corrected image, the predicted preliminary corrected image can be input into the second semantic segmentation network to obtain a predicted intermediate (e.g., residual) offset field of the predicted preliminary corrected image output by the second semantic segmentation network, and the predicted intermediate (e.g., residual) offset field is the difference between the sample true offset field of the sample image to be corrected and the predicted preliminary offset field.

[0176] It should be noted that, for the second semantic segmentation network, the network structure is the same as that of the first semantic segmentation network, and the difference lies in the different network parameters. The second semantic segmentation network, for example, can be a semantic segmentation network UNet, which is mainly composed of three parts: a backbone feature extraction network, an enhanced feature extraction network, and a prediction network. The backbone feature extraction network extracts backbone features of the predicted preliminary corrected image, and then the enhanced feature extraction network extracts enhanced features of the extracted backbone features. Finally, the prediction network predicts the enhanced features to obtain a predicted intermediate (e.g., residual) offset field of the predicted preliminary corrected image, which is not limited in the embodiments of the present application.

[0177] S604, determining a predicted final offset field of the sample image to be corrected according to the predicted preliminary offset field and the predicted intermediate offset field.

[0178] In an embodiment of the present application, for the predicted preliminary offset field of the sample image to be corrected, and the predicted intermediate (for example, residual) offset field of the predicted preliminary corrected image, the predicted final offset field of the sample image to be corrected can be determined based on the predicted preliminary offset field and the predicted intermediate (for example, residual) offset field. The predicted final offset field can be approximately regarded as the sample true offset field of the sample image to be corrected, but there is a certain error between it and the sample true offset field of the sample image to be corrected in the true sense.

[0179] Specifically, for the predicted preliminary offset field of the sample image to be corrected, and the predicted intermediate (e.g., residual) offset field of the predicted preliminary corrected image, the predicted preliminary offset field and the predicted intermediate (e.g., residual) offset field can be added to obtain the predicted final offset field of the sample image to be corrected.

[0180] S605: Perform a second correction on the sample image to be corrected by using the predicted final offset field to obtain a predicted corrected image.

[0181] In the embodiment of the present application, for the predicted final offset field of the sample image to be corrected, the predicted final offset field can be used to perform a second correction on the sample image to be corrected to obtain a predicted corrected image. In the final predicted corrected image, the predicted corrected object is in a flattened state (meaning that the object to be corrected is in a normal state without various deformations), and the flattened state is as follows: Figure 4 shown.

[0182] It should be noted that in the process of performing the second correction on the sample image to be corrected using the predicted final offset field, it is necessary to use the remap algorithm to perform the second correction, which means that the remap algorithm is used to refer to the predicted final offset field to perform the second correction on the sample image to be corrected to obtain the predicted corrected image. Among them, the predicted final offset field represents the predicted final offset direction and the predicted final offset amount of the corresponding position in the sample image to be corrected, and the remap algorithm is used to refer to the predicted final offset direction and the predicted final offset amount to perform the second correction on the corresponding position of the sample image to be corrected to obtain the predicted corrected image.

[0183] In addition, for the predicted corrected image, the predicted corrected image can be mirrored and expanded and morphologically processed to cleverly achieve edge loss completion. In addition, the predicted corrected image can also be smoothed. In this way, the remap algorithm is used to refer to the predicted final offset field after smoothing to perform a second correction on the sample to be corrected image to obtain the predicted corrected image. The image details in the predicted corrected image are smoother and more natural.

[0184] S606: Determine the image loss between the sample corrected image and the predicted corrected image corresponding to the sample image to be corrected.

[0185] S607, training the first semantic segmentation network and the second semantic segmentation network according to the image loss, and stopping the training when the image loss converges, to obtain a pre-trained first semantic segmentation network and a pre-trained second semantic segmentation network.

[0186] In the embodiment of the present application, for a sample image to be corrected, there is a corresponding sample corrected image, for example, Figure 4 As shown, at this time, the image loss between the sample corrected image and the predicted corrected image corresponding to the sample image to be corrected can be determined, and the first semantic segmentation network and the second semantic segmentation network are trained according to the image loss. When the image loss converges, the training is stopped to obtain the pre-trained first semantic segmentation network and the pre-trained second semantic segmentation network.

[0187] In addition, in the embodiment of the present application, the first semantic segmentation network and the second semantic segmentation network can be trained with the help of the multimodal idea. The so-called multimodal refers to the collaborative reasoning of multiple heterogeneous modal data, which can integrate or fuse two or more biometric technologies, utilize the unique advantages of multiple biometric technologies, and combine data fusion technology to make the authentication and identification process more accurate.

[0188] Based on this, the offset field loss between the sample offset field corresponding to the sample image to be corrected and the predicted preliminary offset field is determined (the offset field loss is used to supervise the first semantic segmentation network), the sample intermediate (e.g., residual) offset field corresponding to the predicted preliminary corrected image is determined, and the intermediate (e.g., residual) offset field loss between the sample intermediate (e.g., residual) offset field and the predicted intermediate (e.g., residual) offset field is determined (the intermediate (e.g., residual) offset field loss is used to supervise the second semantic segmentation network), and the first semantic segmentation network and the second semantic segmentation network are trained according to the image loss, the offset field loss, and the intermediate (e.g., residual) offset field loss. When the image loss, the offset field loss, and the intermediate (e.g., residual) offset field loss converge, the training is stopped to obtain the pre-trained first semantic segmentation network and the pre-trained second semantic segmentation network. In this way, not only the offset field loss and the intermediate (e.g., residual) offset field loss are supervised, but also the image loss is supervised, and multimodal training can be achieved for the first semantic segmentation network and the second semantic segmentation network.

[0189] Among them, for the sample intermediate (for example, residual) offset field corresponding to the predicted preliminary corrected image, the sample offset field corresponding to the sample image to be corrected can be subtracted from the predicted preliminary offset field corresponding to the sample image to be corrected to obtain a difference offset field, and the difference offset field is determined as the sample intermediate (for example, residual) offset field corresponding to the predicted preliminary corrected image.

[0190] In addition, it is subjectively believed that the corrected image should essentially pay more attention to the contours of elements (such as text and objects) in the corrected object (such as a document). For this reason, in the process of training the first semantic segmentation network and the second semantic segmentation network, the first semantic segmentation network and the second semantic segmentation network can be made to pay more attention to the contours of elements (such as text and objects), thereby achieving better training effects.

[0191] Based on this, a first contour mask of sample elements in a sample correction object of a sample correction image is determined, a second contour mask of predicted elements in a predicted correction object of a predicted correction image is determined, and a contour mask loss between the first contour mask and the second contour mask is determined. According to the image loss, the offset field loss, the intermediate (e.g., residual) offset field loss, and the contour mask loss, a first semantic segmentation network and a second semantic segmentation network are trained. When the image loss, the offset field loss, the intermediate (e.g., residual) offset field loss, and the contour mask loss converge, the training is stopped to obtain a pre-trained first semantic segmentation network and a pre-trained second semantic segmentation network.

[0192] Among them, contour detection is performed on the sample corrected image to obtain a first contour image of the sample element in the sample corrected object of the sample corrected image, the first contour image is binarized to obtain a first binarized image, and the first binarized image is expanded to obtain a first contour mask of the sample element in the sample corrected object of the sample corrected image, such as Figure 7 shown.

[0193] Perform contour detection on the predicted corrected image to obtain a second contour image of the predicted element in the predicted corrected object of the predicted corrected image, perform binarization on the second contour image to obtain a second binarized image, perform dilation on the second binarized image to obtain a second contour mask of the predicted element in the predicted corrected object of the predicted corrected image.

[0194] In addition, for the image loss, offset field loss, intermediate (such as residual) offset field loss and contour mask loss, a weighted sum can be calculated, and the coefficient of the contour mask loss can be appropriately increased, so as to avoid the problem of local bending of elements (such as text and objects) in the correction object (such as a document). Based on this, the weighted sum of the loss of image loss, offset field loss, intermediate (such as residual) offset field loss and contour mask loss is determined, and the first semantic segmentation network and the second semantic segmentation network are trained according to the weighted sum of the loss, and the training is stopped when the weighted sum of the loss converges (for example, less than a certain threshold), thereby obtaining the pre-trained first semantic segmentation network and the pre-trained second semantic segmentation network.

[0195] In this way, not only the offset field loss and the intermediate (such as residual) offset field loss are supervised, but also the image loss is supervised, and multimodal training can be achieved for the first semantic segmentation network and the second semantic segmentation network, which can improve the prediction effect of the first semantic segmentation network and the second semantic segmentation network. In addition, in the process of multimodal training of the first semantic segmentation network and the second semantic segmentation network, the contour mask loss is introduced, which can make the first semantic segmentation network and the second semantic segmentation network pay more attention to the contours of elements (such as text and objects), thereby achieving better training effects.

[0196] Thus, after the above training, a pre-trained first semantic segmentation network and a pre-trained second semantic segmentation network with better prediction effect and more attention to the contours of elements (such as text and objects) can be obtained. The pre-trained first semantic segmentation network is used to determine the preliminary offset field of the image to be corrected, which improves the prediction effect of the preliminary offset field, and the contours of elements (such as text and objects) in the preliminary offset field are paid special attention to. The preliminary offset field is used to perform a first correction on the image to be corrected to obtain a preliminary corrected image. The pre-trained second semantic segmentation network is used to determine the intermediate (such as residual) of the preliminary corrected image. The intermediate (e.g. residual) offset field is used to improve the prediction effect of the intermediate (e.g. residual) offset field, and the contours of elements (e.g. text and objects) in the intermediate (e.g. residual) offset field are paid special attention to. The intermediate (e.g. residual) offset field is added to the preliminary offset field to obtain a final offset field of the image to be corrected. The final offset field is used to perform a second correction on the image to be corrected to obtain a corrected image. In this way, the image to be corrected is corrected twice to obtain a corrected image, which can avoid using a scanner and still achieve correction of the image to be corrected, thereby avoiding negative impact on the readability of the image and the recognition effect of OCR.

[0197] Corresponding to the above method embodiment, the present application embodiment also provides an image correction device, such as Figure 8 As shown, the device may include: an image acquisition module 810, a preliminary offset field determination module 820, an image preliminary correction module 830, an intermediate offset field determination module 840, a final offset field determination module 850, and an image final correction module 860.

[0198] An image acquisition module 810 is used to acquire an image to be corrected;

[0199] A preliminary offset field determination module 820, configured to determine a preliminary offset field of the image to be corrected;

[0200] An image preliminary correction module 830, configured to perform a first correction on the image to be corrected using the preliminary offset field to obtain a preliminary corrected image;

[0201] An intermediate offset field determination module 840, configured to determine an intermediate offset field of the preliminary corrected image;

[0202] A final offset field determination module 850, configured to determine a final offset field of the image to be corrected according to the intermediate offset field and the preliminary offset field;

[0203] The image final correction module 860 is used to perform a second correction on the image to be corrected using the final offset field to obtain a corrected image.

[0204] In an optional implementation, the final offset field determination module is specifically configured to:

[0205] The intermediate offset field is added to the preliminary offset field to obtain a final offset field of the image to be corrected.

[0206] In an optional implementation, the preliminary offset field represents the preliminary offset direction and preliminary offset amount of the corresponding position in the image to be corrected; and the preliminary image correction module is specifically used for:

[0207] A remapping algorithm is adopted to perform a first correction on a corresponding position of the image to be corrected with reference to the preliminary offset direction and the preliminary offset amount, so as to obtain a preliminary corrected image.

[0208] In an optional implementation, the final offset field represents the final offset direction and final offset amount of the corresponding position in the image to be corrected; and the image final correction module is specifically used for:

[0209] A remapping algorithm is adopted to perform a second correction on a corresponding position of the image to be corrected with reference to the final offset direction and the final offset amount to obtain a corrected image.

[0210] In an optional implementation, the preliminary offset field determination module is specifically configured to:

[0211] Inputting the image to be corrected into a pre-trained first semantic segmentation network to obtain a preliminary offset field of the image to be corrected;

[0212] The intermediate offset field determination module is specifically used for:

[0213] The preliminary corrected image is input into a pre-trained second semantic segmentation network to obtain an intermediate offset field of the preliminary corrected image.

[0214] In an optional embodiment, the device further comprises:

[0215] A sample image to be corrected acquisition module is used to acquire a sample image to be corrected;

[0216] A predicted preliminary offset field determination module, used for inputting the sample image to be corrected into a first semantic segmentation network to obtain a predicted preliminary offset field of the sample image to be corrected;

[0217] A sample image to be corrected preliminary correction module, used for performing a first correction on the sample image to be corrected using the predicted preliminary offset field to obtain a predicted preliminary corrected image;

[0218] A predicted intermediate offset field determination module, used for inputting the predicted preliminary corrected image into a second semantic segmentation network to obtain a predicted intermediate offset field of the predicted preliminary corrected image;

[0219] A predicted final offset field determination module, used to determine the predicted final offset field of the sample image to be corrected according to the predicted preliminary offset field and the predicted intermediate offset field;

[0220] A sample to-be-corrected image final correction module, used to perform a second correction on the sample to-be-corrected image using the predicted final offset field to obtain a predicted corrected image;

[0221] An image loss determination module, used to determine the image loss between the sample corrected image corresponding to the sample image to be corrected and the predicted corrected image;

[0222] The network training module is used to train the first semantic segmentation network and the second semantic segmentation network according to the image loss, and stop the training when the image loss converges to obtain the pre-trained first semantic segmentation network and the pre-trained second semantic segmentation network.

[0223] In an optional implementation, the predicted final offset field determination module is specifically used to:

[0224] The predicted preliminary offset field is added to the predicted intermediate offset field to obtain a predicted final offset field of the sample image to be corrected.

[0225] In an optional implementation, the network training module specifically includes:

[0226] An offset field loss determination submodule, used to determine an offset field loss between a sample offset field corresponding to the sample image to be corrected and the predicted preliminary offset field;

[0227] A sample intermediate offset field determination submodule, used to determine the sample intermediate offset field corresponding to the predicted preliminary corrected image;

[0228] An intermediate offset field loss determination submodule, used to determine an intermediate offset field loss between the sample intermediate offset field and the predicted intermediate offset field;

[0229] The network training submodule is used to train the first semantic segmentation network and the second semantic segmentation network according to the image loss, the offset field loss and the intermediate offset field loss, and stop the training when the image loss, the offset field loss and the intermediate offset field loss converge to obtain the pre-trained first semantic segmentation network and the pre-trained second semantic segmentation network.

[0230] In an optional implementation manner, the sample intermediate offset field determination submodule is specifically used to:

[0231] Subtract the predicted preliminary offset field from the sample offset field to obtain a difference offset field;

[0232] The difference offset field is determined as a sample intermediate offset field corresponding to the predicted preliminary corrected image.

[0233] In an optional implementation, the network training submodule specifically includes:

[0234] a first contour mask determining unit, configured to determine a first contour mask of a sample element in a sample correction object of the sample correction image;

[0235] a second contour mask determining unit, configured to determine a second contour mask of a prediction element in a prediction correction object of the prediction correction image;

[0236] a contour mask loss determining unit, configured to determine a contour mask loss between the first contour mask and the second contour mask;

[0237] A network training unit is used to train the first semantic segmentation network and the second semantic segmentation network according to the image loss, the offset field loss, the intermediate offset field loss and the contour mask loss, and stop training when the image loss, the offset field loss, the intermediate offset field loss and the contour mask loss converge, so as to obtain a pre-trained first semantic segmentation network and a pre-trained second semantic segmentation network.

[0238] In an optional implementation, the first contour mask determining unit is specifically configured to:

[0239] Performing contour detection on the sample corrected image to obtain a first contour image of a sample element in a sample corrected object of the sample corrected image;

[0240] Binarizing the first contour image to obtain a first binary image;

[0241] A dilation operation is performed on the first binary image to obtain a first contour mask of sample elements in the sample correction object of the sample correction image.

[0242] In an optional implementation, the second contour mask determining unit is specifically configured to:

[0243] Performing contour detection on the predicted and corrected image to obtain a second contour image of a predicted element in a predicted and corrected object of the predicted and corrected image;

[0244] performing binarization processing on the second contour image to obtain a second binarized image;

[0245] A dilation operation is performed on the second binarized image to obtain a second contour mask of the prediction element in the prediction correction object of the prediction correction image.

[0246] In an optional implementation, the network training unit is specifically used for:

[0247] determining a weighted sum of the image loss, the offset field loss, the intermediate offset field loss, and the contour mask loss;

[0248] The first semantic segmentation network and the second semantic segmentation network are trained according to the weighted sum of losses, and the training is stopped when the weighted sum of losses converges to obtain a pre-trained first semantic segmentation network and a pre-trained second semantic segmentation network.

[0249] The present application also provides an electronic device, such as Fig. 9 As shown, it includes a processor 91, a communication interface 92, a memory 93 and a communication bus 94, wherein the processor 91, the communication interface 92, and the memory 93 communicate with each other through the communication bus 94.

[0250] Memory 93, used for storing computer programs;

[0251] The processor 91 is used to execute the program stored in the memory 93 to implement the following steps:

[0252] Acquire an image to be corrected, and determine a preliminary offset field of the image to be corrected; perform a first correction on the image to be corrected using the preliminary offset field to obtain a preliminary corrected image; determine an intermediate offset field of the preliminary corrected image; determine a final offset field of the image to be corrected based on the intermediate offset field and the preliminary offset field; perform a second correction on the image to be corrected using the final offset field to obtain a corrected image.

[0253] The communication bus mentioned in the above electronic device can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0254] The communication interface is used for communication between the above electronic device and other devices.

[0255] The memory may include a random access memory (RAM) or a non-volatile memory, such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

[0256] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0257] In another embodiment provided in the present application, a storage medium is also provided, in which instructions are stored. When the instructions are executed on a computer, the computer executes the image correction method described in any one of the above embodiments.

[0258] In another embodiment provided in the present application, a computer program product including instructions is also provided, which, when executed on a computer, enables the computer to execute the image correction method described in any one of the above embodiments.

[0259] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a storage medium, or transmitted from one storage medium to another storage medium, for example, the computer instructions may be transmitted from a website site, a computer, a server or a data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center. The storage medium may be any available medium that a computer can access or a data storage device such as a server or a data center that includes one or more available media integrations. The available medium may be a magnetic medium, (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive Solid State Disk (SSD)), etc.

[0260] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0261] Each embodiment in this specification is described in a related manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0262] The above description is only a preferred embodiment of the present application and is not intended to limit the protection scope of the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application are included in the protection scope of the present application.

Claims

1. An image correction method, characterized in that: The method comprises: Acquiring an image to be corrected, and determining a preliminary offset field of the image to be corrected; Performing a first correction on the image to be corrected using the preliminary offset field to obtain a preliminary corrected image; determining an intermediate offset field of the preliminary corrected image; determining a final offset field of the image to be corrected according to the intermediate offset field and the preliminary offset field; The final offset field is used to perform a second correction on the image to be corrected to obtain a corrected image.

2. The method according to claim 1, characterized in that Determining the final offset field of the image to be corrected according to the intermediate offset field and the preliminary offset field includes: The intermediate offset field is added to the preliminary offset field to obtain a final offset field of the image to be corrected.

3. The method according to claim 1, characterized in that The preliminary offset field represents a preliminary offset direction and a preliminary offset amount of a corresponding position in the image to be corrected; performing a first correction on the image to be corrected by using the preliminary offset field to obtain a preliminary corrected image; A remapping algorithm is used to perform a first correction on a corresponding position of the image to be corrected with reference to the preliminary offset direction and the preliminary offset amount to obtain a preliminary corrected image.

4. The method according to claim 1, wherein The final offset field represents the final offset direction and final offset amount of the corresponding position in the image to be corrected; The method of performing a second correction on the image to be corrected by using the final offset field to obtain a corrected image includes: A remapping algorithm is used to perform a second correction on a corresponding position of the image to be corrected with reference to the final offset direction and the final offset amount to obtain a corrected image.

5. The method according to claim 1, characterized in that The determining of the preliminary offset field of the image to be corrected includes: Inputting the image to be corrected into a pre-trained first semantic segmentation network to obtain a preliminary offset field of the image to be corrected; The determining of the intermediate offset field of the preliminary corrected image comprises: The preliminary rectified image is input into a pre-trained second semantic segmentation network to obtain an intermediate offset field of the preliminary rectified image.

6. The method according to claim 5, characterized in that Before executing the method, the method further includes: Acquire a sample image to be corrected, input the sample image to be corrected into a first semantic segmentation network, and obtain a predicted preliminary offset field of the sample image to be corrected; Performing a first correction on the sample image to be corrected using the predicted preliminary offset field to obtain a predicted preliminary corrected image; Inputting the predicted preliminary rectified image into a second semantic segmentation network to obtain a predicted intermediate offset field of the predicted preliminary rectified image; determining a predicted final offset field of the sample image to be corrected according to the predicted preliminary offset field and the predicted intermediate offset field; Using the predicted final offset field, performing a second correction on the sample image to be corrected to obtain a predicted corrected image; Determining an image loss between a sample rectified image corresponding to the sample image to be rectified and the predicted rectified image; The first semantic segmentation network and the second semantic segmentation network are trained according to the image loss, and the training is stopped when the image loss converges to obtain a pre-trained first semantic segmentation network and a pre-trained second semantic segmentation network.

7. The method according to claim 6, characterized in that Determining the predicted final offset field of the sample image to be corrected according to the predicted preliminary offset field and the predicted intermediate offset field includes: The predicted preliminary offset field and the predicted intermediate offset field are added to obtain a predicted final offset field of the sample image to be corrected.

8. The method according to claim 6, characterized in that The first semantic segmentation network and the second semantic segmentation network are trained according to the image loss, and the training is stopped when the image loss converges to obtain the pre-trained first semantic segmentation network and the pre-trained second semantic segmentation network, including: Determining an offset field loss between a sample offset field corresponding to the sample image to be corrected and the predicted preliminary offset field; determining a sample intermediate offset field corresponding to the predicted preliminary corrected image, and determining an intermediate offset field loss between the sample intermediate offset field and the predicted intermediate offset field; The first semantic segmentation network and the second semantic segmentation network are trained according to the image loss, the offset field loss, and the intermediate offset field loss, and the training is stopped when the image loss, the offset field loss, and the intermediate offset field loss converge to obtain a pre-trained first semantic segmentation network and a pre-trained second semantic segmentation network.

9. The method according to claim 8, characterized in that The determining of the sample intermediate offset field corresponding to the predicted preliminary corrected image includes: Subtracting the predicted preliminary offset field from the sample offset field to obtain a difference offset field; The difference offset field is determined as the sample intermediate offset field corresponding to the predicted preliminary corrected image.

10. The method according to claim 8, characterized in that The first semantic segmentation network and the second semantic segmentation network are trained according to the image loss, the offset field loss, and the intermediate offset field loss, and the training is stopped when the image loss, the offset field loss, and the intermediate offset field loss converge to obtain a pre-trained first semantic segmentation network and a pre-trained second semantic segmentation network, including: determining a first contour mask of a sample element in a sample correction object of the sample correction image; determining a second contour mask of a prediction element in the prediction-rectified object of the prediction-rectified image; determining a contour mask loss between the first contour mask and the second contour mask; The first semantic segmentation network and the second semantic segmentation network are trained according to the image loss, the offset field loss, the intermediate offset field loss, and the contour mask loss, and the training is stopped when the image loss, the offset field loss, the intermediate offset field loss, and the contour mask loss converge to obtain a pre-trained first semantic segmentation network and a pre-trained second semantic segmentation network.

11. The method according to claim 10, characterized in that The determining of a first contour mask of a sample element in the sample correction object of the sample correction image comprises: Performing contour detection on the sample corrected image to obtain a first contour image of a sample element in a sample corrected object of the sample corrected image; performing binarization processing on the first contour image to obtain a first binarized image; A dilation operation is performed on the first binarized image to obtain a first contour mask of a sample element in the sample correction object of the sample correction image.

12. The method according to claim 10, characterized in that The determining of a second contour mask of a prediction element in the prediction correction object of the prediction correction image comprises: Performing contour detection on the predicted and corrected image to obtain a second contour image of a predicted element in the predicted and corrected object of the predicted and corrected image; performing binarization processing on the second contour image to obtain a second binarized image; A dilation operation is performed on the second binarized image to obtain a second contour mask of the predicted element in the predicted and corrected object of the predicted and corrected image.

13. The method according to claim 10, characterized in that The first semantic segmentation network and the second semantic segmentation network are trained according to the image loss, the offset field loss, the intermediate offset field loss, and the contour mask loss, and the training is stopped when the image loss, the offset field loss, the intermediate offset field loss, and the contour mask loss converge to obtain a pre-trained first semantic segmentation network and a pre-trained second semantic segmentation network, including: determining a weighted sum of the image loss, the offset field loss, the intermediate offset field loss, and the contour mask loss; The first semantic segmentation network and the second semantic segmentation network are trained according to the weighted sum of the losses, and the training is stopped when the weighted sum of the losses converges to obtain a pre-trained first semantic segmentation network and a pre-trained second semantic segmentation network.

14. An image correction device, characterized in that: The device comprises: An image acquisition module, used for acquiring the image to be corrected; a preliminary offset field determination module, configured to determine a preliminary offset field of the image to be corrected; an image preliminary correction module, configured to perform a first correction on the image to be corrected using the preliminary offset field to obtain a preliminary corrected image; an intermediate offset field determination module, configured to determine an intermediate offset field of the preliminary corrected image; a final offset field determining module, configured to determine a final offset field of the image to be corrected based on the intermediate offset field and the preliminary offset field; The final image correction module is used to perform a second correction on the image to be corrected using the final offset field to obtain a corrected image.

Citation Information

Patent Citations

  • Image characteristic point matching method, image characteristic point matching device, electronic device and storage medium

    CN108573269A

  • Image correction method and device

    CN113538252A