Picture correction method and apparatus

Abstract

Embodiments of the present application provide a picture correction method and device; the method comprises: obtaining a picture to be corrected; inputting the picture to be corrected into a trained correction model, processing the picture to be corrected by the trained correction model to obtain a corrected picture; wherein the correction model is obtained by training a control point loss function. The method trains the correction model to be trained by a multi-modal loss function composed of a control point loss function, so that the robustness and efficiency of the trained correction model when correcting the picture are better.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to an image correction method and apparatus. Background Technology

[0002] When photographing documents, perspective distortion and geometric distortion often occur, causing the photographed document images to appear distorted and warped. This results in both the images and text within the document images being distorted and warped, negatively impacting the readability of the document images.

[0003] In related correction methods, expensive optical scanning equipment is often used to obtain the three-dimensional structural information of the document through technologies such as lasers or structured light. Then, the distortion parameters are calculated through geometric algorithms, and the document's curvature is flattened accordingly. However, this optical scanning equipment is expensive and bulky, which not only results in excessive costs, but also limits the applicable scenarios.

[0004] In other cases, image algorithms can be used to estimate and model based on parameters such as the curvature layout information of text lines in the document, thereby correcting the curved document. However, this method requires too many hyperparameters to be set, which often cannot be adapted to various scenarios. Too many parameters involved in the calculation result in insufficient robustness of most image algorithms and a long time consumption, which also limits the applicable scenarios.

[0005] Therefore, there is a lack of image correction methods that are applicable to a wide range of scenarios and have good robustness. Summary of the Invention

[0006] This application provides an image correction method that solves the problem that the application scenarios are limited due to the low computational robustness and efficiency of image correction in related technologies. In this method, the correction model to be trained is trained by using a control point loss function, so that the trained correction model can obtain more accurate prediction results. Then, the trained correction model is used to process the image to be corrected, and finally, an accurate corrected image can be obtained.

[0007] In a first aspect, embodiments of this application provide an image correction method, the method comprising: Obtain the image to be corrected; The image to be corrected is input into the trained correction model, and the trained correction model processes the image to obtain the corrected image. The correction model is obtained by training a control point loss function.

[0008] Secondly, embodiments of this application also provide an image correction device, which includes: an acquisition module and a prediction module; The acquisition module is configured to acquire the image to be corrected; The prediction module is configured to input the image to be corrected into a trained correction model, and process the image to be corrected using the trained correction model to obtain a corrected image; wherein the correction model is trained using a control point loss function.

[0009] In some optional implementations, the training module is further specifically configured as follows: The correction model was obtained by training it in the following way: Obtain a training dataset, which includes multiple sets of metadata, each set of metadata including multiple warped training images; Input a predetermined number of metadata sets into the correction model to be trained, and output the prediction offset matrix corresponding to the distorted training image for each set of metadata sets; The pixels in each distorted training image are mapped according to the predicted offset matrix to obtain the corresponding corrected prediction image; the control point loss corresponding to each corrected prediction image is determined according to the control point loss function, and each control point loss represents the degree of distortion of the straight line in the corresponding corrected prediction image. The correction model to be trained is trained based on the control point loss to obtain the completed correction model.

[0010] In some optional implementations, the training module is further specifically configured as follows: Identify at least one first straight line in each corrected prediction image, and identify at least two adjacent vector segments in each first straight line; Determine the cosine similarity between any two adjacent vectors on the first straight line in each corrected prediction image; The average difference between each cosine similarity and the preset value is determined as the corresponding control point loss.

[0011] In some optional implementations, each set of metadata includes the actual corrected image corresponding to each distortion training image, each distortion training image including a first straight line region; the corresponding actual corrected image includes a second straight line region corresponding to the first straight line region; each set of metadata also includes a straight line mask corresponding to the second straight line region in the actual corrected image; Accordingly, the training module is further configured as follows: The first straight line region in the corresponding corrected prediction image is determined based on each straight line mask; Determine each of the first straight lines in the first straight line region.

[0012] In some optional implementations, the training module is further specifically configured as follows: Obtain multiple first control points in each first straight line; Among the three consecutive first control points in the first straight line, the first vector in the first straight line is determined by the middle first control point and the previous first control point. The first control point in the middle and the next first control point are used to determine the next vector in the first straight line that is adjacent to the previous vector.

[0013] In some optional implementations, each set of metadata also includes the actual offset matrix corresponding to the warped training images; Accordingly, the training module is further configured as follows: Obtain multiple second control points in the actual corrected image corresponding to each distortion training image; The actual offset matrix is ​​used to map each second control point in each actual corrected image to the corresponding twisted training image, thereby obtaining multiple twisted image control points in each twisted training image. The predicted offset matrix is ​​used to map each twisted image control point in each twisted training image to the corresponding corrected prediction image, thereby obtaining multiple first control points in each first straight line in the corresponding corrected prediction image.

[0014] In some alternative implementations, the second control point includes a display line control point; Accordingly, the training module is further configured as follows: Obtain the edge of the second straight line region in each actual corrected image; Determine the intersection between the line mask corresponding to the second straight line region in each actual corrected image and the edge of the second straight line region, and define the intersection as each display line in the corresponding actual corrected image; Each displayed straight line is divided into multiple equal parts, and each division position is determined as the display line control point of the corresponding displayed straight line.

[0015] In some optional implementations, each distortion training image includes a first text region; the corresponding actual correction image includes a second text region corresponding to the first text region; the second control point further includes implicit straight line control points; Accordingly, the training module is further configured as follows: Determine the center position of each character in each second text area, and use each center position as a hidden line control point.

[0016] In some optional implementations, each set of metadata also includes the actual corrected image corresponding to each distorted training image; Accordingly, the training module is further configured as follows: Before training the correction model to be trained based on the control point loss to obtain the trained correction model, the following is performed: Each predicted offset matrix and its corresponding actual offset matrix are input into the offset loss function to obtain the corresponding offset loss; Determine the total variational denoising value corresponding to each prediction offset matrix; The image loss between each corrected prediction image and the corresponding actual corrected image is determined based on the image loss function. The step of training the correction model to be trained based on the control point loss to obtain the trained correction model includes: For each corrected prediction image, the corresponding control point loss, the corresponding offset loss, the corresponding total variation denoising value, and the corresponding image loss are weighted to obtain the corresponding multimodal loss. The correction model to be trained is trained based on the multimodal loss to obtain the trained correction model.

[0017] Thirdly, embodiments of this application also provide an image correction device, the device comprising: One or more processors; Storage device, configured to store one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the image correction method described in the embodiments of this application.

[0018] Fourthly, embodiments of this application also provide a non-volatile storage medium for storing computer-executable instructions, which, when executed by a computer processor, are configured to perform the image correction method described in embodiments of this application.

[0019] Fifthly, embodiments of this application also provide a computer program product, which includes a computer program stored in a computer-readable storage medium. At least one processor of the device reads from the computer-readable storage medium and executes the computer program, causing the device to perform the image correction method described in embodiments of this application.

[0020] As can be seen from the above, the image correction method and apparatus provided in this application achieve image correction through a correction model trained with a control point loss function. On the one hand, after specialized training, the correction model can accurately identify and correct distortion problems in images, making the corrected images more consistent with the normal presentation of the actual scene, and greatly improving the readability and visual effect of the images. On the other hand, this method does not rely on expensive and bulky special equipment. It can efficiently process images to be corrected using only the trained correction model, which not only reduces the correction cost but also has strong scene adaptability, can flexibly cope with various image correction needs with distortion, and has a wider range of applications. Attached Figure Description

[0021] To more clearly illustrate the technical solutions in this application or related technologies, the drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the drawings described below are only embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0022] Figure 1 A flowchart of an image correction method provided in an embodiment of this application; Figure 2 A flowchart illustrating a correction model training method provided in this application embodiment; Figure 3 A flowchart illustrating a method for determining control point loss provided in an embodiment of this application; Figure 4 A flowchart illustrating a control point acquisition method provided in an embodiment of this application; Figure 5 A flowchart illustrating a method based on a multimodal loss function provided in this application embodiment; Figure 6 A flowchart illustrating a correction model processing method provided in this application embodiment; Figure 7 A flowchart illustrating an offset matrix prediction method provided in this application embodiment; Figure 8 A structural diagram of an attention unit provided in an embodiment of this application; Figure 9 A structural diagram of a depth-separable convolutional layer provided in an embodiment of this application; Figure 10 A flowchart illustrating a depthwise separable convolution method provided in this application embodiment; Figure 11 A structural diagram of a FasterASPP structure provided in an embodiment of this application; Figure 12A flowchart illustrating a multi-scale feature map determination method provided in this application embodiment; Figure 13 A framework diagram of another correction model training method provided in the embodiments of this application; Figure 14 A structural block diagram of an image correction device provided in an embodiment of this application; Figure 15 This application provides a schematic diagram of the structure of an image correction device. Detailed Implementation

[0023] The embodiments of this application will be further described in detail below with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are merely illustrative of the embodiments of this application and are not intended to limit the scope of the embodiments. Furthermore, it should be noted that, for ease of description, only the parts relevant to the embodiments of this application are shown in the accompanying drawings, not the entire structure.

[0024] The terms "first," "second," etc., used in the specification and claims of this application are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first," "second," etc., are generally of the same class and are not limited in number; for example, a first object can be one or more. Furthermore, "and / or" in the specification and claims indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects are in an "or" relationship. Words such as "comprising" or "including" mean that the element or object preceding the word covers the element or object listed after the word and its equivalents, without excluding other elements or objects. Words such as "connected" or "linked" are not limited to physical or mechanical connections but can include electrical connections, whether direct or indirect. "Above," "below," "left," "right," etc., are used only to indicate relative positional relationships; when the absolute position of the described object changes, the relative positional relationship may also change accordingly.

[0025] As described in the background section, the existing image correction methods are still insufficient to meet the needs of practical use.

[0026] In the process of implementing this application, the applicant discovered that the main problem with the relevant image correction methods is that when correcting images, expensive optical scanning equipment is often used to obtain the three-dimensional structural information of the document through technologies such as lasers or structured light. Then, the distortion parameters are calculated through geometric algorithms, and the document's curvature is flattened accordingly. However, such optical scanning equipment is expensive and bulky, which not only results in excessive costs but also limits the applicable scenarios.

[0027] In other cases, when correcting images, image algorithms are used to estimate and model based on parameters such as the curvature layout information of text lines in the document, thereby correcting the curved document. However, this method requires setting too many hyperparameters, which often cannot adapt to various scenarios. Too many parameters involved in the calculation result in insufficient robustness of most image algorithms and long processing time, which also limits the applicable scenarios.

[0028] Based on this, one or more embodiments of this application provide an image correction method. In the training of the correction model, the correction model is obtained by training using the control point loss function. In some cases, the control point loss function can be combined with the offset loss function and the image loss function to form a multimodal loss function, so that the correction model trained using the multimodal loss function can correct distorted images more accurately and stably.

[0029] Meanwhile, based on dilated convolution with different dilation factors in the correction model, multi-scale feature maps are collected under different receptive fields. This enables accurate and fast prediction of the offset matrix without involving a large number of parameters. Consequently, after mapping the image to be corrected using this offset matrix, an accurate corrected image can be obtained.

[0030] Because the correction model adjusts the receptive field by different inflation factors, it avoids a large number of parameter calculations and achieves lightweight computation. Furthermore, during training, it considers multiple dimensions of loss, including offset loss function, image loss function, and control point loss function. This allows the trained correction model to achieve lightweight computation and obtain better correction results even in application scenarios such as image correction using electronic devices with insufficient computing power or image correction scenarios where optical scanning equipment is not suitable.

[0031] The image correction method provided in this application embodiment can be executed by a computer device. The computer device refers to any electronic device with data computing, processing and storage capabilities, such as mobile phones, PCs (Personal Computers), tablet computers and other terminal devices. This application embodiment does not limit this.

[0032] The embodiments of this application are described in detail below with reference to the accompanying drawings.

[0033] Figure 1 This is a flowchart illustrating an image correction method provided in an embodiment of this application. Figure 1 As shown, it includes the following steps: Step S101: Obtain the image to be corrected.

[0034] Among them, the image to be corrected is an image whose content is distorted, such as images, graphics, photographs and / or text in the image being displayed as curved or other non-realistic distortions.

[0035] Step S102: Input the image to be corrected into the trained correction model, and process the image to be corrected using the trained correction model to obtain the corrected image.

[0036] In the trained correction model, one or more dilated convolutional layers have the same convolutional kernel but different dilation factors. The dilation factor is used to control the receptive field of each dilated convolution.

[0037] Specifically, when performing dilated convolution in a dilated convolution layer, holes can be inserted into the convolution kernel. The corresponding dilation factor can determine the number of inserted holes, thereby creating an effect of expanding the receptive field.

[0038] Based on different dilation factors, each dilated convolutional layer can form different receptive fields, thus enabling each dilated convolution to collect features at different scales of the receptive fields. Furthermore, after fusing the collection results of each dilated convolution, the resulting feature map is a multi-scale feature map that incorporates multiple receptive fields.

[0039] The correction model in this step can be trained using a control point loss function. Before correcting the image to be corrected, the correction model to be trained needs to be trained to obtain a completed correction model, so that the completed correction code model can be used for subsequent steps.

[0040] By using the control point loss function, the training effect can be represented by the straight line reflected in the correction model to be trained. This allows for better training results through various dimensions, resulting in a correction model with better correction performance.

[0041] Figure 2 This is a flowchart illustrating a correction model training method provided in an embodiment of this application. Figure 2 As shown, it includes the following steps: Step S201: Obtain the training dataset, which includes multiple sets of metadata, each set of metadata including multiple distorted training images.

[0042] During the training process, it is necessary to first obtain the training dataset for training.

[0043] The training dataset contains multiple sets of metadata. Each set of original data contains distorted training images used as input to the correction model to be trained. These distorted training images are images with distorted content. Each set of metadata may also include the actual correction images corresponding to the distorted training images. These actual correction images are images that are expected to be free of distortion.

[0044] In this step, during the acquisition of each distortion training image, text, images, or graphics can be captured by means of photography or scanning. During the acquisition, the carrier of the text, image, or graphics is bent to obtain a distortion training image with distorted content. If the carrier of the text, image, or graphics is not bent or the bending is negligible, an actual corrected image with no distortion or negligible distortion is acquired.

[0045] Specifically, the carrier of text, images, or graphics can be, for example, a printed piece of paper with text, images, or graphics. Different bending methods of the same paper can produce multiple different distortion training images. That is, paper with the same text, image, or graphic content can be combined with the same actual correction image to form multiple sets of metadata through different distortion methods.

[0046] Among them, unbent sheets of paper taken from different angles can be used as distorted training images with perspective distortion.

[0047] Step S202: Input a predetermined number of metadata sets into the correction model to be trained, and output the prediction offset matrix corresponding to the distorted training image of each metadata set.

[0048] Based on the training dataset determined in S201 above, all metadata can be divided into multiple parts. That is, each part contains multiple sets of metadata, and each part constitutes all the metadata. In each round of training, the metadata of each set in one part is input into the correction model to be trained.

[0049] In the correction model to be trained, the number of metadata groups processed in each round of training can be set, and in the process of dividing all metadata in the training dataset into multiple parts, the division can be carried out according to the set number of groups, so that the number of metadata groups in each part is the same as the set number of groups.

[0050] Based on this, after inputting the metadata of any part into the correction model to be trained, the correction model to be trained can predict the prediction offset matrix corresponding to the distorted training image for each set of metadata.

[0051] Step S203: Map each pixel in each distorted training image according to the prediction offset matrix to obtain the corresponding corrected prediction image; determine the control point loss corresponding to each corrected prediction image according to the control point loss function, where each control point loss represents the degree of distortion of the straight line in the corresponding corrected prediction image.

[0052] Based on the prediction offset matrices determined in S202 above, the corresponding distorted training images can be corrected according to the prediction offset matrices to obtain corrected prediction images.

[0053] Specifically, since the prediction offset matrix represents the positional difference between each pixel in the distorted training image and the corresponding pixel in the actual corrected image without distortion, the Remap function can be pre-set, and the aforementioned prediction offset matrix can be input into the Remap function to map each pixel in the corresponding distorted training image, thereby obtaining the corrected prediction image corresponding to the distorted training image after the mapping process.

[0054] The Remap function can be a pre-defined function in the database, such as a function in the OpenCV (Open Source Computer Vision) library. The Remap function can adjust the position of each pixel in the distorted training image based on the input prediction offset matrix, so that after adjusting the position of each pixel, a corrected prediction image is obtained. Ideally, the position of each pixel in the corrected prediction image should be the same as the position of the corresponding pixel in the actual corrected image without distortion.

[0055] Based on the pre-set control point loss function and the determined corrected prediction image, the corrected prediction image can be input into the control point loss function for calculation, thereby obtaining the control point loss.

[0056] The control point loss specifically characterizes the degree of distortion of straight lines in the corrected prediction image. Ideally, the degree of distortion of each straight line in the corrected prediction image should be similar to that of the straight lines in the actual corrected image; that is, there should be no distortion, or the degree of distortion should be small.

[0057] The straight lines in the corrected prediction image and the actual corrected image can be, for example, at least one of: displayed straight lines and hidden straight lines. Displayed straight lines can be, for example, straight line segments that actually appear in the corrected prediction image and the actual corrected image; while hidden straight lines can be, for example, straight lines or line boundaries formed by their text lines in the corrected prediction image and the actual corrected image. The line boundaries can be, for example, character lines formed along each straight line of text, and / or boundaries formed by aligning multiple text lines at the beginning of the line.

[0058] In the process of obtaining straight lines in any image, for displayed straight lines, a straight line detection algorithm can be used. For example, the Hough line detection algorithm can be used to filter out each displayed straight line and filter out line segments whose length is shorter than a preset length threshold. For hidden straight lines, text detection can be used. For example, text box detection can be performed on the image based on DBNet (Differentiable Binarization Network) to obtain text detection boxes. Since the text detection boxes are composed of text lines, the straight lines or line boundaries formed by the text lines in the text detection boxes can be identified as hidden straight lines.

[0059] Step S204: Train the correction model to be trained according to the control point loss to obtain the completed correction model.

[0060] Based on the control point loss determined in S203 above, the relationship between the control point loss and the pre-set control point loss threshold can be used to determine whether training is complete.

[0061] In cases where the control point loss exceeds a preset control point loss threshold and / or the training dataset has not completed a predetermined number of training iterations, the parameters in the correction model to be trained are adjusted. Different metadata of the same number of groups are input into the adjusted correction model, and correction prediction images corresponding to the distorted training images of each group of metadata are generated respectively.

[0062] Specifically, if the control point loss is greater than the control point loss threshold, it can be considered that the straight lines in the generated corrected prediction image still have a large error compared with the expected actual corrected image. That is, each straight line still has a serious distortion, and the correction model to be trained needs to be continuously trained.

[0063] Based on this, the parameters in the correction model to be trained can be adjusted according to the difference between the control point loss and the control point loss threshold. After adjustment, based on the multiple parts of the training dataset mentioned above, the metadata of another part is input into the correction model to be trained for the next round of training. In the next round of training, the prediction offset matrix corresponding to each distorted training image is predicted again, and then Remap mapping is performed again based on each prediction offset matrix to generate each correction prediction image, and the control point loss is calculated again. In other cases, the number of training rounds can be preset. If the training dataset has not completed the required number of rounds, the parameters of the correction model to be trained can be adjusted based on the difference between the control point loss and the control point loss threshold. After adjustment, based on the multiple parts of the training dataset mentioned above, the metadata of another part is input into the correction model to be trained for the next round of training. In the next round of training, the prediction offset matrix corresponding to each distorted training image is predicted again, and Remap mapping is performed again based on each prediction offset matrix to generate each correction prediction image, and the control point loss is calculated again. In other cases, the control point loss and the number of training rounds can be combined for judgment. For example, if the control point loss is greater than the control point loss threshold, it is then determined whether the training dataset has completed the predetermined number of rounds. If the number of rounds has not been completed, the parameters in the correction model to be trained are adjusted based on the difference between the control point loss and the control point loss threshold. After adjustment, based on the multiple parts of the training dataset mentioned above, the metadata of another part is input into the correction model to be trained for the next round of training. In the next round of training, the prediction offset matrix corresponding to each distorted training image is predicted again, and Remap mapping is performed again based on each prediction offset matrix to generate each correction prediction image, and the control point loss is calculated again. Based on the control point loss determined in S203 above, if the control point loss is less than or equal to the control point loss value threshold and / or the training dataset has completed a predetermined number of training iterations, the current correction model to be trained is determined as the completed correction model.

[0064] Specifically, if the control point loss is less than or equal to the control point loss threshold, the generated corrected prediction image can be considered to be the same as or close to the expected actual corrected image, that is, the straight lines are not distorted or the distortion degree of each straight line is small, and the corrected model can be considered to have been trained.

[0065] In other cases, based on a pre-set number of training rounds, it can be determined that a trained correction model has been obtained when the training dataset has completed that number of rounds. In other cases, the control point loss and the number of training rounds can be combined for judgment. For example, if the control point loss is less than or equal to the control point loss threshold, it can be determined whether the training dataset has completed the predetermined number of rounds. If the training dataset has completed the predetermined number of rounds, it can be determined that the trained correction model has been obtained.

[0066] It can be seen that training based on the control point loss function brings significant technical improvements to the correction model. By constructing a training dataset containing distorted training images, actual corrected images, and corresponding labeled data, the model can comprehensively learn the complex features when content distortion occurs. During training, the control point loss function is combined with dynamically adjusted correction models, along with a predetermined number of iterations, ensuring the stability and reliability of model convergence. This results in a significantly improved correction model in terms of both accuracy and robustness, enabling it to adapt to image correction needs in various complex scenarios.

[0067] Figure 3 A flowchart illustrating a method for determining control point loss as provided in an embodiment of this application. Figure 3 As shown, it includes the following steps: Step S301: Determine at least one first straight line in each corrected prediction image, and determine at least two adjacent vector segments in each first straight line.

[0068] In the embodiments of this application, each distortion training image includes a first straight line region containing multiple first straight lines. The actual correction image corresponding to each distortion training image includes a second straight line region corresponding to the first straight line region, which contains second straight lines corresponding to each of the first straight lines. Furthermore, each set of metadata also includes a straight line mask corresponding to the second straight line region.

[0069] One method is to determine the line mask using line detection.

[0070] Specifically, since there are both visible and hidden lines in the actual corrected image, different line detection methods can be used to perform two line detections.

[0071] The displayed straight line can be, for example, a straight line that exists and is specifically displayed in the actual corrected image; the hidden straight line can be, for example, a straight line formed by text lines or text columns in the actual corrected image, such as the horizontal straight line represented by each text line, and the vertical straight line displayed between each text line due to formatting settings such as left alignment or right alignment.

[0072] When performing line detection on actual corrected images, the Hough line detection algorithm can be used to filter out each visible line and remove line segments shorter than a preset length threshold. For hidden lines, they can be obtained through text detection. For example, text box detection can be performed on the image based on DBNet (Differentiable Binarization Network) to obtain text detection boxes. Since the text detection boxes are composed of text lines, the lines or line boundaries formed by the text lines in the text detection boxes can be identified as hidden lines.

[0073] Based on this, a line mask can be obtained after line detection.

[0074] Furthermore, by applying the straight line mask to the actual corrected image, a second straight line region containing each of the second straight lines can be obtained, and by applying the straight line mask to the corrected prediction image, a first straight line region containing each of the first straight lines can be obtained.

[0075] Each first control point is a point on each displayed line and hidden line in the corrected prediction image; that is, each first control point is a discontinuous representation of each line in the corrected prediction image.

[0076] Therefore, when there are three consecutive first control points on the same straight line, two adjacent collinear vectors can be determined on that straight line; when there are three consecutive first control points on the same straight line, more than two adjacent collinear vectors can be determined on that straight line.

[0077] Specifically, for three consecutive first control points , and The first control point in the middle can be used. Compared with the previous first control point Calculate the first vector among two adjacent collinear vectors. ,in, It represents the j-th vector among two adjacent collinear vectors, i.e., the preceding vector.

[0078] Furthermore, the first control point located in the middle can be utilized. With the next first control point Calculate the second vector in a pair of adjacent collinear vectors. ,in, It represents the (j+1)th vector among two adjacent collinear vectors, i.e., the next vector.

[0079] Step S302: Determine the cosine similarity of any two adjacent vectors on the first straight line in each corrected prediction image.

[0080] If the corrected prediction image has no distortion or very little distortion, the cosine similarity of each pair of adjacent collinear vectors should be 1 or approximately 1; if the corrected prediction image has significant distortion, the cosine similarity of each pair of adjacent collinear vectors should not be 1.

[0081] Based on this, the directional consistency of each pair of adjacent collinear vectors can be measured by the difference between 1 and cosine similarity.

[0082] In other words, the smaller the difference, the more consistent the directions of each pair of adjacent collinear vectors are, until the difference is 0, at which point the directions of each pair of adjacent collinear vectors are completely consistent, i.e., they are all on a straight line without distortion. On the other hand, the larger the difference, the greater the difference in the directions of each pair of adjacent collinear vectors, i.e., each pair of adjacent collinear vectors are actually on different straight lines. Thus, it can be determined that the line segment where each pair of adjacent collinear vectors are located is not a straight line, but a distorted line segment.

[0083] In some examples, cosine similarity can be calculated using the formula (1) shown below: (1) in, Let represent the cosine similarity of the i-th pair of collinear vectors. This indicates the preset coefficient factor.

[0084] Step S303: Determine the average difference between each cosine similarity and the preset value as the corresponding control point loss.

[0085] Based on the pre-identified similarity determined in step S302, for each straight line in the corrected prediction image, the difference between the corresponding cosine similarity and 1 can be determined, and the mean of each difference can be determined and used as the control point loss. Thus, the degree of distortion of the entire corrected prediction image can be determined using the control point loss.

[0086] In some examples, the control point loss can be expressed as The specific calculation method can be, for example, the formula (2) shown below: (2) in, Let N represent the difference between the 1 and cosine similarity of the i-th pair of adjacent collinear vectors, and let N represent the number of cosine similarity calculations, which is also the number of lines.

[0087] Among them, the difference This can be expressed as formula (3) as shown below: (3) As can be seen, when each first straight line includes at least two vector segments, and each vector segment includes at least two first control points, the control points can be used to determine the two adjacent collinear vectors of each first straight line. Thus, the cosine similarity between the two adjacent collinear vectors can be calculated. Furthermore, the difference between the cosine similarity and 1 can be used to measure whether the first straight line is distorted and the degree of distortion. Based on this, the mean of the above difference for each first straight line can be determined and used as the control point loss for correcting the predicted image.

[0088] It should be noted that the control point loss determined above can be the control point loss of the displayed straight line determined using the displayed straight line. Alternatively, it can be the control point loss of the hidden line determined by the hidden line. It can be selected to display the control point loss of the straight line or hide the control point loss of the straight line, or to use both the control point loss of the straight line and the control point loss of the straight line at the same time, and set their respective weights for the control point loss of the straight line and the control point loss of the straight line respectively.

[0089] Figure 4 This is a flowchart illustrating a control point acquisition method provided in an embodiment of this application. Figure 12 As shown, it includes the following steps: Step S401: Obtain multiple second control points in the actual corrected image corresponding to each distortion training image.

[0090] The second control point includes at least one of a display line control point on a display line and a hidden line control point on a hidden line.

[0091] Based on this, in the process of acquiring multiple second control points for each actual corrected image, the displayed straight line control points and the hidden straight line control points can be acquired in different ways.

[0092] Specifically, for the display line control points on the display lines, each display line can be determined based on the previously determined second line area and the line mask of the previously determined second line area, and each display line control point can be further determined on each display line based on the determined display lines.

[0093] In determining each displayed straight line in each actual corrected image, an edge detection algorithm, such as the Canny operator, can be used to detect the edge of the second straight line region in each actual corrected image.

[0094] In some cases, individual lines can be detected directly from the second line area and used as the corresponding display lines.

[0095] In other cases, based on the line mask corresponding to the second line region determined above, the intersection between the line mask and the edge of the second line region can be taken, thereby obtaining a more accurate display line.

[0096] Based on the identified display lines, the endpoint coordinates of each display line can be detected. For example, Hough line detection can be used to detect the endpoint coordinates of each display line.

[0097] Furthermore, based on the detected endpoint coordinates, the displayed straight line can be divided into multiple equal segments, thereby determining the coordinates of the connection point between every two adjacent segments, i.e., the coordinates of the division point, and defining it as the display line control point of the displayed straight line.

[0098] In the process of obtaining the hidden line control points on the hidden lines, the first text region in each distortion training image and the second text region corresponding to the first text region in the corresponding actual correction image can be determined first. Then, the control points of each hidden line can be determined based on the determined second text region.

[0099] Specifically, based on the second text region in each actual corrected image, a preset character detection algorithm, such as the YOLO algorithm (You Only Look Once Text Detection, a target detection algorithm that can complete target localization and classification in a single forward propagation), can be used to detect each character in the second text region and determine the coordinates of the center position of each character.

[0100] Based on this, the coordinates of the center position of each character in the character row or column arranged in a straight line in the second text area can be used as the hidden line control point of the hidden line formed by the character row or column.

[0101] Step S402: Using the actual offset matrix, map each second control point in each actual corrected image to the corresponding twisted training image to obtain multiple twisted image control points in each twisted training image.

[0102] Based on the second control points obtained for each actual corrected image in step S401 above, in the process of mapping each second control point of the actual corrected image to the corresponding corrected prediction image, each second control point of the actual corrected image can first be mapped to the corresponding distortion training image in the metadata.

[0103] Specifically, since the actual offset matrix in each set of metadata represents the mapping relationship between each pixel in the actual corrected image and the distorted training image in that set of metadata, the actual offset matrix in the metadata corresponding to the actual corrected image can be used to map each second control point in the actual corrected image to the distorted training image, thereby determining multiple corresponding distorted image control points in each distorted training image.

[0104] In the process of mapping each second control point in the actual corrected image to the distorted training image using the actual offset matrix, the actual offset matrix can be used as the backward field of the backward mapping, and the backward field can be transformed into the forward field of the forward mapping.

[0105] Therefore, the forward field can be used to map each of the second control points in the actual correction image to the corresponding distortion training image.

[0106] In the specific process of mapping, since the actual offset matrix represents the positional difference between each pixel in the distorted training image and the corresponding pixel in the undistorted actual corrected image, the forward field corresponding to the actual offset matrix can be used to adjust the position of each control point in the actual corrected image. After adjusting the position of each control point, the control points of each distorted image can be obtained in the corresponding distorted training image.

[0107] Step S403: Using the prediction offset matrix, map each twisted image control point in each twisted training image to the corresponding corrected prediction image to obtain multiple first control points in each first straight line in the corresponding corrected prediction image.

[0108] Based on the multiple warped image control points determined in the warped training image in step S402 above, each warped image control point in the warped training image in each metadata can be mapped to the corresponding corrected prediction image, thereby determining multiple first control points in each corrected prediction image.

[0109] Specifically, since the prediction offset matrix predicted using the distorted training images in each set of metadata represents the mapping relationship between the distorted training images and the corrected prediction images, the predicted prediction offset matrix table can be used to map each distorted image control point in the corresponding distorted training image to the corrected prediction image, thus obtaining multiple corresponding first control points.

[0110] In the specific process of mapping, since the prediction offset matrix represents the positional difference between each pixel in the distorted training image and the corresponding pixel in the expected distortion-free corrected prediction image, the position of each distorted image control point in the distorted training image can be adjusted using the prediction offset matrix. After adjusting the position of each distorted image control point, each first control point can be obtained in the corresponding corrected prediction image.

[0111] As can be seen, by acquiring and mapping control points in stages, a precise reference basis is provided for judging the degree of distortion in the corrected prediction image. Specifically, by extracting explicit and implicit line control points from the actual corrected image, key positions representing the geometric shape in the image can be comprehensively covered; by using the actual offset matrix to map the second control point to the control point of the distorted image, and then using the prediction offset matrix to map the first control point of the corrected prediction image, the entire process establishes a correspondence between control points in the actual corrected image, the distorted training image, and the corrected prediction image. This allows the geometric shape of the corrected prediction image to be intuitively represented by control points, providing clear data support for subsequent judgment of whether lines are distorted and evaluation of the correction effect.

[0112] Meanwhile, this control point acquisition process strictly maps the actual offset matrix to the predicted offset matrix, ensuring the positional correlation and accuracy of control points at each stage and avoiding misjudgments of the correction effect due to control point misalignment. Whether it is the equally divided control points on the displayed straight line or the implicit control points corresponding to the character centers in the text region, the mapping process is closely combined with the offset information related to image distortion. This ensures that the final first control point can truly reflect the geometric state of the predicted image, laying a reliable foundation for subsequently measuring the correction quality through the control point loss function and further improving the accuracy of the correction model in correcting image geometric distortion.

[0113] Before training the corrected model to be trained based on the control point loss to obtain the trained corrected model, a multimodal loss function can be constructed based on the control point loss function. This allows the corrected model to be trained based on the multimodal loss including the control point loss. Figure 5 A flowchart illustrating a training method based on a multimodal loss function provided in this application embodiment. Figure 5 As shown, it includes the following steps: Step S501: Input each predicted offset matrix and the corresponding actual offset matrix into the offset loss function to obtain the corresponding offset loss.

[0114] Based on the offset loss sub-function set in the multimodal loss function mentioned above, the offset loss can be calculated using this offset loss sub-function.

[0115] Specifically, the offset loss represents the difference between the predicted offset matrix output by the correction model to be trained and the actual offset matrix, which can be represented by, for example, L2 (mean squared error loss).

[0116] Step S502: Determine the total variation denoising value corresponding to each prediction offset matrix.

[0117] In the multimodal loss function, a total variation denoising function can be further set. This total variation denoising function can ensure the smoothness of the predicted offset matrix and avoid sudden changes in pixel offset.

[0118] Based on this, the TVLOSS (total variation denoising value) can be calculated on the predicted offset matrix using the total variation denoising function.

[0119] Step S503: Determine the image loss between each corrected prediction image and the corresponding actual corrected image based on the image loss function.

[0120] Based on the image loss sub-function set in the multimodal loss function mentioned above, the image loss between the corrected prediction image and the corresponding actual corrected image can be calculated using this image loss sub-function.

[0121] Specifically, based on the text region loss function and the line region loss function set in the image loss subfunction, the text region loss and the line region loss can be determined respectively.

[0122] In the process of determining the text region loss using the text region loss function, the text edge mask can be used to determine the first text region in the corrected prediction image and the second text region in the actual corrected image.

[0123] Based on this, the difference between the first and second text regions, i.e., the text region loss, can be determined using the text region loss function, which can be expressed as: ).

[0124] Among them, the text region loss function This indicates the calculation of text region loss using the equation for mean absolute error loss. This indicates the area defined in the actual corrected image using a text edge mask. This represents the region identified in the corrected prediction image through a text edge mask. The text region loss can also be called the high-frequency information reconstruction loss, which is used to represent the loss of high-frequency information during the reconstruction process.

[0125] In some examples, The specific calculation method can be, for example, the formula (4) shown below: (4) Where n represents the actual number of corrected images. Indicates the i-th , Indicates the i-th .

[0126] In the embodiments of this application, the text portions in the correction prediction image and the actual correction image, as well as the edge regions of the text portions, can all be regarded as text regions.

[0127] Furthermore, in the process of determining the straight-line region loss using the straight-line region loss function, a straight-line mask can be used to determine the first straight-line region in the corrected prediction image and the second straight-line region in the actual corrected image.

[0128] Based on this, the difference between the first and second straight-line regions can be determined using the straight-line region loss function, which can be expressed as: .

[0129] Among them, the loss function in the straight line region This represents the equation for calculating the loss in the straight-line region using the mean absolute error loss formula. This represents the region defined in the actual corrected image using a linear mask. This represents the region identified in the corrected prediction image using a line mask. The line region loss, also known as the line reconstruction loss, is used to represent the loss of line data during the reconstruction process.

[0130] In some examples, The specific calculation method can be, for example, the formula (5) shown below: (5) in, Indicates the j-th , Indicates the j-th After determining the text region loss and the line region loss, the image loss can be obtained by weighting the text region loss and the line region loss.

[0131] The image loss can be expressed as shown in the following formula (6): (6) in, Indicates image loss. Indicates text area loss. Indicates the loss over the linear region. The weights representing the text region loss range from 0 to 1, and can be, for example, 0.1, 0.2, or 0.3; The weight represents the loss of the straight line region, and its value ranges from 0 to 1. For example, it can be 0.9, 0.2 or 0.7. The sum of the weight of the text region loss and the weight of the straight line region loss is 1.

[0132] In other cases, where there are images or graphics in the corrected prediction image, a graphics loss function can be constructed using the mean absolute error loss, and the difference between the image or graphics region in the corrected prediction image and the corresponding image or graphics region in the actual corrected image can be calculated and used as the image reconstruction loss.

[0133] Specifically, the graphics loss function can be expressed as: .

[0134] Among them, the graph loss function This represents the equation used to calculate image reconstruction loss using the mean absolute error loss formula. This refers to the image or graphic region in the actual corrected image. This indicates the corresponding image or graphic region in the corrected prediction image.

[0135] In some examples, The specific calculation method can be, for example, the formula (7) shown below: (7) in, Indicates the kth , Indicates the kth .

[0136] Based on this, the image loss can be obtained by weighting the image reconstruction loss, text region loss, and line region loss.

[0137] In this case, the image loss can be expressed as shown in formula (8) below. (8) in, Indicates the image reconstruction loss. Indicates text area loss. Indicates the loss over the linear region. The weights represent the image reconstruction loss and range from 0 to 1, such as 0.1, 0.2, or 0.3. The weights representing the text region loss range from 0 to 1, and can be, for example, 0.3, 0.4, or 0.5. The weight representing the loss of the straight line region ranges from 0 to 1, and can be, for example, 0.6, 0.4, or 0.2; the sum of the weights of the image reconstruction loss, the text region loss, and the straight line region loss is 1.

[0138] Step S504: For each corrected prediction image, the corresponding control point loss, the corresponding offset loss, the corresponding total variation denoising value, and the corresponding image loss are weighted to obtain the corresponding multimodal loss.

[0139] Based on the offset loss determined in S501, the total variation denoising value determined in S502, the image loss determined in S503, and the control point loss determined in the aforementioned embodiment, the losses of multiple dimensions can be weighted to obtain a multimodal loss function representing multiple dimensions.

[0140] The multimodal loss function with multiple dimensions can be expressed as shown in the following formula (9): (9) Here, Loss represents the multimodal loss across multiple dimensions. The weights representing the offset loss range from 0 to 1, and can be, for example, 0.2, 0.3, or 0.4. The weights represent the total variation denoising values, and their values ​​range from 0 to 1, for example, 0.3, 0.4 or 0.5; This represents the weight of the control point loss for the displayed straight line, and its value ranges from 0 to 1; The weight of the control point loss for the hidden line is denoised as 0, and its value ranges from 0 to 1. The sum of the weights of the offset loss, the total variation denoising value, the image reconstruction loss, the text region loss, the line region loss, the control point loss for the displayed line, and the control point loss for the hidden line is 1.

[0141] Step S505: Train the correction model to be trained according to the multimodal loss to obtain the completed correction model.

[0142] In this step, during the training of the correction model to be trained, a multimodal loss function can be used for training. Since the multimodal loss function contains loss sub-functions of multiple dimensions, the training effect of the correction model to be trained can be measured from multiple dimensions during training.

[0143] Specifically, the offset loss function can be used to measure the training effect of the correction model under training in the offset field dimension; the image loss function can be used to measure the training effect of the correction model under training in the overall image dimension; the control point loss function can be used to represent the training effect by the straight line reflected in the correction model under training; and the total variation denoising function can be used to ensure the smoothness of the predicted offset matrix.

[0144] Based on this, the correction model in this step is trained through multi-dimensional multimodal supervision. In each training round, in addition to the difference between the predicted image and the actual distortion-free image in the image dimension, the difference between the offset field predicted by the training image and the actual offset field in the offset field dimension is also utilized to obtain better training results and a correction model with better correction effect.

[0145] As can be seen, the embodiments of this application train the correction model to be trained using a multimodal loss function. By combining the synergistic effects of the offset loss sub-function, image loss sub-function, and control point loss sub-function, the model training process is constrained from multiple dimensions, effectively improving the correction performance of the trained model. Compared to models trained using traditional single-dimensional loss functions, this method allows the correction model to fully focus on key features related to image distortion during the learning process, significantly enhancing the model's adaptability to various types of distorted images and ensuring the accuracy and stability of the correction results.

[0146] In a specific example, based on the multimodal loss determined in S504 above, the completion of training can be determined by comparing the magnitude relationship between the multimodal loss and the preset multimodal loss threshold, as well as the preset number of training iterations.

[0147] In cases where the multimodal loss value is greater than the preset loss value threshold and / or the training dataset has not completed a predetermined number of training iterations, the parameters in the correction model to be trained are adjusted, and different metadata of the same number of groups are input into the adjusted correction model to generate correction prediction images corresponding to the distorted training images of each group of metadata.

[0148] Specifically, if the multimodal loss value is greater than the loss threshold, it can be considered that the generated corrected prediction image still has a large error compared with the expected actual corrected image, and the correction model to be trained still needs to be continuously trained.

[0149] Based on this, the parameters in the correction model to be trained can be adjusted according to the difference between the multimodal loss value and the loss threshold. After adjustment, based on the multiple parts of the training dataset mentioned above, the metadata of another part is input into the correction model to be trained for the next round of training. In the next round of training, the prediction offset matrix corresponding to each distorted training image is predicted again, and then Remap mapping is performed again based on each prediction offset matrix to generate each correction prediction image, and the multimodal loss value is calculated again. In other cases, the number of training rounds can be preset. Before the training dataset completes the required number of rounds, the parameters in the correction model to be trained can be adjusted based on the difference between the multimodal loss value and the loss threshold. After adjustment, based on the multiple parts of the training dataset mentioned above, the metadata of another part is input into the correction model to be trained for the next round of training. In the next round of training, the prediction offset matrix corresponding to each distorted training image is predicted again, and Remap mapping is performed again based on each prediction offset matrix to generate each corrected prediction image, and the multimodal loss value is calculated again. In other cases, the multimodal loss value and the number of training rounds can be combined for judgment. For example, if the multimodal loss value is greater than the loss threshold, it is then determined whether the training dataset has completed the predetermined number of rounds. If the number of rounds has not been completed, the parameters in the correction model to be trained are adjusted based on the difference between the multimodal loss value and the loss threshold. After adjustment, based on the multiple parts of the training dataset mentioned above, the metadata of another part is input into the correction model to be trained for the next round of training. In the next round of training, the prediction offset matrix corresponding to each distorted training image is predicted again, and Remap mapping is performed again based on each prediction offset matrix to generate each corrected prediction image, and the multimodal loss value is calculated again. Based on the determined multimodal loss value, if the multimodal loss value is less than or equal to the loss value threshold and / or the training dataset has completed a predetermined number of training iterations, the current correction model to be trained is determined as the completed correction model.

[0150] Specifically, if the multimodal loss value is less than or equal to the loss threshold, the generated corrected prediction image can be considered to be the same as or close to the expected actual corrected image, with a small error between them, and the corrected model can be considered to have been trained.

[0151] In other cases, based on a pre-set number of training rounds, it can be determined that a trained correction model has been obtained when the training dataset has completed that number of rounds. In other cases, the multimodal loss value and the number of training epochs can be combined for judgment. For example, if the multimodal loss value is less than or equal to the loss threshold, it can be determined whether the training dataset has completed the predetermined number of times. If the training dataset has completed the number of times, it can be determined that the trained correction model has been obtained.

[0152] The above-mentioned hierarchical supervision significantly improves the training effect and correction accuracy of the correction model. Based on a joint optimization strategy using offset loss, image loss, and control point loss functions, multi-dimensional supervision of the offset matrix, image content, and the lines corresponding to each control point is achieved. The offset loss ensures geometric consistency between the predicted offset matrix and the actual distortion, while TVLoss effectively maintains the spatial smoothness of the offset matrix, avoiding unnatural abrupt changes in the corrected image. By further refining the image loss function into text region loss and line region loss, targeted supervision of text and line regions is strengthened, enabling the correction model to maintain good performance in complex distortion correction tasks. Simultaneously, the introduction of the control point loss function allows for additional monitoring of whether each expected line in the corrected prediction image is distorted, thus characterizing whether the corresponding corrected prediction image exhibits distortion.

[0153] Figure 6 A flowchart illustrating a correction model processing method provided in an embodiment of this application. Figure 6 As shown, it includes the following steps: Step S601: The image to be corrected is offset using the trained correction model to obtain the offset matrix of the image to be corrected.

[0154] The offset matrix represents the distortion of the image to be corrected. Each pixel in the offset matrix can be, for example, the positional difference between each pixel in the distorted image to be corrected and the corresponding pixel in the undistorted image.

[0155] In this step, after the image to be corrected is input into the correction model, the correction model can accurately predict the offset matrix of the image to be corrected.

[0156] Step S602: Map each pixel in the image to be corrected according to the offset matrix to obtain the corrected image.

[0157] Based on the offset matrix predicted by the correction model in S601, each element in the offset matrix can be used to map each pixel in the image to be corrected. Thus, the position of each element in the image to be corrected can be adjusted by the aforementioned positional difference to obtain the corrected image.

[0158] Specifically, based on the obtained offset matrix, which represents the positional difference between each pixel in the image to be corrected and the corresponding pixel in the expected distortion-free image, the correction model can adjust the position of each pixel in the image to be corrected according to this offset matrix. After adjusting the position of each pixel, the corrected image is obtained. Ideally, the position of each pixel in the corrected image should be the same as the position of the corresponding pixel in the distortion-free image.

[0159] In some alternative implementations, the mapping operation of the image to be corrected based on the offset matrix can also be performed by another mapping model. That is, after the correction model predicts the offset matrix, the predicted offset matrix and the image to be corrected are input into the mapping model, and the mapping process is performed through the mapping model to obtain the corresponding corrected image.

[0160] Figure 7 This is a flowchart illustrating an offset matrix prediction method provided in an embodiment of this application. Figure 7 As shown, the following steps are performed on the warped training images for each set of metadata: Step S701: Perform depthwise convolution and pointwise convolution on the distorted training image using a depthwise separable convolutional layer to obtain a predicted channel spatial feature map.

[0161] The correction model to be trained in this application includes sequentially connected depthwise separable convolutional layers, pyramid pooling units, and attention units.

[0162] In the process of generating the prediction offset matrix using the correction model to be trained, after inputting each distorted training image into the correction model to be trained, it can first be subjected to depthwise convolution and pointwise convolution by a depthwise separable convolutional layer to obtain the prediction channel space feature map.

[0163] Specifically, for each distorted training image, a depthwise separable convolutional layer is used to extract spatial features from multiple channels and fuse the spatial features of each channel to obtain a predicted channel spatial feature map. The data of each spatial feature can be, for example, a local feature within a single channel. Local features can be, for example, features of dimensions such as the edges, corners, and textures of the image. Multiple channels can be, for example, multiple different pixel values ​​or different grayscale values.

[0164] In this step, after processing by depthwise separable convolutional layers, the width and height of the predicted channel space feature map are both half that of the distorted training image, thus significantly reducing the computational load of subsequent pyramid pooling units and attention units.

[0165] Step S702: Perform regular convolution and dilated convolution on the predicted channel spatial feature map using pyramid pooling units to obtain a multi-scale feature map.

[0166] Based on the predicted channel spatial feature map determined by the aforementioned S702, feature extraction can be performed on it using pyramid pooling units to obtain a multi-scale feature map.

[0167] Specifically, the pyramid pooling unit can extract features from the feature map of the prediction channel space according to multiple different receptive field scales, thereby obtaining feature maps at different receptive field scales, and then fuse the feature maps at different receptive field scales to obtain a multi-scale feature map.

[0168] As can be seen, the features in this multi-scale feature map represent the fusion results of features extracted at different receptive field scales. Therefore, the features in the distorted training image can be represented from multiple scales.

[0169] Step S703: The attention unit performs attention weighting on the multi-scale feature map to generate a predicted weighted feature map.

[0170] Based on the multi-scale feature map determined in S702 above, attention units can be set to weight the features in the multi-scale feature map according to the importance of each region or feature. When a higher weight is set for more important regions or features, the resulting predicted weighted feature map can enhance the performance of important regions or features.

[0171] The dimension of the offset matrix can be twice the product of the length and width of the distorted training image.

[0172] Figure 8 This is a structural diagram of an attention unit provided in an embodiment of this application.

[0173] like Figure 8 As shown, the attention unit can be, for example, SAM (Spatial Attention Unit), where, Figure 8 The input features are the features in the multi-scale feature map obtained in S702 above. By assigning corresponding weights to each feature, attention can be increased to important features. After attention weighting of each feature according to the weights, the optimized feature is the prediction weighted feature map.

[0174] It should be noted that when using the sequentially connected deep separable convolutional layer, pyramid pooling unit, and attention unit in the correction model for feature extraction, after the attention unit outputs the predicted weighted feature map for the first time, the predicted weighted feature map output for the first time can be returned to the deep separable convolutional layer, and the feature extraction operation can be performed again through the deep separable convolutional layer, pyramid pooling unit, and attention unit to obtain a more accurate offset matrix.

[0175] Step S704: Generate a prediction offset matrix based on the prediction weighted feature map.

[0176] Based on the predicted weighted feature map determined in S703 above, the correction model to be trained can generate a predicted offset matrix according to the predicted weighted feature map.

[0177] The prediction offset matrix is ​​a matrix that represents the distortion of the distorted training image. Each pixel in the prediction offset matrix can be, for example, the positional difference between each pixel in the distorted training image and the corresponding pixel in the undistorted image.

[0178] It should be noted that for the trained correction model, the offset matrix of the image to be corrected can also be generated in the manner described in S701-S704 above.

[0179] As described above, through the design of a multimodal loss function, efficient end-to-end model optimization is achieved. This not only automatically generates text edge masks and line masks using paired distorted training images and actual corrected images, significantly reducing manual annotation costs, but also effectively avoids jagged edges or breaks in the corrected images by constraining the smoothness of the offset field through TV Loss. The collaborative design of depthwise separable convolution and the FasterASPP module enables the model to converge quickly during training, significantly improving training efficiency.

[0180] In the embodiments of this application, in the process of determining the multimodal loss using the multimodal loss function, the multimodal loss function can specifically evaluate the corrected and predicted image from the dimensions of the image, the dimensions of the offset matrix, and the angle of the line in the image. Therefore, the multimodal loss function can specifically include an image loss sub-function, an offset loss sub-function, and a control point loss sub-function.

[0181] Since the distorted training image includes text regions and straight line regions, the image loss sub-function can further evaluate the corrected prediction image from the dimensions of the text regions and straight line regions of the distorted training image. Therefore, the image loss sub-function can include text region loss function and straight line region loss function.

[0182] Based on this, in the process of evaluating the corrected prediction image from the dimension of the offset matrix, it is necessary to distort the actual offset matrix of the training image; and in the process of evaluating the corrected prediction image from the dimensions of the text region and the straight line region, it is necessary to use the text edge mask corresponding to the text region and the straight line mask corresponding to the straight line region.

[0183] Therefore, the metadata for each group in the training dataset also includes the actual offset matrix corresponding to the distorted training image, the text edge mask corresponding to the text region, and the line mask corresponding to the line region.

[0184] Based on this, a multimodal loss function can be constructed, and a multimodal loss value can be determined to characterize the difference between the corrected prediction image and the corresponding distorted training image from multiple dimensions.

[0185] In a specific example, the text edge mask and the line mask can be determined by text edge detection and line detection.

[0186] Specifically, for the actual corrected image, text edge detection can be performed. After detecting the text edges, the text region and / or edge region are binarized, and the binarized actual corrected image is dilated to obtain a text edge mask.

[0187] Furthermore, since there are both visible and hidden lines in the actual corrected image, different line detection methods can be used to perform two line detections.

[0188] The displayed straight line can be, for example, a straight line that exists and is specifically displayed in the actual corrected image; the hidden straight line can be, for example, a straight line formed by text lines or text columns in the actual corrected image, such as the horizontal straight line represented by each text line, and the vertical straight line displayed between each text line due to formatting settings such as left alignment or right alignment.

[0189] When performing line detection on actual corrected images, lines that are explicitly displayed can be detected first, and lines that are implicit can be detected, such as those that are implicit, using methods such as DBNet.

[0190] Based on this, a line mask can be obtained after line detection.

[0191] In the embodiments of this application, the depth-separable convolutional layer includes a depthwise convolutional layer and a pointwise convolutional layer.

[0192] Figure 9 This is a structural diagram of a depth-separable convolutional layer provided in an embodiment of this application.

[0193] like Figure 9 As shown, depth-separable convolutional layers include cascaded depthwise convolutional layers and pointwise convolutional layers.

[0194] based on Figure 9 The structure of the depth-separable convolutional layer is shown. Figure 10 A flowchart illustrating a depthwise separable convolution method provided in an embodiment of this application. Figure 8 As shown, the following steps are performed on the warped training images for each set of metadata: Step S1001: Perform spatial depth convolution on the distorted training image using a depth convolution layer to obtain spatial information.

[0195] After inputting each distorted training image into the correction model to be trained, the depthwise separable convolutional layer can first extract features from each distorted training image in both channel and spatial dimensions.

[0196] based on Figure 7 The structure shown can first perform spatial depth convolution on each twisted training image by a depth convolutional layer, and then perform spatial convolution on each channel of the twisted training image separately to obtain the corresponding spatial information.

[0197] Step S1002: The spatial information of each channel is convolved and mixed by the pointwise convolutional layer to generate a channel spatial feature map.

[0198] Based on the spatial information of each distorted training image determined in S1001, it can be input into a series of pointwise convolutional layers.

[0199] The kernel of the pointwise convolutional layer can be 1x1.

[0200] Based on this, pointwise convolutional layers can perform cross-channel fusion on each pixel in the distorted training image, thereby fusing the spatial information of each channel of each pixel to obtain the channel spatial feature map of the distorted training image.

[0201] Step S1003: Perform regular convolution and dilated convolution on the predicted channel spatial feature map using pyramid pooling units to obtain a multi-scale feature map.

[0202] Step S1004: The attention unit performs attention weighting on the multi-scale feature map to generate a predicted weighted feature map.

[0203] Step SS1005: Generate the prediction offset matrix based on the prediction weighted feature map.

[0204] As mentioned above, depthwise separable convolutional layers have a significantly reduced number of parameters compared to standard convolutions, enabling lightweight correction models.

[0205] In embodiments of this application, the structure of the pyramid pooling unit may specifically be, for example, a FasterASPP (Faster Atrous Spatial Pyramid Pooling) structure, which includes one or more parallel dilated convolutional layers, each of which may be further connected in parallel with a single conventional convolutional layer.

[0206] Figure 11 This is a structural diagram of a FasterASPP structure provided in an embodiment of this application.

[0207] like Figure 11 As shown, the FasterASPP structure includes a regular convolutional layer with a 3x3 kernel and two dilated convolutional layers connected in parallel with the regular convolutional layer. The kernel of each dilated convolutional layer is also the same 3x3, but the two layers have different dilation factors.

[0208] based on Figure 11 The FasterASPP structure shown is as follows: Figure 12 This is a flowchart illustrating a multi-scale feature map determination method provided in an embodiment of this application. Figure 12 As shown, it includes the following steps: Step S1201: Perform depthwise convolution and pointwise convolution on the distorted training image using a depthwise separable convolutional layer to obtain the predicted channel spatial feature map.

[0209] Step S1202: Perform regular convolution on the spatial feature map of each prediction channel by a regular convolutional layer to generate the corresponding first-scale feature map.

[0210] Combination Figure 11 As shown, the data output from the previous layer is simultaneously input into a regular convolutional layer and two dilated convolutional layers.

[0211] In this step, the channel spatial feature map determined in S1201 is as follows: Figure 11 The data output from the previous layer can be convolved using a regular convolutional layer to generate a first-scale feature map.

[0212] Among them, such as Figure 11 As shown, the receptive field of a regular convolutional layer when performing regular convolution is 3x3. The receptive field of the convolution kernel is 1.

[0213] It can be seen that the first-scale feature map represents the features collected when the convolution kernel is 3x3.

[0214] In some cases, such as Figure 11 As shown, after convolution in a regular convolutional layer, an activation function is set. This activation function is used to perform non-linear processing on the convolution result of the regular convolutional layer, thereby enhancing the convergence ability of the correction model. It can be, for example, the ReLU function.

[0215] Furthermore, the result after processing with the ReLU function can be used as the first-scale feature map.

[0216] Step S1203: Each dilated convolutional layer performs dilated convolution on the spatial feature map of each prediction channel with different receptive fields, thereby generating multiple second-scale feature maps with different receptive fields.

[0217] In this step, based on the channel spatial feature map determined in S1201, the channel spatial feature map can be dilated by each dilated convolutional layer, and a second-scale feature map can be generated by each dilated convolutional layer.

[0218] Among them, such as Figure 11 As shown, the two dilated convolutional layers, each with a 3x3 kernel, have different dilation factors: one dilated convolutional layer has a dilation factor of 2, and the other dilated convolutional layer has a dilation factor of 4.

[0219] Based on this, when performing dilated convolution, dilated convolution with an inflation factor of 2 will be based on a 3x3 convolution kernel, with dilated rows and columns inserted between the rows and columns of its acquisition window. The number of dilated rows and columns corresponds to the inflation factor, thereby expanding the area actually covered by the acquisition window, i.e., the receptive field, to be equivalent to the receptive field corresponding to a 5x5 convolution kernel, achieving the effect of expanding the receptive field while keeping the convolution kernel unchanged.

[0220] Meanwhile, the dilated convolution with a dilation factor of 4 is also based on a 3x3 convolution kernel. Dilated rows and columns are inserted between the rows and columns of its acquisition window. The number of dilated rows and columns corresponds to the dilation factor, thereby expanding the area actually covered by the acquisition window, i.e., the receptive field, to be equivalent to the receptive field corresponding to a 9x9 convolution kernel. This achieves the effect of expanding the receptive field while keeping the convolution kernel unchanged.

[0221] It can be seen that each second-scale feature map represents the features acquired under the corresponding receptive field conditions.

[0222] In some cases, such as Figure 9 As shown, after each dilated convolutional layer is convolved, a corresponding activation function is set for each dilated convolutional layer. This activation function is used to perform nonlinear processing on the convolution result of the corresponding dilated convolutional layer, thereby enhancing the convergence ability of the correction model. It can be, for example, the ReLU function.

[0223] Furthermore, the result of processing each corresponding ReLU function can be used as a second-scale feature map.

[0224] Step S1204: Concatenate the first-scale feature map and the corresponding second-scale feature maps corresponding to each predicted channel spatial feature map to obtain the corresponding multi-scale feature map.

[0225] Based on the first-scale feature map determined in S1202 and the various second-scale feature maps determined in S1203, they can be stitched together to obtain a multi-scale feature map.

[0226] In this step, such as Figure 11As shown, after the regular convolution and each dilated convolution, a concatenation layer is set in series. Since the first-scale feature map and each second-scale feature map focus on different features, the first-scale feature map and each second-scale feature map can be simultaneously input into the concatenation layer. The concatenation layer is used to concatenate the first-scale feature map and each second-scale feature map to obtain a multi-scale feature map.

[0227] In some cases, such as Figure 11 As shown, a convolutional layer with a 1x1 kernel can be set after the stitching layer to compress the high-dimensional multi-scale features, thereby reducing the amount of computation and performing non-linear processing between channels. This can enhance the representation of key information and reduce noise generated during the stitching process.

[0228] Step S1205: The attention unit performs attention weighting on the multi-scale feature map to generate a predicted weighted feature map.

[0229] Step S1206: Generate a prediction offset matrix based on the prediction weighted feature map.

[0230] As described above, the multi-scale feature fusion mechanism significantly improves the correction model's ability to represent distortions. Based on an architecture design that combines conventional convolutions and dilated convolutions with varying dilation rates, the conventional convolutional layers focus on capturing detailed features within the 3×3 local receptive field, while the dilated convolutional layers with dilation factors of 2 and 4 extract mid-to-long-range geometric features with equivalent receptive fields of 5×5 and 9×9, respectively. This achieves full-scale feature coverage from local detailed distortions to macroscopic global distortions. By organically fusing feature maps of different scales through a stitching layer and performing channel compression and feature reorganization, the correction model can significantly improve the geometric accuracy of offset field prediction while maintaining lightweight computation.

[0231] Figure 13 This is a framework diagram of another correction model training method provided in an embodiment of this application. Figure 13 As shown, the correction model to be trained includes not only the aforementioned depthwise separable convolutional layers, pyramid pooling units, and attention units, but also multiple convolutional layers with 3x3 kernels.

[0232] After inputting the correction training images into the correction model to be trained, the correction training images are first convolved using a 3x3 convolutional layer. The convolution result is then input into a depthwise separable convolutional layer for processing. The processing result is then input into a pyramid pooling unit of a FasterASPP structure for further processing. Finally, the processing result is input into an attention unit for weighted summation to obtain a weighted feature map.

[0233] like Figure 13As shown, the depthwise separable convolutional layer, the pyramid pooling unit of the FasterASPP structure, and the attention unit are connected in series. After the attention unit obtains the weighted feature map, if this attention weighting is the first attention weighting process, the weighted feature map can be input into the depthwise separable convolutional layer again. That is, the processing of the depthwise separable convolutional layer, the pyramid pooling unit of the FasterASPP structure, and the SAM unit is performed twice, so as to obtain a more accurate weighted feature map.

[0234] Furthermore, if this attention weighting is the second attention weighting process, the weighted feature map obtained after processing the depth-separable convolutional layer, the pyramid pooling unit of the FasterASPP structure, and the SAM unit twice can be input into another convolutional layer with a 3x3 kernel. The output of this convolutional layer with a 3x3 kernel is concatenated with a third convolutional layer with a 3x3 kernel. After the convolution operation of two convolutional layers with a 3x3 kernel, the prediction offset matrix can be obtained.

[0235] like Figure 13 As shown, the corrected prediction image is obtained by inputting the predicted offset matrix into the pre-set remap function to map the corrected training image.

[0236] Furthermore, the difference between the predicted offset matrix and the actual offset matrix can be determined using the offset loss function, that is... Figure 13 The offset loss in the image; using the text region loss function, the difference between the text regions in the predicted image and the actual text regions in the corrected image can be determined, that is... Figure 13 The text region loss function is used to determine the difference between the line region in the predicted image and the actual line region in the corrected image. Figure 13 The straight-line region loss in the image; by using each first control point in the corrected prediction image, the control point loss of the corrected prediction image with respect to the straight-line dimension can be determined.

[0237] Furthermore, by weighting the offset loss function, text region loss function, line region loss function, and control point loss function, a multimodal loss function can be used between the predicted image and the actual corrected image, and the multimodal loss can be determined.

[0238] Furthermore, the parameters of the correction model to be trained can be adjusted based on this multimodal loss.

[0239] Based on the same inventive concept, and corresponding to the methods of any of the above embodiments, the embodiments of this application also provide an image correction device.

[0240] Figure 14This is a structural block diagram of an image correction device provided in an embodiment of this application. The device is configured to execute the image correction method provided in the above embodiment, and has corresponding functional modules and beneficial effects for executing the method. For example... Figure 14 As shown, the device includes: an acquisition module 1401 and a prediction module 1402; The acquisition module 1401 is configured to acquire the image to be corrected. The prediction module 1402 is configured to input the image to be corrected into a trained correction model, and process the image to be corrected using the trained correction model to obtain a corrected image; wherein the correction model is trained using a control point loss function.

[0241] In some optional implementations, the training module 1402 is further specifically configured as follows: The correction model was obtained by training it in the following way: Obtain a training dataset, which includes multiple sets of metadata, each set of metadata including multiple warped training images; Input a predetermined number of metadata sets into the correction model to be trained, and output the prediction offset matrix corresponding to the distorted training image for each set of metadata sets; The pixels in each distorted training image are mapped according to the predicted offset matrix to obtain the corresponding corrected prediction image; the control point loss corresponding to each corrected prediction image is determined according to the control point loss function, and each control point loss represents the degree of distortion of the straight line in the corresponding corrected prediction image. The correction model to be trained is trained based on the control point loss to obtain the completed correction model.

[0242] In some optional implementations, the training module 1402 is further specifically configured as follows: Identify at least one first straight line in each corrected prediction image, and identify at least two adjacent vector segments in each first straight line; Determine the cosine similarity between any two adjacent vectors on the first straight line in each corrected prediction image; The average difference between each cosine similarity and the preset value is determined as the corresponding control point loss.

[0243] In some optional implementations, each set of metadata includes the actual corrected image corresponding to each distortion training image, each distortion training image including a first straight line region; the corresponding actual corrected image includes a second straight line region corresponding to the first straight line region; each set of metadata also includes a straight line mask corresponding to the second straight line region in the actual corrected image; Accordingly, the training module 1402 is further configured as follows: The first straight line region in the corresponding corrected prediction image is determined based on each straight line mask; Determine each of the first straight lines in the first straight line region.

[0244] In some optional implementations, the training module 1402 is further specifically configured as follows: Obtain multiple first control points in each first straight line; Among the three consecutive first control points in the first straight line, the first vector in the first straight line is determined by the middle first control point and the previous first control point. The first control point in the middle and the next first control point are used to determine the next vector in the first straight line that is adjacent to the previous vector.

[0245] In some optional implementations, each set of metadata also includes the actual offset matrix corresponding to the warped training images; Accordingly, the training module 1402 is further configured as follows: Obtain multiple second control points in the actual corrected image corresponding to each distortion training image; The actual offset matrix is ​​used to map each second control point in each actual corrected image to the corresponding twisted training image, thereby obtaining multiple twisted image control points in each twisted training image. The predicted offset matrix is ​​used to map each twisted image control point in each twisted training image to the corresponding corrected prediction image, thereby obtaining multiple first control points in each first straight line in the corresponding corrected prediction image.

[0246] In some alternative implementations, the second control point includes a display line control point; Accordingly, the training module 1402 is further configured as follows: Obtain the edge of the second straight line region in each actual corrected image; Determine the intersection between the line mask corresponding to the second straight line region in each actual corrected image and the edge of the second straight line region, and define the intersection as each display line in the corresponding actual corrected image; Each displayed straight line is divided into multiple equal parts, and each division position is determined as the display line control point of the corresponding displayed straight line.

[0247] In some optional implementations, each distortion training image includes a first text region; the corresponding actual correction image includes a second text region corresponding to the first text region; the second control point further includes implicit straight line control points; Accordingly, the training module 1402 is further configured as follows: Determine the center position of each character in each second text area, and use each center position as a hidden line control point.

[0248] In some optional implementations, each set of metadata also includes the actual corrected image corresponding to each distorted training image; Accordingly, the training module 1402 is further configured as follows: Before training the correction model to be trained based on the control point loss to obtain the trained correction model, the following is performed: Each predicted offset matrix and its corresponding actual offset matrix are input into the offset loss function to obtain the corresponding offset loss; Determine the total variational denoising value corresponding to each prediction offset matrix; The image loss between each corrected prediction image and the corresponding actual corrected image is determined based on the image loss function. The step of training the correction model to be trained based on the control point loss to obtain the trained correction model includes: For each corrected prediction image, the corresponding control point loss, the corresponding offset loss, the corresponding total variation denoising value, and the corresponding image loss are weighted to obtain the corresponding multimodal loss. The correction model to be trained is trained based on the multimodal loss to obtain the trained correction model.

[0249] For ease of description, the above apparatus is described in terms of its functions, divided into various modules. Of course, in implementing the embodiments of this application, the functions of each module can be implemented in one or more software and / or hardware.

[0250] The apparatus of the above embodiments is used to implement the corresponding image correction method in any of the foregoing embodiments, and has the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0251] Based on the same inventive concept, corresponding to the methods of any of the above embodiments, the embodiments of this application also provide an image correction device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements the image correction method of any of the above embodiments.

[0252] Figure 15 This is a schematic diagram of the structure of an image correction device provided in an embodiment of this application, as shown below. Figure 15 As shown, the device includes a processor 1501, a memory 1502, an input device 1503, and an output device 1504; the number of processors 1501 in the device can be one or more. Figure 15Taking a processor 1501 as an example; the processor 1501, memory 1502, input device 1503, and output device 1504 in the device can be connected via a bus or other means. Figure 15 Taking a bus connection as an example, the memory 1502, as a computer-readable storage medium, can be configured to store software programs, computer-executable programs, and modules, such as program instructions / modules for implementing the image correction method in the embodiments of this application. The processor 1501 executes various functional applications and data processing of the device by running the software programs, instructions, and modules stored in the memory 1502, thereby implementing the aforementioned image correction method. The input device 1503 can be configured to receive input digital or character information and generate key signal inputs related to user settings and function control of the device. The output device 1504 may include a display screen or other display device.

[0253] The apparatus of the above embodiments is used to implement the corresponding image correction method in any of the foregoing embodiments, and has the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0254] Based on the same inventive concept, corresponding to the methods of any of the above embodiments, this application also provides a non-volatile storage medium containing computer-executable instructions. When executed by a computer processor, the computer-executable instructions are configured to perform an image correction method described in the above embodiments. This method includes: training a correction model to be trained using a multimodal loss function, combining the synergistic effects of offset loss sub-functions, image loss sub-functions, and control point loss sub-functions to constrain the model training process from multiple dimensions, effectively improving the correction performance of the trained model. Compared to models trained using traditional single-dimensional loss functions, this method allows the correction model to fully focus on key features related to image distortion during the learning process, significantly enhancing the model's adaptability to various distorted images and ensuring the accuracy and stability of the correction results.

[0255] It is worth noting that in the above-described embodiments of the image correction device, the various units and modules included are only divided according to functional logic, but are not limited to the above division, as long as the corresponding functions can be achieved; in addition, the specific names of each functional unit are only for easy differentiation and are not configured to limit the protection scope of the embodiments of this application.

[0256] In some possible implementations, various aspects of the methods provided in this application can also be implemented as a program product, which includes program code. When the program product is run on a computer device, the program code is configured to cause the computer device to perform the steps of the methods according to the various exemplary embodiments of this application described above. For example, the computer device can perform the image correction method described in the embodiments of this application. The program product can be implemented using any combination of one or more readable media, and has the beneficial effects of the corresponding method embodiments, which will not be elaborated further here.

Claims

1. An image correction method, characterized in that, include: Obtain the image to be corrected; The image to be corrected is input into the trained correction model, and the trained correction model processes the image to obtain the corrected image. The correction model is obtained by training a control point loss function.

2. The image correction method according to claim 1, characterized in that, The correction model was trained in the following manner: Obtain a training dataset, which includes multiple sets of metadata, each set of metadata including multiple warped training images; Input a predetermined number of metadata sets into the correction model to be trained, and output the prediction offset matrix corresponding to the distorted training image for each set of metadata sets; The pixels in each distorted training image are mapped according to the predicted offset matrix to obtain the corresponding corrected prediction image; the control point loss corresponding to each corrected prediction image is determined according to the control point loss function, and each control point loss represents the degree of distortion of the straight line in the corresponding corrected prediction image. The correction model to be trained is trained based on the control point loss to obtain the completed correction model.

3. The image correction method according to claim 2, characterized in that, The step of determining the control point loss corresponding to each corrected prediction image based on the control point loss function includes: Identify at least one first straight line in each corrected prediction image, and identify at least two adjacent vector segments in each first straight line; Determine the cosine similarity between any two adjacent vectors on the first straight line in each corrected prediction image; The average difference between each cosine similarity and the preset value is determined as the corresponding control point loss.

4. The image correction method according to claim 3, characterized in that, Each set of metadata includes the actual corrected image corresponding to each distortion training image. Each distortion training image includes a first straight line region; the corresponding actual corrected image includes a second straight line region corresponding to the first straight line region; each set of metadata also includes a straight line mask corresponding to the second straight line region in the actual corrected image. Determining at least one first straight line in each corrected prediction image includes: The first straight line region in the corresponding corrected prediction image is determined based on each straight line mask; Determine each of the first straight lines in the first straight line region.

5. The image correction method according to claim 3, characterized in that, Determining at least two adjacent vector segments in each first straight line includes: Obtain multiple first control points in each first straight line; Among the three consecutive first control points in the first straight line, the first vector in the first straight line is determined by the middle first control point and the previous first control point. The first control point in the middle and the next first control point are used to determine the next vector in the first straight line that is adjacent to the previous vector.

6. The image correction method according to claim 5, characterized in that, Each set of metadata also includes the actual offset matrix corresponding to the distorted training image; The step of obtaining multiple first control points in each first straight line includes: Obtain multiple second control points in the actual corrected image corresponding to each distortion training image; The actual offset matrix is ​​used to map each second control point in each actual corrected image to the corresponding twisted training image, thereby obtaining multiple twisted image control points in each twisted training image. The predicted offset matrix is ​​used to map each twisted image control point in each twisted training image to the corresponding corrected prediction image, thereby obtaining multiple first control points in each first straight line in the corresponding corrected prediction image.

7. The image correction method according to claim 6, characterized in that, The second control point includes a display line control point; The step of obtaining multiple second control points in the actual corrected image corresponding to each distortion training image includes: Obtain the edge of the second straight line region in each actual corrected image; Determine the intersection between the line mask corresponding to the second straight line region in each actual corrected image and the edge of the second straight line region, and define the intersection as each display line in the corresponding actual corrected image; Each displayed straight line is divided into multiple equal parts, and each division position is determined as the display line control point of the corresponding displayed straight line.

8. The image correction method according to claim 7, characterized in that, Each distortion training image includes a first text region; the corresponding actual correction image includes a second text region corresponding to the first text region; the second control point also includes a hidden line control point; The method further includes: Determine the center position of each character in each second text area, and use each center position as a hidden line control point.

9. The image correction method according to claim 2, characterized in that, Each set of metadata also includes the actual corrected image corresponding to each distorted training image; Before training the correction model to be trained based on the control point loss to obtain the trained correction model, the method further includes: Each predicted offset matrix and its corresponding actual offset matrix are input into the offset loss function to obtain the corresponding offset loss; Determine the total variation denoising value corresponding to each prediction offset matrix; The image loss between each corrected prediction image and the corresponding actual corrected image is determined based on the image loss function. The step of training the correction model to be trained based on the control point loss to obtain the trained correction model includes: For each corrected prediction image, the corresponding control point loss, the corresponding offset loss, the corresponding total variation denoising value, and the corresponding image loss are weighted to obtain the corresponding multimodal loss. The correction model to be trained is trained using the multimodal loss to obtain the trained correction model.

10. An image correction device, characterized in that, include: Acquisition module and prediction module; The acquisition module is configured to acquire the image to be corrected; The prediction module is configured to input the image to be corrected into a trained correction model, and process the image to be corrected using the trained correction model to obtain a corrected image; wherein the correction model is trained using a control point loss function.

Images