Image correction method, image correction system, electronic equipment and vehicle

Abstract

The invention relates to an image correction method, an image correction system, electronic equipment and a vehicle, and the method comprises the steps: inputting a to-be-corrected first distortion image into an initial image correction network for training, obtaining an intermediate image correction network, correcting the first distortion image through the intermediate image correction network, and obtaining a preliminary correction image; according to the method, a first distortion image and a preliminary correction image are acquired, then multi-scale loss training is performed on an intermediate image correction network according to the first distortion image and the preliminary correction image to acquire a target image correction network, and finally a to-be-corrected second distortion image is inputted to the target image correction network for correction to acquire a target correction image, so that the accuracy and efficiency of distortion image correction are improved.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to an image correction method, an image correction system, an electronic device, a vehicle, a computer-readable storage medium, and a computer program product. Background Art

[0002] Nowadays, image distortion correction mainly adopts regression-based and generation-based methods. The regression-based method refers to the correction by predicting nonlinear model parameters through convolutional neural network (CNN). The generation-based method refers to directly generating the correction image through generative adversarial network (GAN) to achieve label-free, single-stage correction operation.

[0003] However, the regression-based approach is limited by discrete parameters, has poor effects on complex image correction, and requires a large amount of label data such as edge parameters and distortion parameters for model training, which increases the difficulty of training. The generation-based approach has a heavy network computation burden, which can easily lead to blurred content and incomplete structure of the corrected image. Summary of the invention

[0004] The embodiment of the present application provides an image correction method, which can improve the efficiency and accuracy of image correction to at least partially solve the above-mentioned technical problems.

[0005] In order to achieve the above object, according to a first aspect of the present application, an image correction method is provided, the method comprising:

[0006] Inputting the first distorted image to be corrected into an initial image correction network for training to obtain an intermediate image correction network, and correcting the first distorted image through the intermediate image correction network to obtain a preliminary corrected image;

[0007] According to the first distorted image and the preliminary corrected image, performing multi-scale loss training on the intermediate image correction network to obtain a target image correction network;

[0008] The second distorted image to be corrected is input into the target image correction network for correction to obtain a target corrected image.

[0009] Optionally, the step of inputting the first distorted image to be corrected into an initial image correction network for training to obtain an intermediate image correction network comprises:

[0010] Synchronously inputting the first distorted image into a flow estimation module and a distortion correction module to obtain appearance flow information of the first distorted image output by the flow estimation module and a first corrected image of the first distorted image output by the distortion correction module;

[0011] The flow estimation module and the distortion correction module are synchronously trained according to the appearance flow information, the first corrected image and the first distorted image to obtain the intermediate image correction network.

[0012] Optionally, the synchronously training the flow estimation module and the distortion correction module according to the appearance flow information, the first corrected image and the first distorted image to obtain the intermediate image correction network includes:

[0013] Inputting the appearance flow information into the distortion correction module, and iteratively training the distortion correction module by combining the appearance flow information and the first distorted image;

[0014] Inputting the first corrected image into the flow estimation module, and iteratively training the flow estimation module in combination with the first distorted image and the first corrected image;

[0015] When the flow estimation module and the distortion correction module meet the training completion condition, the training is terminated to obtain the intermediate image correction network.

[0016] Optionally, the method further comprises:

[0017] The first distorted image is input into the flow estimation module, image features of the first distorted image are extracted through the convolutional neural layer of the flow estimation module, and appearance flow information of the first distorted image is generated based on the image features.

[0018] Optionally, the step of inputting the appearance flow information into the distortion correction module, and iteratively training the distortion correction module in combination with the appearance flow information and the first distorted image comprises:

[0019] According to the first corrected image and the undistorted real image corresponding to the first corrected image, constructing a reconstruction loss for minimizing the difference between the first corrected image and the real image;

[0020] Constructing an adversarial loss for enhancing the overall authenticity of the first corrected image according to a second corrected image generated by the generator of the distortion correction module based on the first distorted image and a discrimination result of the second corrected image and the real image by the discriminator of the distortion correction module;

[0021] Constructing an enhancement loss for enriching local details of the first corrected image according to a first feature map and a first linear matrix of the first corrected image, and a second feature map and a second linear matrix of the real image;

[0022] The distortion correction module is iteratively trained according to the first corrected image and the real image until the reconstruction loss, the adversarial loss and the enhancement loss converge, and then the training is terminated to obtain a trained distortion correction module.

[0023] Optionally, constructing an enhancement loss for enriching local details of the first corrected image includes:

[0024] constructing a content loss for minimizing an image content difference between the first corrected image and the real image according to a difference between the first feature map and the second feature map;

[0025] constructing a style loss for minimizing the image style difference between the first corrected image and the real image according to the difference between the first linear matrix and the second linear matrix;

[0026] The enhancement loss is constructed according to the sum of the content loss and the style loss.

[0027] Optionally, performing multi-scale loss training on the intermediate image correction network according to the first distorted image and the preliminary corrected image to obtain a target image correction network includes:

[0028] constructing a multi-scale loss for minimizing distortion defects of features at different scales according to the first distorted image and the preliminary corrected image;

[0029] Constructing an overall loss of the target image correction network according to the reconstruction loss, the adversarial loss, the enhancement loss and the multi-scale loss;

[0030] The initial image correction network is iteratively trained until the overall loss converges and the training is terminated to obtain the target image correction network.

[0031] Optionally, constructing a multi-scale loss for minimizing distortion defects of features at different scales comprises:

[0032] Performing multi-scale feature extraction on the first distorted image to generate first feature maps of different scales;

[0033] Acquire original features of each of the first feature maps and corrected features of the preliminary corrected image;

[0034] Performing feature concatenation on the original feature and the corrected feature of the corresponding scale to obtain a concatenated feature, and performing a convolution operation on the concatenated feature to generate a corrected second feature map;

[0035] Sampling the second feature map to obtain a multi-scale second feature map;

[0036] The multi-scale loss is constructed based on the difference between the multi-scale second feature maps and the real image of the corresponding scale.

[0037] Optionally, constructing the overall loss of the target image correction network according to the reconstruction loss, the adversarial loss, the enhancement loss and the multi-scale loss includes:

[0038] Acquire a first hyperparameter for adjusting the reconstruction loss and a second hyperparameter for adjusting the multi-scale loss;

[0039] Adjusting the reconstruction loss according to the first hyperparameter to obtain an adjusted reconstruction loss, and adjusting the multi-scale loss according to the second hyperparameter to obtain an adjusted multi-scale loss;

[0040] The overall loss is constructed according to the sum of the adjusted reconstruction loss, the adjusted multi-scale loss, the adversarial loss and the enhancement loss.

[0041] According to a second aspect of the present application, an image correction system is also provided for executing the method as described above, the system comprising an image acquisition unit, a communication unit, a computing unit and an image display unit, wherein:

[0042] The image acquisition unit is used to acquire the distorted image to be corrected;

[0043] The communication unit is used to transmit the distorted image to the computing unit;

[0044] The computing unit is used to correct the distorted image to obtain a corrected image, and send the corrected image to the image display unit;

[0045] The image display unit is used to receive and display the corrected image.

[0046] According to a third aspect of the present application, a computer-readable storage medium is further provided, wherein program instructions are stored in the computer-readable storage medium, and the program instructions implement the method described above when executed.

[0047] According to a fourth aspect of the present application, a computer program product is also provided, wherein the computer program product comprises program instructions, and when the program instructions are executed by a processor, the method as described above is implemented.

[0048] According to a fifth aspect of the present application, an electronic device is also provided for executing the method as described above.

[0049] According to a sixth aspect of the present application, a vehicle is also provided, comprising the electronic device as described above.

[0050] In the embodiment of the present application, an intermediate image correction network is obtained by training the initial image correction network, and the distorted image is pre-corrected by the intermediate image correction network to obtain a pre-corrected image for subsequent training, which is equivalent to embedding a feature-level correction layer in the jump connection, thereby effectively reducing the computational burden of the decoder and improving the operational efficiency and correction effect of the correction network. In addition, the target image correction network is obtained by multi-scale training the intermediate correction network with distorted images and pre-corrected images, so that the distorted image can be optimized layer by layer, the consistency of the content and structure of the corrected image can be ensured, the blur and incomplete structure of the corrected image can be reduced, and the number of labels required for training can be reduced, thereby reducing the difficulty of model training and the cost of data annotation in the image correction process.

[0051] Other features and advantages of the present application will be described in detail in the subsequent detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application, and those skilled in the art can obtain other drawings based on these drawings without creative work.

[0053] In order to more completely understand the present application and its beneficial effects, the following description will be given in conjunction with the accompanying drawings, wherein the same figure numbers represent the same parts in the following description.

[0054] Figure 1 is a flowchart of an image correction method provided in an exemplary embodiment of the present application;

[0055] Figure 2 is an architecture diagram of an image correction network provided in an exemplary embodiment of the present application;

[0056] Figure 3 is a block diagram of an image correction system provided in an exemplary embodiment of the present application;

[0057] Figure 4 is a block diagram of an electronic device provided in an exemplary embodiment of the present application;

[0058] Figure 5 is a block diagram of a vehicle provided in an exemplary embodiment of the present application. DETAILED DESCRIPTION

[0059] The technical solutions in the embodiments of the present application will be described clearly and completely below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present application.

[0060] Nowadays, the correction of distorted images is mainly carried out in the following two ways. One is based on regression, which usually uses convolutional neural networks to predict nonlinear model parameters to achieve distorted image correction. This method requires parameterizing the image through a regression model, but it is limited by finite discrete parameters. When faced with complex distorted images, the expression ability of its parameterized model is poor and it is difficult to adapt to various complex scenes. In addition, the regression-based method requires a large amount of labeled data, such as edge labels and distortion parameter labels, in order to effectively train the model, but the large number of label requirements not only increases the cost of data annotation, but also significantly increases the complexity of model training.

[0061] The other type is based on generation, such as using generative adversarial network technology to correct distorted images, and achieving the effect of unlabeled training and single-stage correction by generating corrected images. For example, a clearer corrected image can be generated by DR-GAN (Disentangled Representation learning-Generative Adversarial Network). However, the network calculation burden of this method is large, resulting in blurred image content in most network environments, and the structure of the corrected image is not complete.

[0062] Therefore, existing distorted image correction methods have limited effects when processing complex distorted images, and there is still a need for a technical solution that can efficiently and accurately correct distorted images under label-free conditions.

[0063] Based on the above problems, the present application provides an image correction method, which is described below in conjunction with specific embodiments.

[0064] Figure 1 is a schematic diagram of an image correction method provided in an embodiment of the present application, such as Figure 1 As shown, the method may include the following steps S101-S103, which are specifically described as follows.

[0065] Step S101: input the first distorted image to be corrected into the initial image correction network for training to obtain an intermediate image correction network, and correct the first distorted image through the intermediate image correction network to obtain a preliminary corrected image.

[0066] In the scenario of the present application, the distorted image may be an image with obvious distortion characteristics produced by shooting with a fisheye lens. Due to the ultra-wide-angle viewing angle of the fisheye lens, the image will be obviously distorted, especially in the edge area, showing phenomena such as stretching and deformation. This distortion makes the objects and scenes in the image not conform to the natural proportional relationship, resulting in the destruction of the integrity of the image structure and the formation of a distorted image.

[0067] In some embodiments, the acquired distorted image may be scaled to obtain a first distorted image of a standard size, for example, the standard size may be a pixel size of 256 × 256. By scaling the distorted image to a fixed size, the structural design of the image correction network can be simplified, and the features of the distorted image can be trained and processed at a uniform scale.

[0068] In the specific implementation, Figure 2 As shown, the first distorted image to be corrected can first be input into an initial image correction network for training. The initial image correction network can be an untrained image correction network or model, and the initial image correction network can be trained based on the input first distorted image to learn the basic features and rules of distorted image correction, and identify and extract common distortion features in distorted images, such as edge stretching, arc distortion, etc.

[0069] It can be understood that the model parameters of the initial image correction network can be continuously optimized during the training process, gradually establishing an understanding of the distortion characteristics, and can initially generate relatively accurate correction results. After the training is completed, an intermediate image correction network is obtained. Compared with the initial image correction network, the intermediate image correction network has better distortion correction capabilities, but the correction effect may not be perfect enough.

[0070] In some embodiments, the first distorted image or other distorted images used for training may be input into an intermediate image correction network, and the intermediate image network may be used to perform distortion correction processing on the input image to obtain a preliminary corrected image.

[0071] Through the above method, the present application processes the input distorted image and outputs a preliminary corrected image through the distortion correction capability mastered by the intermediate image correction network, which corrects the distortion of the distorted image to a certain extent. Therefore, it is equivalent to embedding a feature-level correction layer in the jump connection of the image correction network to perform pre-correction processing on the distorted image, and using the features of the pre-corrected preliminary corrected image as the input for subsequent multi-scale supervised training, which can improve the decoder of the image correction network's utilization of distortion-free information, alleviate the distortion diffusion problem, and improve the accuracy and efficiency of image correction.

[0072] In some embodiments, step S101 may include the following steps:

[0073] First, the first distorted image is synchronously input into a flow estimation module and a distortion correction module to obtain appearance flow information of the first distorted image output by the flow estimation module and a first corrected image of the first distorted image output by the distortion correction module;

[0074] Next, according to the appearance flow information, the first corrected image and the first distorted image, the flow estimation module and the distortion correction module are synchronously trained to obtain an intermediate image correction network.

[0075] In some embodiments, the first distorted image may be input into a flow estimation module, image features of the first distorted image may be extracted through a convolutional neural layer of the flow estimation module, and appearance flow information of the first distorted image may be generated based on the image features.

[0076] The flow estimation module can generate appearance flow information based on the input first distorted image, that is, describing the displacement information of each pixel in the first distorted image, such as the motion vector of each pixel. The appearance flow information is a kind of optical flow data, which can reflect the motion pattern of the pixels of the distorted image on the image plane and indicate the direction and magnitude of the image distortion.

[0077] In a specific implementation, after the first distorted image is passed to the flow estimation module, the flow estimation module can extract features of the input first distorted image through the convolutional neural network layer. The convolutional neural layer can extract features of different scales such as edges and shapes in the image layer by layer, and calculate the optical flow information of the first distorted image based on the image features extracted by the convolutional layer, that is, the displacement vector field of the pixel movement in the image. By analyzing the optical flow in the distorted image, the position, direction and intensity of the image distortion can be identified to obtain the distortion information of the first distorted image. The distortion information can be subsequently passed to the distortion correction module, so that the distortion correction module can use the above distortion information to restore the real structure of the image during the image correction process, and finally generate a distortion-free image.

[0078] It should be noted that the flow estimation module of the present application is trained using an unsupervised learning method. The unsupervised flow estimation module can estimate the structure of the distorted image, which helps to capture more structural information of the distorted image and provide it to the distortion correction module, so that the distortion correction module can improve its understanding and correction capabilities of the distorted structure of the distorted image.

[0079] In some embodiments, after receiving the distorted image, the distortion correction module can use the internal preliminary correction network to perform preliminary correction on the image and output the corresponding first corrected image. The first corrected image retains some structural information of the real image, but the correction effect is not accurate enough, and it can be used as a training sample for training the preliminary correction network.

[0080] In some embodiments, the obtained appearance flow information, the first corrected image, and the first distorted image may be used to perform synchronous training of the flow estimation module and the distortion correction module to further optimize the image correction effect.

[0081] Specifically, the appearance flow information output by the flow estimation module can provide pixel displacement information of the distorted image, so that the image correction network can understand and estimate the distortion of each pixel, which helps to restore the original structure of the undistorted image during the image correction process. The first corrected image can be used as a preliminary correction result of a pre-correction process. The first corrected image provides a preliminary structural recovery reference for subsequent training, guiding the initial correction network to gradually improve the details in further correction. The first distorted image and its original image features can be used for comparison with the corrected image. Through the joint optimization of the flow estimation module and the correction module, an image result that is closer to undistorted is generated.

[0082] like Figure 2 As shown, in the image correction network, the appearance flow information output by the flow estimation module can be provided to the distortion correction module. Correspondingly, the first corrected image generated by the distortion correction module can also be provided to the flow estimation module. The two modules share data and complement each other during synchronous training. This complementary method can avoid abruptness and discontinuity in the correction process, making the correction transition of the image smoother and more natural, and the corrected details and contours are gradually improved. The final corrected image is more fluent and smooth, which can reduce image edge distortion and visual discomfort caused by jump correction.

[0083] By synchronously training the flow estimation module and the distortion correction module, the parameters of the two modules are gradually adjusted so that the appearance flow information output by the flow estimation module and the image correction effect of the distortion correction module are mutually adapted, further improving the correction effect. The intermediate image correction network obtained by synchronous training has higher image correction accuracy based on the preliminary correction network, can handle image distortion more accurately, and provides a more accurate basis for the subsequent generation of the final target image correction network.

[0084] In some embodiments, the intermediate image correction network may be trained in the following manner:

[0085] First, the appearance flow information is input into the distortion correction module, and the distortion correction module is iteratively trained by combining the appearance flow information and the first distorted image;

[0086] Next, the first corrected image is input into a flow estimation module, and the flow estimation module is iteratively trained by combining the first distorted image and the first corrected image;

[0087] Finally, when the flow estimation module and the distortion correction module meet the training completion conditions, the training is terminated to obtain the intermediate image correction network.

[0088] In a specific implementation, the appearance flow information generated by the flow estimation module can be input into the distortion correction module, and the distortion correction module can combine the appearance flow information with the first distorted image as input data for iterative training. Guided by the appearance flow information, the distortion correction module can gradually adjust parameters to more accurately correct the distortion features in the image. During the training process, the distortion correction module can generate a first corrected image guided by the appearance flow information.

[0089] In some embodiments, the first corrected image output by the distortion correction module can be input into the flow estimation module to further improve the accuracy of the flow estimation module in capturing distortion information. The flow estimation module is iteratively trained in combination with the first corrected image and the first distorted image. By comparing the first distorted image and the first corrected image, the flow estimation module can better identify the direction and magnitude of image distortion and adjust the generated appearance flow to more accurately describe the distortion in the image. This process enables the flow estimation module to estimate the distortion information more accurately, thereby providing more effective distortion guidance information for the distortion correction module.

[0090] In the above manner, the flow estimation module and the distortion correction module influence each other and are optimized alternately. Each module depends on the output of another module during training, and continuously adjusts its correction strategy to form complementarity in data and training. In a progressive and complementary training method, the distortion correction module and the flow estimation module gradually adapt to the distortion characteristics and correction requirements of the image, so that the correction effect is gradually improved in each iteration, further avoiding the problem of image detail loss or unevenness during the correction process.

[0091] During the iteration process, the outputs of the flow estimation module and the distortion correction module gradually stabilize. When the preset training completion conditions, such as the error threshold or the number of iterations, are met, the training process ends, and the resulting combination of the flow estimation module and the distortion correction module is called the intermediate image correction network. This intermediate image correction network has a high correction accuracy and can be used for further optimization or as part of the final correction network.

[0092] The present application realizes progressive distortion correction by mutually transmitting the appearance flow information and the first correction image and training the distortion correction module and the flow estimation module alternately and iteratively. The flow estimation module provides more accurate distortion information to the distortion correction module in each iteration, and the correction module outputs an updated correction image as a reference. Finally, by satisfying the training completion condition, an intermediate image correction network with high correction accuracy is formed.

[0093] In some embodiments, the distortion correction module is trained by:

[0094] Firstly, according to the first corrected image and the distortion-free real image corresponding to the first corrected image, a reconstruction loss for minimizing the difference between the first corrected image and the real image is constructed;

[0095] Next, according to the second corrected image generated by the generator of the distortion correction module based on the first distorted image, and the discrimination result between the second corrected image and the real image by the discriminator of the distortion correction module, an adversarial loss for enhancing the overall authenticity of the first corrected image is constructed;

[0096] Then, according to the first feature map and the first linear matrix of the first corrected image, and the second feature map and the second linear matrix of the real image, an enhancement loss for enriching the local details of the first corrected image is constructed;

[0097] Finally, the distortion correction module is iteratively trained according to the first corrected image and the real image until the reconstruction loss, adversarial loss, and enhancement loss converge and the training is terminated to obtain a trained distortion correction module.

[0098] In the specific implementation, the distortion correction module can optimize the corrected image by constructing reconstruction loss, adversarial loss and enhancement loss, making it closer to the real image in terms of structural similarity, realism and detail richness.

[0099] The reconstruction loss can be used to ensure that the corrected image, such as the first corrected image, is structurally consistent with the undistorted true image. out , the real image without distortion is I gt , then the reconstruction loss L recon The expression is:

[0100] L recon =||I out -I gt ||1;

[0101] Among them, I out is the corrected image, I out is the true image, and ‖·‖1 represents the L1 norm, which is the absolute difference in pixel level between the true image and the rectified image.

[0102] The reconstruction loss directly measures the pixel difference between the corrected image and the real image, making the corrected image gradually approach the real image in structure. The distortion correction module can retain the structural consistency of the image during the correction process and reduce the structural deviation caused by error accumulation.

[0103] Among them, the adversarial loss can be used to improve the overall realism of the corrected image, making the corrected image more consistent with the texture and details of the real image in terms of visual effect. cAnd discriminator D. The generator generates a second corrected image based on the input first distorted image, and the discriminator tries to distinguish the second corrected image from the real undistorted image, adversarial loss L adv The expression can be:

[0104] L adv =minmax((E[logD(I gt )]+E[log(1-D(G c (I in )))]);

[0105] Among them, I gt is the real image, I in is the first distorted image of the input, G c is a generator for generating rectified images, and D is a discriminator that tries to distinguish rectified images from real images.

[0106] The generator learns the detailed texture of the image through adversarial loss, making the corrected image more natural and realistic in visual details. The adversarial loss enhances the overall realism of the corrected image and reduces the artifacts and blur that may be generated during the correction process.

[0107] Among them, the enhancement loss can be used to enrich the local details of the corrected image, making the corrected image closer to the real image in terms of content and style. The enhancement loss ensures the consistency of the corrected image in local details and texture distribution, making the image more detailed at all scales and levels.

[0108] In some embodiments, the first corrected image and the real image can be continuously input into the distortion correction module, and the model parameters can be adjusted based on the feedback of the reconstruction loss, the adversarial loss, and the enhancement loss. When the reconstruction loss, the adversarial loss, and the enhancement loss are all stable and no longer significantly reduced, the training ends and the final distortion correction module is obtained. The distortion correction module after training can generate high-quality corrected images, which not only retains the structural similarity, but also has a high sense of reality and rich details in visual effects.

[0109] In this application, the structural consistency between the corrected image and the real image is ensured through reconstruction loss, the overall realism of the corrected image is enhanced through adversarial loss, and the details and style of the corrected image are enriched through enhancement loss. Under the synergistic effect of the above three types of loss functions, the distortion correction module can gradually correct the distorted image, so that the corrected image is close to the real undistorted image in terms of vision and content, achieving a high-precision and high-quality correction effect.

[0110] In some embodiments, the enhancement loss can be constructed as follows:

[0111] First, according to the difference between the first feature map and the second feature map, a content loss for minimizing the image content difference between the first corrected image and the true image is constructed;

[0112] Next, constructing a style loss for minimizing the image style difference between the first corrected image and the true image according to the difference between the first linear matrix and the second linear matrix;

[0113] Finally, the enhancement loss is constructed based on the sum of the content loss and the style loss.

[0114] Among them, the content loss is used to ensure that the details of the corrected image are consistent with the real image by comparing the feature maps of the corrected image and the real image on a specific layer of the pre-trained VGG-16 network.

[0115] In a specific implementation, the content loss can be constructed based on the difference between the first feature map of the first corrected image and the second feature map of the true undistorted image. The content loss minimizes the difference in content between the corrected image and the true image by comparing the feature maps of the two. The feature map is usually extracted by a pre-trained neural network, such as a VGG-16 network, at a specific layer to represent the high-level semantic information of the image. Content loss The expression can be:

[0116]

[0117] in, Represents the feature map of the jth layer of the VGG network, I out is the first corrected image, I gt is the real image, C j , H j , W j They represent the number of channels, height, and width of the feature map respectively.

[0118] Content loss can ensure that the corrected image matches the real image in terms of content information, retain the key details and structural information of the image, and make the corrected image closer to the semantic content of the real image.

[0119] The style loss is used to keep the overall texture style of the corrected image and the real image consistent. In a specific implementation, the style loss can be constructed based on the difference between the first linear matrix of the first corrected image and the second linear matrix of the undistorted real image.

[0120] Style loss can measure the style consistency of the two by comparing the linear matrix difference between the corrected image and the real image. The linear matrix is ​​usually obtained by calculating the Gram matrix of the feature map, which can capture the texture and style information of the image. The expression can be:

[0121]

[0122] in, represents the Gram matrix calculated on the j-th layer feature map, I out is the first corrected image, I gt is a real image, represents the squared Frobenius norm.

[0123] Style loss can ensure that the texture and style features of the corrected image are as consistent as possible with the real image, making the corrected image more coordinated and natural in overall style, and improving the visual consistency between the corrected image and the undistorted real image.

[0124] Among them, the enhancement loss can be a weighted combination of content loss and style loss. The expression of the enhancement loss can be:

[0125] L e =L c +λ s L s ;

[0126] Among them, L e is the enhancement loss, L c is the content loss, L s is the enhancement loss, λ s is a hyperparameter used to adjust the weights of content loss and style loss in the overall enhancement loss.

[0127] By combining content loss and style loss, the enhancement loss can optimize both the content and style of the corrected image. The content loss can ensure that the details of the corrected image are consistent with the content of the real image, while the style loss ensures that the style and texture of the image are natural and harmonious.

[0128] Through the above methods, the enhancement loss of this application can achieve dual optimization of the local details and overall style of the corrected image. The content loss and style loss respectively ensure the accuracy and consistency of the corrected image in terms of content and style, so that the corrected image is visually closer to the real image, achieving a more natural and detailed correction effect.

[0129] Step S102: Perform multi-scale loss training on the intermediate image correction network according to the first distorted image and the preliminary corrected image to obtain a target image correction network.

[0130] Among them, multi-scale loss is a mechanism for loss calculation based on feature maps of different scales. It optimizes the correction effect layer by layer from coarse to fine and from global to local, allowing the model to learn features at different resolutions and levels, making the correction effect more comprehensive and smooth.

[0131] During the multi-scale loss training process, the first distorted image and the preliminary corrected image can be input into the intermediate image correction network, and the correction ability of the network can be optimized layer by layer through multi-scale loss. Feature maps of different resolutions are generated on convolutional layers of different scales, and multi-scale loss is applied to these feature maps to ensure that the feature maps at each level can accurately describe the corrected image structure.

[0132] In some embodiments, the output of each layer can be corrected step by step by multi-scale loss by calculating the difference between the corrected image and the real image at each scale, for example, by using L1 or L2 norm), so that the model can more accurately restore the distorted features of the image.

[0133] The image correction process of multi-scale loss training can be divided into coarse-scale low-level correction and fine-scale high-level correction. Low-level correction is to first correct the global distortion structure of the image at the low-resolution level of the image correction network, so that large distorted areas can be initially restored. High-level correction is that as the convolution level gradually increases, the image correction network can correct local details at a higher resolution scale to capture tiny distortions and detail differences. Through the progressive correction training method, the image correction network can restore the overall structure while retaining the details of the image, thereby enhancing the network training effect and the image correction effect.

[0134] Through layer-by-layer training of multi-scale loss, collaborative optimization can be achieved between low-resolution and high-resolution feature layers. Multi-scale loss optimizes the correction results at each scale to ensure that the corrected image achieves the desired correction effect in both global structure and local details. When the multi-scale loss converges, the training process ends and the optimized target image correction network is obtained. The network can comprehensively correct distorted images at different scales and has excellent correction accuracy and detail restoration capabilities.

[0135] The present application performs multi-scale loss training on the intermediate image correction network through the first distorted image and the preliminary corrected image, so that the image correction network can optimize the correction effect layer by layer in the features of different scales from coarse to fine, thereby generating the final target image correction network. Subsequently, the target correction network can integrate the correction requirements of the global structure and local details to generate a high-quality corrected image, ensuring the smoothness of the correction effect and the retention of details.

[0136] In some embodiments, step S102 may include:

[0137] Firstly, a multi-scale loss for minimizing the distortion defects of features of different scales is constructed according to the first distorted image and the preliminary corrected image;

[0138] Next, the overall loss of the target image correction network is constructed based on the reconstruction loss, adversarial loss, enhancement loss, and multi-scale loss.

[0139] Finally, the initial image correction network is iteratively trained until the overall loss converges and the training is terminated to obtain the target image correction network.

[0140] In some embodiments, multi-scale loss can be used to minimize the distortion defects of the correction network at different scales to ensure that the correction effect maintains consistency and smoothness at all levels and scales. In a specific implementation, multi-scale feature maps can be generated at different resolutions based on the first distorted image and the preliminary corrected image.

[0141] In some embodiments, the overall loss function can optimize the overall performance of the correction network layer by layer by integrating reconstruction loss, adversarial loss, enhancement loss and multi-scale loss, so that it is closer to the undistorted image in both global and local features.

[0142] In some embodiments, the first distorted image and the preliminary corrected image can be input into the initial image correction network, and iterative training can be performed based on the overall loss function. In each round of training, the preliminary correction network can continuously adjust its parameters to minimize the overall loss and gradually improve the quality of the corrected image. When the change in the overall loss reaches the convergence standard, that is, the value of the overall loss tends to be stable and no longer decreases significantly, the training ends and the initial correction network is optimized to the target image correction network.

[0143] In some embodiments, the multi-scale loss can be constructed as follows:

[0144] Firstly, multi-scale feature extraction is performed on the first distorted image to generate first feature maps of different scales;

[0145] Next, the original features of each first feature map and the corrected features of the preliminary corrected image are obtained;

[0146] Then, the original features and the corrected features of the corresponding scale are concatenated to obtain the concatenated features, and a convolution operation is performed on the concatenated features to generate a corrected second feature map;

[0147] Then, the second feature map is sampled to obtain a multi-scale second feature map;

[0148] Finally, a multi-scale loss is constructed based on the difference between the second feature maps of multi-scales and the real images of the corresponding scales.

[0149] In a specific implementation, multi-scale feature extraction can be performed on the first distorted image to generate first feature maps of different scales. For example, the input first distorted image can be processed through operations such as convolutional layers in the intermediate image correction network to obtain multiple first feature maps of different resolutions, i.e., multi-scale features. The first feature map of each scale contains image information at different levels, such as detail features and global information.

[0150] In some embodiments, original features and corrected features of the preliminary corrected image may be extracted from the first feature map of each scale, respectively. The original features may come from the original feature map in the decoder, and the corrected features may come from the feature correction layer.

[0151] In some embodiments, the original features and the corrected features of the corresponding scale can be connected in series, and the series operation can combine the two feature maps along the channel dimension to form a new second feature map. Specifically, a convolution operation can be performed on the concatenated feature map, for example, a 3x3 convolution kernel can be used to obtain a corrected second feature map, and the features are fused and converted into a higher-level expression through the convolution operation.

[0152] In some embodiments, a sampling operation may be performed on the obtained second feature map, for example, a downsampling operation may be performed to reduce the resolution of the feature map while maintaining important information in the original image. Second feature maps of different scales may be generated by performing downsampling operations of different times, thereby performing loss calculations at multiple scales, and each downsampling operation may generally halve the size of the image.

[0153] In some embodiments, the difference between the second feature map of each scale and the real image of the corresponding scale can be calculated. The difference can be measured by calculating the L1 norm between the two, and the expression of the multi-scale loss can be:

[0154]

[0155] Among them, (I gt ,i) is the feature of the real image at the i-th scale, represents the original features output by the decoder, They represent the corrected features output by the intermediate image correction network, and ⊕ represents the feature concatenation operation. C is a convolution operation, which is used to decode the concatenated features into an RGB image. S is a downsampling operation, which is used to reduce the real image to the corresponding scale. ‖·‖1 is the L1 norm, which is used to calculate the difference between features.

[0156] Multi-scale loss can compare the correction results of each scale with the real image and calculate the error between them, thereby guiding the network to optimize the feature correction process of each scale. By supervising the feature map at each scale through multi-scale loss, it can effectively promote the network to learn the feature correction of each scale. Compared with the traditional single-scale loss, the use of multi-scale loss can take into account the errors at different resolutions, ensure that the features at different scales can be effectively corrected, and improve the effect of image correction.

[0157] This application combines multi-scale feature map training to perform fine correction on images at different levels of features, ensures that feature maps of multiple scales are corrected with minimized distortion, and supervises the image correction network for multi-scale optimization training by calculating the difference between the corrected image and the real image at each scale, thereby improving the overall correction performance of the image correction network.

[0158] In some embodiments, the overall loss is constructed as follows:

[0159] First, a first hyperparameter for adjusting the reconstruction loss and a second hyperparameter for adjusting the multi-scale loss are obtained;

[0160] Next, adjusting the reconstruction loss according to the first hyperparameter to obtain an adjusted reconstruction loss, and adjusting the multi-scale loss according to the second hyperparameter to obtain an adjusted multi-scale loss;

[0161] Finally, the overall loss is constructed based on the sum of the adjusted reconstruction loss, the adjusted multi-scale loss, the adversarial loss and the enhancement loss.

[0162] The first hyperparameter can be used to adjust the reconstruction loss and control the weight of the reconstruction loss in the overall loss. The second hyperparameter can be used to adjust the multi-scale loss and control the contribution of the multi-scale loss in the overall loss.

[0163] In some embodiments, the adjusted reconstruction loss, the adjusted multi-scale loss, the adversarial loss, and the enhancement loss may be combined to construct an overall loss function, and the expression of the overall loss function may be:

[0164] L=λ r L r +L adv +λ m L m +L e ;

[0165] Among them, L is the overall loss, L r is the reconstruction loss, λ r is the first hyperparameter, L adv is the loss against which Lm is the multi-scale loss, λ m is the second hyperparameter, L e It is the enhancement loss.

[0166] The overall loss function achieves comprehensive constraints and optimization of the correction network through the combination of multiple loss terms. Each loss term focuses on the structure, details, smoothness and realism of the image, and together improves the quality of the corrected image.

[0167] This application can achieve progressive and complementary training of the image correction network through overall loss, optimize the correction effect layer by layer through multi-scale loss, and gradually improve the correction accuracy from low resolution to high resolution. The reconstruction loss, adversarial loss, and enhancement loss complement each other, so that the correction network can be continuously optimized in different aspects, and the quality of the corrected image is gradually improved. The overall loss function combines the advantages of each loss term, so that the network is continuously adjusted and optimized during iterative training, realizing the progressive and complementary joint training of the correction network, thereby obtaining higher quality corrected images.

[0168] Step S103: input the second distorted image to be corrected into the target image correction network for correction to obtain a target corrected image.

[0169] The second distorted image may be an original fisheye distorted image to be corrected, which has obvious distortion features, such as edge stretching, bending, and nonlinear distortion. It should be noted that the second distorted image and the first distorted image may be the same image or different images, and the embodiment of the present application does not limit this.

[0170] In some embodiments, the second distorted image can be processed by a trained target image correction network to be corrected into a target corrected image. The target corrected image is close to the undistorted image in structure and details, and the visual effect is natural and consistent with the overall style of the real image.

[0171] In the embodiment of the present application, an intermediate image correction network is obtained by training the initial image correction network, and the distorted image is pre-corrected by the intermediate image correction network to obtain a pre-corrected image for subsequent training, which is equivalent to embedding a feature-level correction layer in the jump connection, thereby effectively reducing the computational burden of the decoder and improving the operational efficiency and correction effect of the correction network. In addition, the target image correction network is obtained by multi-scale training the intermediate correction network with distorted images and pre-corrected images, so that the distorted image can be optimized layer by layer, the consistency of the content and structure of the corrected image can be ensured, the blur and incomplete structure of the corrected image can be reduced, and the number of labels required for training can be reduced, thereby reducing the difficulty of model training and the cost of data annotation in the image correction process.

[0172] Figure 3is a block diagram of an image correction system provided in an embodiment of the present application. The image correction system 200 is used to perform the above image correction method. The image correction system 200 may include an image acquisition unit 201, a communication unit 202, a computing unit 203 and an image display unit 204, wherein:

[0173] The image acquisition unit 201 is used to acquire the distorted image to be corrected; the communication unit 202 is used to transmit the distorted image to the computing unit 203; the computing unit 203 is used to correct the distorted image to obtain the corrected image, and send the corrected image to the image display unit 204; the image display unit 204 is used to receive and display the corrected image.

[0174] In a specific implementation, the image acquisition unit 201 can use a camera module development board equipped with a fisheye lens to shoot the real environment to capture a distorted image with obvious edge stretching and distortion, which requires a correction algorithm to restore the normal visual effect of the image.

[0175] The communication unit 202 is responsible for transmitting the distorted image from the image acquisition unit 201 to the computing unit 203 for correction. For example, the distorted image can be sent to the computing server through the communication protocol, which can effectively utilize the computing resources and realize real-time and fast image correction.

[0176] The computing unit 203 is the core part of the entire system and is responsible for executing the correction process of the distorted image. The computing unit 203 can receive the distorted image transmitted in real time from the communication unit 202, execute the end-to-end image correction algorithm of the present application, and perform multi-scale and feature-level correction processing on the distorted image. The network's multi-scale loss, reconstruction loss, and adversarial loss can be used for image correction to generate a distortion-free output image. With the help of an efficient inference engine and hardware optimization, the computing unit 203 can process the distorted image in a short time to meet the needs of real-time processing. The corrected image can also be sent to a designated terminal device through the communication unit 202 for the user to view or further process. The optimized design of the computing unit 203 can ensure the real-time and image quality of the correction process, and provide users with clear and accurate corrected images.

[0177] In some embodiments, the image display unit 204 can be a vehicle-mounted terminal, a mobile phone APP terminal, and a Web terminal. The corrected image can be displayed on different terminals to meet different usage requirements. In the vehicle-mounted terminal in parking or assisted driving scenarios, the corrected image is displayed in real time through the vehicle-mounted system to help the driver view the vehicle's surroundings more accurately. The mobile phone APP terminal is suitable for image viewing on mobile devices, such as driving recorder applications, and users can view the corrected image in real time on their mobile phones. The Web terminal supports viewing the corrected image in a browser for easy monitoring and remote viewing. In response to different industry needs, the image can be transmitted to a specific terminal device or platform to meet specific application requirements. The corrected image is transmitted to a designated terminal via a network or wirelessly, and the user views the corrected distortion-free image in real time on the corresponding device.

[0178] The image correction system 200 can realize real-time distortion correction of distorted images through the coordinated work of the image acquisition unit 201, the communication unit 202, the computing unit 203 and the image display unit 204. The computing unit 203 uses algorithms such as multi-scale, reconstruction, adversarial and enhanced loss to correct images at different scales and feature levels, and transmits high-quality corrected images to different terminal devices to meet the needs of different application scenarios. This design effectively improves the user experience, so that the corrected image can adapt to the display requirements of different terminals, thereby enhancing the practicality and flexibility of the system.

[0179] Figure 4 FIG. 1 is a block diagram of an electronic device 300 according to an exemplary embodiment. Figure 4 As shown, the electronic device 300 may include: a processor 301 and a memory 302. The electronic device 300 may also include one or more of a multimedia component 303, an input / output (I / O) component 304, and a communication component 305.

[0180] The processor 301 is used to control the overall operation of the electronic device 300 to complete all or part of the steps in the above-mentioned image correction method. The memory 302 is used to store various types of data to support the operation of the electronic device 300, which may include instructions for any application or method used to operate on the electronic device 300, as well as application-related data, such as contact data, messages sent and received, pictures, audio, video, etc. The memory 302 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (Static Random Access Memory, referred to as SRAM), electrically erasable programmable read-only memory (Electrically Erasable Programmable Read-Only Memory, referred to as EEPROM), erasable programmable read-only memory (Erasable Programmable Read-Only Memory, referred to as EPROM), programmable read-only memory (Programmable Read-Only Memory, referred to as PROM), read-only memory (Read-Only Memory, referred to as ROM), magnetic memory, flash memory, disk or optical disk. The multimedia component 303 may include a screen and an audio component. The screen may be, for example, a touch screen, and the audio component is used to output and / or input audio signals. For example, the audio component may include a microphone for receiving external audio signals. The received audio signal may be further stored in the memory 302 or sent through the communication component 305. The audio component also includes at least one speaker for outputting audio signals. The I / O component 304 provides an interface between the processor 301 and other interface modules, and the other interface modules may be keyboards, mice, buttons, etc. These buttons may be virtual buttons or physical buttons. The communication component 305 is used for wired or wireless communication between the electronic device 300 and other devices. Wireless communication, such as WiFi, Bluetooth, Near Field Communication (NFC), 2G, 3G, 4G, NB-IOT, eMTC, or other 5G, etc., or a combination of one or more of them, is not limited here. Therefore, the corresponding communication component 305 may include: WiFi module, Bluetooth module, NFC module, etc.

[0181] In an exemplary embodiment, the electronic device 300 can be implemented by one or more application specific integrated circuits (ASIC), digital signal processors (DSP), digital signal processing devices (DSPD), programmable logic devices (PLD), field programmable gate arrays (FPGA), controllers, microcontrollers, microprocessors or other electronic components to perform the above-mentioned image correction method.

[0182] In another exemplary embodiment, a computer-readable storage medium including program instructions is also provided, and when the program instructions are executed by the electronic device 300, the steps of the above-mentioned image correction method are implemented. For example, the computer-readable storage medium may be the above-mentioned memory 302 including program instructions, and the above-mentioned program instructions may be executed by the processor 301 of the electronic device 300 to complete the following steps:

[0183] Inputting the first distorted image to be corrected into the initial image correction network for training to obtain an intermediate image correction network, and correcting the first distorted image through the intermediate image correction network to obtain a preliminary corrected image;

[0184] According to the first distorted image and the preliminary corrected image, a multi-scale loss training is performed on the intermediate image correction network to obtain a target image correction network;

[0185] The second distorted image to be corrected is input into the target image correction network for correction to obtain a target corrected image.

[0186] Figure 5 is a block diagram of a vehicle provided in an embodiment of the present application, such as Figure 5 As shown, the vehicle 400 includes the electronic device 300 described above.

[0187] An embodiment of the present application also provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program for electronic data exchange, and the computer program enables a computer to execute part or all of the steps of any image correction method recorded in the above method embodiments.

[0188] It should be noted that, for the aforementioned method embodiments, for the sake of simplicity, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the described order of actions, because according to the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present application.

[0189] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0190] In the several embodiments provided in the present application, it should be understood that the disclosed device can be implemented in other ways. For example, the device embodiments described above are only schematic, such as the division of the units, which is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, and the indirect coupling or communication connection of the device or unit can be electrical or other forms.

[0191] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0192] In addition, the functional units in the various embodiments of the application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated units may be implemented in the form of hardware or in the form of software program modules.

[0193] If the integrated unit is implemented in the form of a software program module and sold or used as an independent product, it can be stored in a computer-readable storage unit. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage unit, including a number of instructions to enable a computer device (which can be a personal computer, server or network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage unit includes: U disk, read-only memory (ROM), random access memory (RAM), mobile hard disk, disk or optical disk and other media that can store program codes.

[0194] A person of ordinary skill in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructing related hardware through a program, and the program can be stored in a computer-readable storage unit, which can include: a flash drive, a read-only storage unit, a random access memory, a magnetic disk or an optical disk, etc.

[0195] The preferred embodiments of the present application are described in detail above in conjunction with the accompanying drawings; however, the present application is not limited to the specific details in the above embodiments. Within the technical concept of the present application, a variety of simple modifications can be made to the technical solution of the present application, and these simple modifications all fall within the protection scope of the present application.

[0196] It should also be noted that the various specific technical features described in the above specific embodiments can be combined in any suitable manner without contradiction. In order to avoid unnecessary repetition, this application will not further describe various possible combinations.

[0197] In addition, the various implementation modes of the present application may be arbitrarily combined, and as long as they do not violate the concept of the present application, they should also be regarded as the contents disclosed in the present application.

Claims

1. An image correction method, characterized in that: The method comprises: Inputting the first distorted image to be corrected into an initial image correction network for training to obtain an intermediate image correction network, and correcting the first distorted image through the intermediate image correction network to obtain a preliminary corrected image; According to the first distorted image and the preliminary corrected image, performing multi-scale loss training on the intermediate image correction network to obtain a target image correction network; The second distorted image to be corrected is input into the target image correction network for correction to obtain a target corrected image.

2. The image correction method according to claim 1, characterized in that: The step of inputting the first distorted image to be corrected into the initial image correction network for training to obtain an intermediate image correction network comprises: Synchronously inputting the first distorted image into a flow estimation module and a distortion correction module to obtain appearance flow information of the first distorted image output by the flow estimation module and a first corrected image of the first distorted image output by the distortion correction module; The flow estimation module and the distortion correction module are synchronously trained according to the appearance flow information, the first corrected image and the first distorted image to obtain the intermediate image correction network.

3. The image correction method according to claim 2, characterized in that: The step of synchronously training the flow estimation module and the distortion correction module according to the appearance flow information, the first corrected image, and the first distorted image to obtain the intermediate image correction network includes: Inputting the appearance flow information into the distortion correction module, and iteratively training the distortion correction module by combining the appearance flow information and the first distorted image; Inputting the first corrected image into the flow estimation module, and iteratively training the flow estimation module in combination with the first distorted image and the first corrected image; When the flow estimation module and the distortion correction module meet the training completion condition, the training is terminated to obtain the intermediate image correction network.

4. The image correction method according to claim 3, characterized in that: The method further comprises: The first distorted image is input into the flow estimation module, image features of the first distorted image are extracted through the convolutional neural layer of the flow estimation module, and appearance flow information of the first distorted image is generated based on the image features.

5. The image correction method according to claim 3, characterized in that: The step of inputting the appearance flow information into the distortion correction module and iteratively training the distortion correction module in combination with the appearance flow information and the first distorted image comprises: According to the first corrected image and the undistorted real image corresponding to the first corrected image, constructing a reconstruction loss for minimizing the difference between the first corrected image and the real image; Constructing an adversarial loss for enhancing the overall authenticity of the first corrected image according to a second corrected image generated by the generator of the distortion correction module based on the first distorted image and a discrimination result of the second corrected image and the real image by the discriminator of the distortion correction module; Constructing an enhancement loss for enriching local details of the first corrected image according to a first feature map and a first linear matrix of the first corrected image, and a second feature map and a second linear matrix of the real image; The distortion correction module is iteratively trained according to the first corrected image and the real image until the reconstruction loss, the adversarial loss and the enhancement loss converge, and then the training is terminated to obtain a trained distortion correction module.

6. The image correction method according to claim 5, characterized in that: The constructing an enhancement loss for enriching local details of the first corrected image comprises: constructing a content loss for minimizing an image content difference between the first corrected image and the real image according to a difference between the first feature map and the second feature map; constructing a style loss for minimizing the image style difference between the first corrected image and the real image according to the difference between the first linear matrix and the second linear matrix; The enhancement loss is constructed according to the sum of the content loss and the style loss.

7. The image correction method according to claim 5, characterized in that: The step of performing multi-scale loss training on the intermediate image correction network according to the first distorted image and the preliminary corrected image to obtain a target image correction network includes: constructing a multi-scale loss for minimizing distortion defects of features at different scales according to the first distorted image and the preliminary corrected image; Constructing an overall loss of the target image correction network according to the reconstruction loss, the adversarial loss, the enhancement loss and the multi-scale loss; The initial image correction network is iteratively trained until the overall loss converges and the training is terminated to obtain the target image correction network.

8. The image correction method according to claim 7, characterized in that: The multi-scale loss constructed to minimize the distortion defects of features at different scales includes: Performing multi-scale feature extraction on the first distorted image to generate first feature maps of different scales; Acquire original features of each of the first feature maps and corrected features of the preliminary corrected image; Performing feature concatenation on the original feature and the corrected feature of the corresponding scale to obtain a concatenated feature, and performing a convolution operation on the concatenated feature to generate a corrected second feature map; Sampling the second feature map to obtain a multi-scale second feature map; The multi-scale loss is constructed based on the difference between the multi-scale second feature maps and the real image of the corresponding scale.

9. The image correction method according to claim 7, characterized in that: The step of constructing the overall loss of the target image correction network according to the reconstruction loss, the adversarial loss, the enhancement loss and the multi-scale loss includes: Acquire a first hyperparameter for adjusting the reconstruction loss and a second hyperparameter for adjusting the multi-scale loss; Adjusting the reconstruction loss according to the first hyperparameter to obtain an adjusted reconstruction loss, and adjusting the multi-scale loss according to the second hyperparameter to obtain an adjusted multi-scale loss; The overall loss is constructed according to the sum of the adjusted reconstruction loss, the adjusted multi-scale loss, the adversarial loss and the enhancement loss.

10. An image correction system, used to perform the method according to any one of claims 1 to 9, characterized in that: The system includes an image acquisition unit, a communication unit, a computing unit and an image display unit, wherein: The image acquisition unit is used to acquire the distorted image to be corrected; The communication unit is used to transmit the distorted image to the computing unit; The computing unit is used to correct the distorted image to obtain a corrected image, and send the corrected image to the image display unit; The image display unit is used to receive and display the corrected image.

11. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores program instructions, which implement the method according to any one of claims 1 to 9 when executed.

12. A computer program product, characterized in that The computer program product comprises program instructions, and when the program instructions are executed by a processor, the method according to any one of claims 1 to 9 is implemented.

13. An electronic device, characterized in that: Used to perform the method according to any one of claims 1 to 9.

14. A vehicle, characterized in that: The electronic device comprising the electronic device described in claim 13 above.

Images