AVM image distortion correction method based on cross-domain migration network

Through the variational adversarial network model of the cross-domain migration network, combined with the generation adversarial network and the variational autoencoding network, the problems of low image distortion correction accuracy and poor scene generalization ability are solved, and high efficiency in various environments is achieved , accurate image correction, improving the visual perception performance of the intelligent driving system.

CN118505545BActive Publication Date: 2024-11-12AUTOLINK INFORMATION TECHNOLOGY CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202410918168.4
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-07-10
Publication Date
2024-11-12
Estimated Expiration
2044-07-10

AI Technical Summary

Technical Problem

In the prior art, the image distortion correction accuracy of online self-calibration for vehicle-mounted cameras is limited and the scene generalization performance is poor, making it difficult to achieve efficient and accurate image correction in various environments.

Method used

Using a variational adversarial network model based on a cross-domain migration network, combining the generation adversarial network and a variational autoencoding network, the non-distortion characteristics of the input image are extracted through the encoder, the decoder generates a corrected image, and the discriminator uses the discriminator to judge the virtual image and the standard image. The authenticity of the image can be realized in real time.

Benefits of technology

It improves the accuracy of image correction and scene generalization capabilities of on-board cameras, can correct distorted images in real time in various environments, and enhances the visual perception accuracy of the intelligent driving system.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118505545B_ABST
    Figure CN118505545B_ABST
Patent Text Reader

Abstract

The present application provides an AVM image distortion correction method based on a cross-domain migration network, which uses a variational autoencoder network model to replace the generator network in the generative adversarial network model to obtain the variational adversarial network model, fully combining the respective advantages of the adversarial generative network and the variational autoencoder network, and can effectively utilize the powerful discriminative ability of the discriminative network, fully guaranteeing the generation of virtual samples that are consistent with the pixel distribution probability of the real sample space, and with the help of adversarial training, the generation ability of the generation network is strong enough to effectively extract non-distorted domain features in the distorted image, thereby achieving the purpose of correcting the distorted image.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of image recognition, and in particular to an AVM image distortion correction method based on a cross-domain migration network. Background Art

[0002] With the rapid development of new energy vehicle technology, high-level autonomous driving solutions with BEV perception as the core are rapidly popularizing. Among them, the image distortion of the on-board panoramic imaging system (AVM) has an important impact on the improvement of intelligent driving perception performance. The manufacturing process of the on-board camera and the deformation of the vehicle body will cause the AVM image to be distorted to varying degrees. This will seriously restrict the improvement of the visual perception accuracy of the intelligent driving system. The internal parameters of the camera manufacturing process are solidified when it leaves the factory. These parameters are generally focal length, lens distortion coefficient, pixel size, principal point coordinates and distortion parameters (radial distortion and tangential distortion). Image distortion correction caused by such factors is more difficult. The image distortion caused by the transformation of the external position relationship of the camera is mainly reflected in the translation and rotation of the image. At present, the camera external parameter calibration method is mainly used to correct the position of the camera, so that the camera can return to its original position and generate a normal image.

[0003] like Figure 1 As shown in (a) in the figure, camera calibration is simply to calibrate the camera from the world coordinate system (using X w ,Y w ,Z w ) into the camera coordinate system (with X c ,Y c ,Z c denoted by), usually based on the use of homogeneous coordinate transformation matrix; then the camera coordinate system is converted to the image coordinate system (expressed by X, Y). This transformation can be regarded as a simple projective transformation (the camera is regarded as a pinhole imaging model), transforming the three-dimensional coordinates into two-dimensional coordinates; where f is the focal length of the camera; since the basic unit of the image is pixel, the pixel position of the point on the image in the image storage matrix is ​​stored in the pixel coordinate system, with the origin of the coordinates in the upper left corner, and the coordinate system is represented by u, v, such as Figure 1 As shown in (b) in the figure, the transformation from the image coordinate system to the pixel coordinate system is: Assume that there are f pixels per millimeter in the x direction and f pixels per millimeter in the y direction, then: dx, dy represent the number of millimeters in each column and row respectively; (u0, v0) are the coordinates of the origin of the image coordinate system in the pixel coordinate system. The camera calibration process is the process of finding the projection matrix.

[0004] For the pinhole camera model, camera calibration refers to determining the intrinsic parameter matrix and the extrinsic parameter matrix to determine the unique camera model.

[0005] Camera intrinsic parameters: mainly include focal length, image principal point coordinates, and distortion parameters, i.e. f x ,f y ,c x ,c y ,k1,k2,k3,p2,p2;

[0006] Camera external parameters: including rotation matrix and translation matrix, namely R,t.

[0007] In the prior art, the distortion correction of vehicle-mounted cameras is mainly divided into two methods: offline calibration and online self-calibration. The former generally takes the vehicle to a specific place and uses the checkerboard calibration method to jointly calibrate the four cameras on the vehicle body; the latter usually only requires the vehicle to self-calibrate the external parameters of the camera according to the lane line or parking space line during normal driving. Of these two distortion correction methods, the former has higher correction accuracy but is not convenient enough. In order to improve efficiency, many companies use the latter method. However, online self-calibration can perform adaptive correction but often has limited accuracy and poor scene generalization performance. Summary of the invention

[0008] In order to solve the problem in the prior art that online self-calibration can adaptively correct but often has limited accuracy and poor scene generalization performance, the present invention provides an AVM image distortion correction method based on a cross-domain migration network, which can more accurately correct the distorted image output by the vehicle-mounted camera in real time, and the method has strong scene generalization ability.

[0009] The technical solution of the present invention is as follows: a method for AVM image distortion correction based on a cross-domain migration network, characterized in that it comprises the following steps:

[0010] S1: Combining the generative adversarial network model and the variational autoencoder network model to construct a variational adversarial network model;

[0011] The variational adversarial network model is obtained by replacing the generator network in the generative adversarial network model with a variational autoencoder network model; the variational adversarial network model includes: an encoder inference network, a decoder generation network and a discriminator network connected in sequence;

[0012] The encoder inference network unit compresses the input data into a low-dimensional code; the encoder inference network unit gradually learns the ability to extract non-distorted features of the input data samples by minimizing the sample reconstruction error and minimizing the KL divergence of the pixel distribution between the input image and the sampled sample;

[0013] The decoder generates a network that decodes the low-dimensional code into reconstructed data similar to the original data;

[0014] The discriminator network distinguishes between the virtual image generated by the decoder generation network and the input standard image;

[0015] S2: Construct a training dataset based on the distorted images and the standard images corresponding to each distorted image;

[0016] S3: Training the variational adversarial network model based on the training data set to obtain a trained variational adversarial network model;

[0017] S4: construct image correction unit;

[0018] The image correction unit includes: a feature extraction network and a trained decoder generation network;

[0019] The feature extraction network extracts features from the input image to be corrected, extracts underlying edge features and generates a high-dimensional feature vector, and sends the high-dimensional feature vector to the decoder generation network. The decoder generation network outputs reconstructed data similar to the original data, which is the corrected image;

[0020] S5: sending the image captured by the vehicle-mounted panoramic imaging system as the image to be corrected to the image correction unit in real time, and outputting the corresponding corrected image.

[0021] It is further characterized by:

[0022] The workflow in the variational adversarial network model is:

[0023] The distorted image is input as the input image x1 into the encoder inference network of the variational generative adversarial network, and the encoder inference network e1(x) is used to extract the expected μ1(x) and variance σ1(x) of the pixel distribution of the input image x1; a random sample N(0,1) conforming to the standard normal distribution is constructed, and the variational generative adversarial network calculates the KL divergence between μ1(x) and the sampled sample, and the KL divergence between σ1(x) and the sampled sample, respectively. After obtaining the constraint of the pixel distribution similarity between the input image and the sampled sample, a latent variable Z1 is sampled from the random sample N(0,1); the latent variable Z1 is input into the decoder generation network d1(Z) to generate a virtual sample; finally, the standard image and the virtual image corresponding to the distorted image are respectively sent to the discriminator network D1, and the authenticity of the input virtual image is judged based on the discriminator network, and the judgment result of whether the distorted image and the standard image belong to the same sample is output;

[0024] The overall loss function of the variational generative adversarial network is for:

[0025] ,

[0026] in, is the KL divergence of the pixel distribution between the input image and the sample, To counter the discriminative loss function, λ1 and λ2 are two empirical thresholds;

[0027] In the variational adversarial network model, the KL divergence of the pixel distribution between the input image and the sampled sample is calculated as follows:

[0028] ,

[0029] in, is the KL divergence of the pixel distribution between the input image and the sample; D KL is the relative entropy, which indicates the distance constraint between the real sample and the sampled sample; p(Z i 1) is the noise distribution, q(Z i 1|x i 1) The decoder takes the input low-dimensional code Z i 1 output; i represents the serial number of the sample sampled from the random sample N(0,1); Z i 1 is the variational adversarial network model that generates latent variables by sampling from random samples N(0,1);

[0030] In the variational adversarial network model, the loss function for generating virtual sample reconstruction error and discrimination is:

[0031] ,

[0032] in, To combat the discriminant loss function, G is the generator and D is the discriminator; x1 represents the input distorted image, D1(.) represents the output of the discriminator for the input data, d1(Z i 1) Generate network d1 for decoder for Z i 1 is the generated virtual sample;

[0033] The feature extraction network is implemented based on a deep convolutional network model CNN.

[0034] The present application provides an AVM image distortion correction method based on a cross-domain migration network, which uses a variational autoencoder network model to replace the generator network in the generative adversarial network model to obtain the variational adversarial network model, fully combines the respective advantages of the adversarial generative network and the variational autoencoder network, can effectively utilize the powerful discriminant ability of the discriminant network, and fully guarantees the generation of virtual samples with the same pixel distribution probability as the real sample space. With the help of adversarial training, the generation ability of the generation network is strong enough to effectively extract non-distortion domain features in the distorted image, thereby achieving the purpose of correcting the distorted image; the method optimizes the generator model for image distortion correction based on a deep learning method, and can perform real-time correction on the distorted image based on the model and the image decoder, and the model has a strong scene generalization ability. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 It is a schematic diagram of a camera extrinsic calibration algorithm in the prior art;

[0036] Figure 2 A flowchart of a generative adversarial network algorithm in the prior art;

[0037] Figure 3 It is a flow chart of the variational autoencoder network algorithm in the prior art;

[0038] Figure 4 This is a flowchart of the variational adversarial network model in this application;

[0039] Figure 5 This is a flow chart of the AVM image distortion correction method in this application. DETAILED DESCRIPTION

[0040] Generative models such as Generative Adversarial Networks (GANs) and Variational Autoencoders (VAEs) have achieved remarkable success in the fields of images, language, and text.

[0041] Generative Adversarial Networks (GANs) is a deep learning algorithm. The core idea of ​​the algorithm is to train by constructing a network consisting of a generator and a discriminator. The goal of the generator is to generate fake data similar to real data, and the goal of the discriminator is to distinguish between the generated fake data and real data. This adversarial process between the generator and the discriminator enables the generator to continuously learn how to generate fake data that is closer to real data, thereby achieving the goal of data generation.

[0042] like Figure 2 As shown in the figure, the algorithm principle and specific operation steps of the Generative Adversarial Network (GANs) are as follows:

[0043] 1) Initialize the parameters of the generator and discriminator.

[0044] 2) Training the generator: The goal of the generator is to generate fake data that is similar to real data. By minimizing the generative adversarial loss function, the generator can gradually learn to generate fake data that is similar to real data.

[0045] 3) Training the discriminator: The goal of the discriminator is to distinguish between generated fake data and real data. By minimizing the generative adversarial loss function, the discriminator can gradually learn the ability to distinguish between generated fake data and real data.

[0046] 4) Repeat steps 2 and 3 until the parameters of the generator and discriminator converge.

[0047] The mathematical model formula of Generative Adversarial Networks (GANs) is as follows:

[0048] ,

[0049] Among them, V(D,G) is the production adversarial loss function, D(x) represents the output of the discriminator for the input data x, G(z) represents the output of the generator for the input noise z, pdata represents the real data distribution, and pz represents the noise distribution.

[0050] Variational Autoencoders (VAE) are also a deep learning algorithm. The core idea of ​​VAEs is to train through a network consisting of an encoder and a decoder. The goal of the encoder is to compress the input data into a low-dimensional code, and the goal of the decoder is to decode this low-dimensional code into reconstructed data similar to the original data. VAEs are trained by minimizing the reconstruction error and the KL divergence between the encoder and decoder, thereby achieving the goals of data generation and data compression.

[0051] like Figure 3 As shown in the figure, the algorithm principle and specific operation steps of variational autoencoders (VAEs) are as follows:

[0052] 1) Initialize the parameters of the encoder and decoder.

[0053] 2) Training the encoder: The goal of the encoder is to compress the input data into a low-dimensional code. By minimizing the reconstruction error and the KL divergence between the encoder and decoder, the encoder can gradually learn the ability to compress the input data.

[0054] 3) Training the decoder: The goal of the decoder is to decode the low-dimensional code into reconstructed data similar to the original data. By minimizing the reconstruction error, the decoder can gradually learn the ability to decode the low-dimensional code.

[0055] 4) Repeat steps 2 and 3 until the parameters of the encoder and decoder converge.

[0056] The mathematical model formula of variational autoencoders (VAEs) is as follows:

[0057] ,

[0058] Where p(z) is the noise distribution, q(x│z) is the output of the decoder for the input low-dimensional code z, and D KL (q(z|x)||p(z)) is the KL divergence, which represents the distance between the encoder and decoder.

[0059] However, in practical applications, it is found that the GANs network training process is difficult to converge and the image generation effect is poor; the VAEs network generation effect is good, but the diversity of generated samples is limited. This scheme makes full use of the advantages of both to build a variational adversarial network (VGAN), using this network as the basic network unit to construct a cross-domain migration network to solve the distortion correction of AVM images. Considering the advantages and disadvantages of the generative adversarial network and the variational autoencoder network, the two are integrated to construct a new variational generative adversarial network for AVM image distortion correction.

[0060] like Figure 4 and Figure 5 As shown, the present invention includes an AVM image distortion correction method based on a cross-domain migration network, which includes the following steps.

[0061] S1: Combining the Generative Adversarial Network model with the Variational Autoencoder Network model to construct the Variational Generative Adversarial Networks (VGAN);

[0062] A variational adversarial network model is obtained by replacing the generator network in the generative adversarial network model with a variational autoencoder network model; the variational adversarial network model includes: an encoder inference network, a decoder generation network and a discriminator network connected in sequence;

[0063] The encoder inference network unit compresses the input data into a low-dimensional code by minimizing the reconstruction error and the KL divergence between the encoder and decoder;

[0064] The decoder generation network decodes the low-dimensional code into reconstructed data similar to the original data;

[0065] The discriminator network distinguishes between generated fake data and real data.

[0066] like Figure 4 As shown in the figure, the workflow in the variational adversarial network model is:

[0067] The distorted image is input as the input image x1 into the encoder inference network of the variational generative adversarial network, and the encoder inference network e1(x) is used to extract the expected μ1(x) and variance σ1(x) of the pixel distribution of the input image x1; a random sample N(0,1) that conforms to the standard normal distribution is constructed, and the variational generative adversarial network calculates the KL divergence between μ1(x) and the sampled sample, and the KL divergence between σ1(x) and the sampled sample, respectively. After obtaining the constraints on the similarity of the pixel distribution between the input image and the sampled sample, the latent variable Z1 is sampled from the random sample N(0,1); the latent variable Z1 is input into the decoder generation network d1(Z) to generate a virtual sample; finally, the standard image and the virtual image corresponding to the distorted image are respectively sent to the discriminator network D1, and the authenticity of the input virtual image is judged based on the discriminator network, and the judgment result of whether the distorted image and the standard image belong to the same sample is output.

[0068] The outstanding advantage of generative adversarial networks (GANs) is that its discriminator has a strong ability to distinguish the authenticity of samples. Variational autoencoders (VAEs) generate new data samples by learning the probability distribution of data. They can generate relatively clear data samples and the model training is relatively easy to converge. This application incorporates the excellent true and false discrimination ability of GANs into the abstract reconstruction error function of the VAE model for joint optimization. The generator network unit in GANs is replaced by VAE to construct a novel variational adversarial network.

[0069] S2: Construct a training dataset based on the distorted images and the standard images corresponding to each distorted image.

[0070] S3: Train the variational adversarial network model based on the training data set to obtain a trained variational adversarial network model.

[0071] S4: construct image correction unit;

[0072] The image correction unit includes: a feature extraction network and a trained decoder generation network;

[0073] The feature extraction network extracts features from the input image to be corrected, extracts the underlying edge features and generates a high-dimensional feature vector, which is then fed into the decoder generation network. The reconstructed data output by the decoder generation network that is similar to the original data is the corrected image.

[0074] In this embodiment, the feature extraction network is implemented based on a deep convolutional network model CNN.

[0075] After completing the training of the variational adversarial network model, an image correction unit is constructed based on the trained decoder generation network to perform large-scale correction on the AVM distorted image. Figure 5As shown in the figure, in the image correction stage, the distorted image is first input into the deep convolution-based feature extraction network CNNs to extract the underlying edge features of the image and generate a high-dimensional feature vector. Subsequently, the high-dimensional feature vector of the distorted image is sent to the decoder generation network in the network training stage. The model will correct the distorted features and generate a corrected image that is close to the standard image.

[0076] S5: sending the image collected by the on-board panoramic imaging system as the image to be corrected to the image correction unit in real time, and outputting the corresponding corrected image.

[0077] Among them, the overall loss function of the variational generative adversarial network is for:

[0078] ,

[0079] in, is the KL divergence of the pixel distribution between the input image and the sample, To combat the discriminative loss function, λ1 and λ2 are two empirical thresholds.

[0080] In the variational adversarial network model, the KL divergence of the pixel distribution between the input image and the sampled sample is calculated as:

[0081] ,

[0082] in, is the KL divergence of the pixel distribution between the input image and the sample; D KL is the relative entropy, which indicates the distance constraint between the real sample and the sampled sample; p(Z i 1) is the noise distribution, q(Z i 1|x i 1) The decoder takes the input low-dimensional code Z i 1 output; i represents the serial number of the sample sampled from the random sample N(0,1); Z i 1 is the variational adversarial network model that generates latent variables by sampling from random samples N(0,1);

[0083] In the variational adversarial network model, the loss function for generating virtual sample reconstruction error and discrimination is:

[0084] ,

[0085] in, To combat the discriminant loss function, G is the generator and D is the discriminator; x1 represents the input distorted image, D1(.) represents the output of the discriminator for the input data, d1(Z i 1) Generate network d1 for decoder for Zi 1 is the generated virtual sample.

[0086] The loss function used in the variational generative adversarial network training phase in this application consists of three parts: ① The pixel distribution similarity measure between the input distorted image and the randomly sampled sample, namely KL divergence. ② The reconstruction error between the generated virtual sample and the input distorted image. ③ The discrimination loss between the generated virtual sample and the input standard sample. The overall loss function is used. After completing the variational generative adversarial network training, the image correction unit composed of the encoder generation network d1(Z) and the feature extraction network CNNs can realize the migration of the input distorted image to the standard image pixel distribution domain, thereby correcting the AVM distorted image. By designing the variational loss function It can effectively solve the mode collapse (i.e., the difficulty in convergence) that occurs during the training and optimization process of VAE, ensuring that the variational adversarial network model in this application has both strong sample generation and discrimination capabilities and more abstract sample reconstruction capabilities.

[0087] Traditional image correction methods generally require the use of standard reference objects to calibrate the parameters of the camera, and use the internal and external parameters of the camera to correct the distorted image. This type of image distortion correction method has poor real-time performance, low correction accuracy, and insufficient generalization capabilities for complex scenes. This application optimizes the generator model for image distortion correction based on a deep learning method. Based on this model and an image decoder, distorted images can be corrected in real time, and the model has a strong scene generalization capability.

[0088] After using the technical solution of the present invention, the distorted image output by the vehicle-mounted camera can be corrected in real time, thereby improving the safety of intelligent driving.

Claims

1. A method for AVM image distortion correction based on cross-domain migration network, It is characterized in that It includes the following steps: S1: Combining the generative adversarial network model and the variational autoencoder network model to construct a variational adversarial network model; Replacing the generator network in the generative adversarial network model with a variational autoencoder network model to obtain the variational adversarial network model; The variational adversarial network model includes: an encoder inference network, a decoder generation network and a discriminator network connected in sequence; The encoder inference network unit compresses the input data into a low-dimensional code; the encoder inference network unit gradually learns the ability to extract non-distorted features of the input data samples by minimizing the sample reconstruction error and minimizing the KL divergence of the pixel distribution between the input image and the sampled sample; The decoder generates a network that decodes the low-dimensional code into reconstructed data similar to the original data; The discriminator network distinguishes between the virtual image generated by the decoder generation network and the input standard image; S2: Construct a training dataset based on the distorted images and the standard images corresponding to each distorted image; S3: Training the variational adversarial network model based on the training data set to obtain a trained variational adversarial network model; S4: construct image correction unit; The image correction unit includes: a feature extraction network and a trained decoder generation network; The feature extraction network extracts features from the input image to be corrected, extracts underlying edge features and generates a high-dimensional feature vector, and sends the high-dimensional feature vector to the decoder generation network. The decoder generation network outputs reconstructed data similar to the original data, which is the corrected image; S5: sending the image captured by the vehicle-mounted panoramic imaging system as the image to be corrected to the image correction unit in real time, and outputting the corresponding corrected image.

2. According to the AVM image distortion correction method based on cross-domain migration network described in claim 1, Features: The workflow in the variational adversarial network model is: Take the distorted image as input image x 1 Input into the encoder inference network of the variational generative adversarial network, using the encoder inference network e 1 (x) Extract the input image x 1 The expected pixel distribution μ 1 (x) and variance σ 1 (x); construct a random sample N(0,1) that conforms to the standard normal distribution, and the variational generative adversarial network calculates μ 1 (x) and the KL divergence between the sampled samples, and σ 1 (x) and the KL divergence between the sampled samples, after obtaining the constraint of the pixel distribution similarity between the input image and the sampled samples, the latent variable Z is generated by sampling from the random sample N(0,1) 1 ; The latent variable Z 1 Input to the decoder to generate the network d 1 (Z), a virtual sample is generated; finally, the standard image and the virtual image corresponding to the distorted image are sent to the discriminator network D 1 , based on the discriminator network, the authenticity of the input virtual image is judged, and the judgment result of whether the distorted image and the standard image belong to the same sample is output.

3. According to the AVM image distortion correction method based on cross-domain migration network as described in claim 2, Features: The overall loss function of the variational generative adversarial network is for: , in, is the KL divergence of the pixel distribution between the input image and the sample, To combat the discriminative loss function, λ 1 and λ 2 are two empirical thresholds.

4. According to claim 3, a method for correcting AVM image distortion based on a cross-domain migration network, Features: In the variational adversarial network model, the KL divergence of the pixel distribution between the input image and the sampled sample is calculated as follows: , in, is the KL divergence of the pixel distribution between the input image and the sample; D KL is the relative entropy, which indicates the distance constraint between the real sample and the sampled sample; p(Z i 1 ) is the noise distribution, q(Z i 1 |x i 1 ) is the decoder for the input low-dimensional code Z i 1 The output of; i represents the serial number of the sample sampled from the random sample N(0,1); Z i 1 Generate latent variables for the variational adversarial network model from random samples N(0,1).

5. According to the AVM image distortion correction method based on cross-domain migration network as described in claim 4, Features: In the variational adversarial network model, the loss function for generating virtual sample reconstruction error and discrimination is: , in, is the adversarial discriminant loss function, G is the generator, D is the discriminator; x 1 represents the input distorted image, D 1 (.) represents the output of the discriminator for the input data, d 1 (Z i 1 ) Generate network d for the decoder 1 For Z i 1 is the generated virtual sample.

6. According to claim 1, a method for AVM image distortion correction based on a cross-domain migration network, Features: The feature extraction network is implemented based on a deep convolutional network model CNN.