Panoramic image calibration method and system based on algorithm fusion

Abstract

The invention discloses a panoramic image calibration method and system based on algorithm fusion, and relates to the technical field of intelligent vehicles. The method comprises the following steps: acquiring a to-be-calibrated image sequence, and preprocessing the image sequence; constructing a parameter calibration model by utilizing an expert network, constructing a plurality of sub-networks according to different calibration algorithms, calibrating an image sequence by utilizing the sub-networks, scoring calibration results of all the sub-networks, and taking the image with the highest score as a preliminary calibration image; and constructing a parameter correction model by using a deep learning network, and performing parameter correction on the preliminary calibration image by using the parameter correction model according to other calibration results except the preliminary calibration image to obtain a final calibration image. According to the invention, through combination of the expert network model and the deep learning model, various calibration algorithms can be evaluated, the advantages of various algorithm calibration processes can be learned, and a more accurate panoramic image can be obtained.

Description

Technical Field

[0001] The present invention relates to the field of intelligent vehicle technology, and in particular to a panoramic image calibration method and system based on algorithm fusion. Background Art

[0002] The statements in this section merely provide background information related to the present invention and do not necessarily constitute prior art.

[0003] With the development of ADAS and autonomous driving technology, the field of view limitations of a single camera have prompted automakers to adopt multi-camera systems, especially camera arrays arranged around the vehicle body, to achieve all-round and all-around environmental perception. The panoramic imaging system stitches and fuses images collected by multiple cameras to form a continuous and seamless panoramic view, providing drivers with more intuitive and comprehensive driving environment information, which is crucial for application scenarios such as parking assistance, blind spot monitoring, panoramic parking, and traffic condition prediction.

[0004] Vehicle camera calibration is the basis for ensuring the accuracy and reliability of the vehicle's advanced driver assistance system (ADAS), autonomous driving system, and other visual perception functions. It involves determining the position and orientation of the vehicle camera relative to the vehicle coordinate system (external parameters), and determining the mathematical model of the camera's internal imaging characteristics (internal parameters).

[0005] The inventors found that the errors caused when determining the internal and external parameters will not only increase the complexity of image stitching, but also affect the accuracy of the panoramic view obtained after stitching. However, there are many calibration algorithms at present, and each has its own advantages and disadvantages. In the process of calculating the internal and external parameters, the use of a single optimization model has certain limitations, and its effectiveness cannot be evaluated, resulting in uneven quality of panoramic images. Therefore, how to integrate multiple calibration algorithms to achieve high-precision calibration of panoramic images has become a problem that needs to be solved urgently in the prior art. Summary of the invention

[0006] In view of the shortcomings of the prior art, the purpose of the present invention is to provide a panoramic image calibration method and system based on algorithm fusion. By combining the expert network model and the deep learning model, it can not only evaluate a variety of calibration algorithms, but also learn the advantages of the calibration process of various algorithms, so as to obtain more accurate panoramic images.

[0007] In order to achieve the above object, the present invention is implemented through the following technical solutions:

[0008] A first aspect of the present invention provides a panoramic image calibration method based on algorithm fusion, comprising the following steps:

[0009] Obtain an image sequence to be calibrated and preprocess the image sequence;

[0010] The parameter calibration model is constructed using an expert network. Multiple sub-networks are constructed according to different calibration algorithms. The sub-networks are used to calibrate the image sequences respectively. The calibration results of all sub-networks are scored, and the image with the highest score is used as the preliminary calibration image.

[0011] A parameter correction model is constructed using a deep learning network. The parameter correction model is used to perform parameter correction on the preliminary calibration image according to the remaining calibration results except the preliminary calibration image to obtain the final calibration image.

[0012] Furthermore, preprocessing the image sequence includes image cropping, color conversion, normalization and noise reduction operations.

[0013] Furthermore, the parameter calibration model consists of a gating network and three sub-networks. The three sub-networks use different calibration algorithms to calibrate the image sequence. The gating network scores according to the size of the calibration error, and the image with the highest score is used as the preliminary calibration image.

[0014] Furthermore, the three sub-networks are respectively a sub-network composed of an adversarial learning algorithm, a sub-network composed of a graph neural network, and a sub-network composed of a grey wolf genetic algorithm.

[0015] Furthermore, the specific steps for scoring the calibration results of all sub-networks are as follows:

[0016] The gating network receives the calibration results and corresponding errors of the three sub-networks.

[0017] The gating network uses a softmax layer to calculate the confidence of each sub-network as the scoring result.

[0018] Furthermore, the training steps of the parameter correction model are:

[0019] Collect image sequences of scene contents and diverse camera posture changes at different calibration stations to form a data set, and label the image sequences according to the internal and external parameters of the camera;

[0020] The dataset is expanded using the output images and parameters of each sub-network in the expert network;

[0021] Divide the expanded data set into training set and test set;

[0022] The deep learning network model is trained using the training set, and the training results are tested using the test set to obtain the trained parameter correction model.

[0023] A second aspect of the present invention provides a panoramic image calibration system based on algorithm fusion, comprising:

[0024] A data acquisition module is configured to acquire an image sequence to be calibrated and pre-process the image sequence;

[0025] The parameter calibration module is configured to construct a parameter calibration model using an expert network, construct multiple sub-networks according to different calibration algorithms, calibrate the image sequences using the sub-networks respectively, score the calibration results of all sub-networks, and use the image with the highest score as the preliminary calibration image;

[0026] The parameter correction module is configured to use a deep learning network to build a parameter correction model, and use the parameter correction model to perform parameter correction on the preliminary calibration image according to the remaining calibration results except the preliminary calibration image to obtain a final calibration image.

[0027] A third aspect of the present invention provides a medium having a program stored thereon, which, when executed by a processor, implements the steps in the panoramic image calibration method based on algorithm fusion as described in the first aspect of the present invention.

[0028] The fourth aspect of the present invention provides a device, including a memory, a processor, and a program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps in the panoramic image calibration method based on algorithm fusion as described in the first aspect of the present invention are implemented.

[0029] A fifth aspect of the present invention provides a computer program product, including a computer program, which, when executed by a processor, implements the steps in the panoramic image calibration method based on algorithm fusion as described in the first aspect of the present invention.

[0030] One or more of the above technical solutions have the following beneficial effects:

[0031] The present invention discloses a panoramic image calibration method and system based on algorithm fusion. In order to solve the problem of large calibration error caused by a single optimization model in the prior art, the present invention adopts an expert network combined with a deep learning model, adopts multiple calibration algorithms for fusion, and learns the advantages of different algorithms. After screening and weight distribution by the expert network, it is used as a reference for deep learning model learning, which greatly reduces the amount of calculation, does not blindly learn multiple algorithms, and improves the calibration efficiency.

[0032] Advantages of additional aspects of the present invention will be given in part in the following description, and in part will become obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] The accompanying drawings in the specification, which constitute a part of the present invention, are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.

[0034] Figure 1 This is a flow chart of a panoramic image calibration method based on algorithm fusion in Embodiment 1 of the present invention. DETAILED DESCRIPTION

[0035] It should be noted that the following detailed descriptions are exemplary and are intended to provide further explanation of the present invention. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present invention belongs.

[0036] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit the exemplary embodiments according to the present invention. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "include" and / or "include" are used in this specification, it indicates the presence of features, steps, operations, devices, components and / or their combinations;

[0037] Embodiment 1:

[0038] Embodiment 1 of the present invention provides a panoramic image calibration method based on algorithm fusion, such as Figure 1 As shown, the following steps are included:

[0039] Step 1: Obtain the image sequence to be calibrated and preprocess the image sequence.

[0040] In a specific implementation, preprocessing the image sequence includes performing image cropping, color conversion, normalization, and noise reduction operations on the images.

[0041] Specifically, resizing images to a uniform size and converting images from RGB to grayscale or black and white can simplify analysis and reduce noise. Using techniques such as Gaussian blur, median blur, and bilateral filtering can reduce noise and smooth images. Normalizing pixel values ​​to a standard range, such as 0 to 1 or -1 to 1, helps algorithms work better.

[0042] Step 2: Use the expert network to build a parameter calibration model, build multiple sub-networks according to different calibration algorithms, use the sub-networks to calibrate the image sequences respectively, score the calibration results of all sub-networks, and use the image with the highest score as the preliminary calibration image.

[0043] Step 2.1: Use the expert network to build a parameter calibration model and construct multiple sub-networks based on different calibration algorithms.

[0044] In a specific implementation, the parameter calibration model consists of a gating network and three sub-networks. The three sub-networks calibrate the image sequence using different calibration algorithms. The gating network scores the image according to the calibration error, and the image with the highest score is used as the preliminary calibration image.

[0045] In this embodiment, the three sub-networks are respectively a sub-network composed of an adversarial learning algorithm, a sub-network composed of a graph neural network, and a sub-network composed of a gray wolf genetic algorithm.

[0046] Step 2.2: Use the sub-networks to calibrate the image sequences separately.

[0047] Step 2.2.1: First sub-network, this embodiment uses an adversarial network as the first sub-network. The adversarial network includes a generator and a discriminator, wherein the generator is used to generate data close to the real calibration parameters, and the discriminator is used to distinguish the generated calibration parameters from the real calibration parameters.

[0048] Generator structure: Input layer: accepts random noise as input, which can be generated by a random number generator.

[0049] Hidden layers: A series of convolutional layers and activation functions to learn the mapping from noise to calibration parameters.

[0050] Output layer: Output predicted calibration parameters, such as camera intrinsic parameter matrix, distortion coefficient, etc.

[0051] Discriminator structure: Input layer: accepts the calibration parameters and true calibration parameters output by the generator.

[0052] Hidden layers: A series of convolutional layers and activation functions to learn features that distinguish true from false calibration parameters.

[0053] Output layer: Output a probability value, indicating the possibility that the input calibration parameter is the true calibration parameter.

[0054] The training process for the first sub-network is:

[0055] Data preparation: Collect calibration plate images and their corresponding true calibration parameters, which will be used as datasets for training GAN.

[0056] Adversarial training: Train the generator to generate more and more realistic calibration parameters, and train the discriminator to distinguish between true and false calibration parameters.

[0057] Through adversarial training, the generator learns to generate more accurate calibration parameters, while the discriminator provides a feedback mechanism to guide the training of the generator.

[0058] Generator network optimization: Use the trained generator network to predict calibration parameters for new calibration images.

[0059] The generated calibration parameters are compared with the true calibration parameters, the loss function is calculated, and the parameters of the generator network are optimized.

[0060] Discriminator network optimization: The discriminator network is trained using the calibration parameters generated by the generator and the true calibration parameters.

[0061] Through the feedback of the discriminator network, the generator network is further optimized to improve the accuracy of the calibration parameters.

[0062] Model evaluation and optimization: Evaluate model performance on the validation set and measure the calibration effect through indicators such as reprojection error and calibration accuracy.

[0063] Iteratively optimize the model, adjust hyperparameters, increase training data, etc.

[0064] Step 2.2.2: Second sub-network, this embodiment uses the gray wolf genetic algorithm (GWO) to construct the second sub-network. The gray wolf group has a clear hierarchy, and the GWO algorithm simulates this hierarchy, dividing the individuals in the population into α (Alpha, leader), β (Beta, deputy leader), δ (Delta, followers) and ω (Omega, ordinary members).

[0065] The Grey Wolf Genetic Algorithm consists of the following steps:

[0066] Initialize the population: randomly generate a certain number of gray wolves (solutions), each of which represents a potential solution, that is, a set of calibration parameters.

[0067] Fitness evaluation: Calculate the fitness value of each gray wolf, which is usually related to the calibration error. The smaller the error, the higher the fitness value.

[0068] Update the leader wolf: According to the fitness value, select three wolves α, β, and δ, which represent the three best solutions in the current population.

[0069] Update the position of the wolf pack: Other wolves (ω wolves) update their own positions according to the positions of α, β, and δ wolves, simulating the encirclement and attack behavior of the wolf pack when hunting.

[0070] In this embodiment, the specific steps of using the gray wolf genetic algorithm to perform image calibration are:

[0071] Initialization parameters: set parameters such as population size, number of iterations, and convergence accuracy.

[0072] Initialize the gray wolf positions: Randomly initialize the positions of gray wolf individuals according to the upper and lower bounds of the variables. These positions represent candidate calibration parameters.

[0073] Calculate the fitness value: Calculate the fitness value of each wolf (solution), evaluate the quality of the solution, and save the best, second best and third best solutions as α, β, and δ wolves respectively.

[0074] Position update: Update the position of each ω wolf based on the position information of α, β, δ wolves and the values ​​of parameters a, A, and C. The position update reflects the hunting behavior of the gray wolf approaching the leader wolf.

[0075] Among them, parameter a is a control factor, which decreases linearly from 2 to 0 during the iteration process. This parameter affects the search behavior of the search agent (grey wolf) for the location of the prey (optimal solution). As the value of a decreases, the gray wolf gradually narrows the search range and approaches the optimal solution. Therefore, a simulates the behavior of the wolf pack approaching the prey. Parameter A is a coefficient used to adjust the distance when the gray wolf searches for the prey location. The value of A is calculated by the formula A=2a·r1-a, where r1 is a random number in the range [0,1]. The value of A determines the search range of the gray wolf during the search process. When |A|>1, the gray wolf tends to disperse the search, while when |A|<1, the gray wolf concentrates on searching a certain area. Parameter C is a random weight used to represent the randomness of the impact of the wolf's location on the prey. The value of C is calculated by the formula C=2·r2, where r2 is a random number in the range [0,1]. The randomness of C helps the algorithm avoid falling into local optimality and enhances the algorithm's global search ability, robustness and convergence.

[0076] These parameters work together to enable the gray wolf optimization algorithm to effectively simulate the social behavior and hunting strategies of gray wolves to find the optimal solution to the problem.

[0077] Parameter update: As the iterations proceed, the value of parameter a is gradually reduced to simulate the behavior of the gray wolf gradually approaching its prey during hunting, and A and C are updated according to the value of parameter a.

[0078] Iterative optimization: Repeat the above fitness calculation, position update, and parameter update steps until the maximum number of iterations is reached or other termination conditions are met.

[0079] Output optimal solution: Output the position of α wolf as the optimal solution, that is, the best calibration parameter.

[0080] Through the above structure and steps, the improved gray wolf genetic algorithm can effectively search for the optimal camera calibration parameters and improve the calibration accuracy. This method is particularly suitable for complex optimization problems because it has strong global search capabilities and fast convergence speed.

[0081] Step 2.2.3: The third sub-network. In this embodiment, a graph neural network is used to construct the third sub-network.

[0082] At the calibration station, a special device such as a laser is used to project a light pattern of known shape, size or coded information onto the ground of the calibration site. These patterns can be regular grids (such as Gray code stripes, two-dimensional orthogonal stripes), randomly distributed speckles, or other designed coded patterns.

[0083] The images captured by the camera are processed using a graph convolutional neural network to extract key information from the image and identify and extract the key features of the projected pattern. The initial parameters are corrected using the extracted key feature information and the known physical properties of the projection (such as fringe spacing, coding meaning, etc.), especially to compensate for specific types of calibration errors (such as lens distortion, nonlinear distortion, temperature dependence effects, etc.). Combining multiple sets of data, the internal and external parameters of the camera and the three-dimensional information of the scene are inferred.

[0084] Step 2.3: Score the calibration results of all sub-networks.

[0085] Step 2.3.1: The gating network receives the calibration results and corresponding errors of the three sub-networks.

[0086] Step 3.3.2: The gating network uses the softmax layer to calculate the confidence of each sub-network as the scoring result.

[0087] The gating network receives the calibration results from each sub-network as input. These input features may include calibration errors, calculated position deviations, and other parameters that may affect the calibration quality.

[0088] The gated network contains learnable weight parameters, which are optimized by the back-propagation algorithm during the training process. The weight parameters determine the contribution of each sub-network to the final decision.

[0089] The softmax layer in the gating network converts the input features into a probability distribution, and each output represents the confidence or score of the corresponding sub-network. The output of the softmax layer is a probability distribution, and each probability value represents the score of the corresponding sub-network. These scores reflect the gating network's assessment of the quality of the calibration results of each sub-network.

[0090] According to the probability distribution of the softmax layer output, the sub-network calibration result with the highest score is selected as the preliminary calibration image. This means that the sub-network with the highest confidence is considered to be the network that provides the most reliable calibration result.

[0091] It should be noted that the learning parameters of the gating network and the confidence of all sub-networks are retained and used as a reference for the deep learning model to learn different calibration algorithms. Algorithms with large contributions will be studied in detail.

[0092] Step 3: Use the deep learning network to build a parameter correction model, and use the parameter correction model to perform parameter correction on the preliminary calibration image according to the remaining calibration results except the preliminary calibration image to obtain the final calibration image.

[0093] Step 3.1: Use the deep learning network to build a parameter correction model. The deep learning network includes: Input layer: The input is the features of the calibration plate image. Hidden layer: Use multiple convolutional layers and pooling layers to extract image features. Output layer: The output layer is a fully connected layer that outputs the camera's internal and external parameters. Loss function: Use mean square error (MSE) as the loss function to optimize the network weights.

[0094] Step 3.2: Use the parameter correction model to perform parameter correction on the preliminary calibration image according to the remaining calibration results except the preliminary calibration image.

[0095] Step 3.2.1: Collect image sequences of scene contents and diverse camera pose changes at different calibration stations to form a dataset, and label the image sequences according to the internal and external parameters of the camera.

[0096] Step 3.2.2: Preprocess the collected image sequences, including normalization and enhancement.

[0097] Step 3.2.3: Use the collected dataset to train the deep learning model and optimize the network parameters.

[0098] The dataset is expanded using the output images and parameters of each sub-network in the expert network, including the parameters and confidence of the output images of each sub-network retained.

[0099] Divide the expanded data set into training set and test set;

[0100] The deep learning network model is trained using the training set, and the training results are tested using the test set to obtain the trained parameter correction model.

[0101] Through the above steps, the model can combine the advantages of different calibration algorithms to achieve accurate camera calibration.

[0102] Embodiment 2:

[0103] Embodiment 2 of the present invention provides a panoramic image calibration system based on algorithm fusion, including:

[0104] A data acquisition module is configured to acquire an image sequence to be calibrated and pre-process the image sequence;

[0105] The parameter calibration module is configured to construct a parameter calibration model using an expert network, construct multiple sub-networks according to different calibration algorithms, calibrate the image sequences using the sub-networks respectively, score the calibration results of all sub-networks, and use the image with the highest score as the preliminary calibration image;

[0106] The parameter correction module is configured to use a deep learning network to build a parameter correction model, and use the parameter correction model to perform parameter correction on the preliminary calibration image according to the remaining calibration results except the preliminary calibration image to obtain a final calibration image.

[0107] Embodiment three:

[0108] Embodiment 3 of the present invention provides a medium on which a program is stored. When the program is executed by a processor, the steps in the panoramic image calibration method based on algorithm fusion as described in Embodiment 1 of the present invention are implemented.

[0109] Embodiment 4:

[0110] Embodiment 4 of the present invention provides a device, including a memory, a processor, and a program stored in the memory and executable on the processor. When the processor executes the program, the steps in the panoramic image calibration method based on algorithm fusion as described in Embodiment 1 of the present invention are implemented.

[0111] Embodiment five:

[0112] Embodiment 5 of the present invention provides a computer program product, including a computer program, which, when executed by a processor, implements the steps in the panoramic image calibration method based on algorithm fusion as described in Embodiment 1 of the present invention.

[0113] The steps involved in the apparatuses of the above embodiments 2, 3, 4 and 5 correspond to the method embodiment 1, and the specific implementation methods can refer to the relevant description part of embodiment 1. The term "computer-readable storage medium" should be understood as a single medium or multiple media including one or more instruction sets; it should also be understood to include any medium that can store, encode or carry an instruction set for execution by a processor and enable the processor to execute any method of the present invention.

[0114] Those skilled in the art should understand that the modules or steps of the present invention described above can be implemented by a general-purpose computer device, or alternatively, they can be implemented by a program code executable by a computing device, so that they can be stored in a storage device and executed by the computing device, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. The present invention is not limited to any specific combination of hardware and software.

[0115] Although the above describes the specific implementation mode of the present invention in conjunction with the accompanying drawings, it is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art on the basis of the technical solution of the present invention without creative work are still within the scope of protection of the present invention.

Claims

1. A panoramic image calibration method based on algorithm fusion, characterized in that: The following steps are involved: Obtain an image sequence to be calibrated and preprocess the image sequence; The parameter calibration model is constructed using an expert network. Multiple sub-networks are constructed according to different calibration algorithms. The sub-networks are used to calibrate the image sequences respectively. The calibration results of all sub-networks are scored, and the image with the highest score is used as the preliminary calibration image. A parameter correction model is constructed using a deep learning network. The parameter correction model is used to perform parameter correction on the preliminary calibration image according to the remaining calibration results except the preliminary calibration image to obtain the final calibration image.

2. The panoramic image calibration method based on algorithm fusion according to claim 1, characterized in that: The preprocessing of image sequences includes image cropping, color conversion, normalization and noise reduction operations.

3. The panoramic image calibration method based on algorithm fusion according to claim 1, characterized in that: The parameter calibration model consists of a gating network and three sub-networks. The three sub-networks use different calibration algorithms to calibrate the image sequence. The gating network scores according to the calibration error, and the image with the highest score is used as the preliminary calibration image.

4. The panoramic image calibration method based on algorithm fusion according to claim 3, characterized in that: The three sub-networks are the sub-network composed of the adversarial learning algorithm, the sub-network composed of the graph neural network, and the sub-network composed of the gray wolf genetic algorithm.

5. The panoramic image calibration method based on algorithm fusion according to claim 3, characterized in that: The specific steps for scoring the calibration results of all sub-networks are: The gating network receives the calibration results and corresponding errors of the three sub-networks; The gating network uses a softmax layer to calculate the confidence of each sub-network as the scoring result.

6. The panoramic image calibration method based on algorithm fusion according to claim 1, characterized in that: The training steps of the parameter correction model are: Collect image sequences of scene contents and diverse camera posture changes at different calibration stations to form a data set, and label the image sequences according to the internal and external parameters of the camera; The dataset is expanded using the output images and parameters of each sub-network in the expert network; Divide the expanded data set into training set and test set; The deep learning network model is trained using the training set, and the training results are tested using the test set to obtain the trained parameter correction model.

7. A panoramic image calibration system based on algorithm fusion, characterized in that: include: A data acquisition module is configured to acquire an image sequence to be calibrated and pre-process the image sequence; The parameter calibration module is configured to construct a parameter calibration model using an expert network, construct multiple sub-networks according to different calibration algorithms, calibrate the image sequences using the sub-networks respectively, score the calibration results of all sub-networks, and use the image with the highest score as the preliminary calibration image; The parameter correction module is configured to use a deep learning network to build a parameter correction model, and use the parameter correction model to perform parameter correction on the preliminary calibration image according to the remaining calibration results except the preliminary calibration image to obtain a final calibration image.

8. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the panoramic image calibration method based on algorithm fusion described in any one of claims 1 to 6 is implemented.

9. A computer-readable storage medium, characterized in that: A plurality of instructions are stored therein, and the instructions are suitable for being loaded by a processor of a terminal device and executing the panoramic image calibration method based on algorithm fusion described in any one of claims 1 to 6.

10. A terminal device, characterized in that: It includes a processor and a computer-readable storage medium, the processor is used to implement each instruction; the computer-readable storage medium is used to store multiple instructions, and the instructions are suitable for being loaded by the processor and executing the panoramic image calibration method based on algorithm fusion described in any one of claims 1-6.

Images