A panoramic image calibration method and system based on algorithm fusion

Abstract

The application discloses a kind of panoramic image calibration method and system based on algorithm fusion, it is related to intelligent vehicle technical field.The method includes the following steps: obtaining the image sequence to be calibrated, pre-processing image sequence;Parameter calibration model is constructed using expert network, according to different calibration algorithm, multiple sub-networks are constructed, and the image sequence is calibrated using sub-network respectively, the calibration results of all sub-networks are scored, and the image with the highest score is used as the preliminary calibration image;Parameter correction model is constructed using deep learning network, and the preliminary calibration image is corrected according to the remaining calibration results except the preliminary calibration image using parameter correction model, to obtain the final calibration image.The application combines expert network model and deep learning model, not only can evaluate multiple calibration algorithms, but also can learn the advantages in the calibration process of multiple algorithms, and can obtain more accurate panoramic image.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of intelligent vehicles, in particular to a panoramic image calibration method and system based on algorithm fusion. BACKGROUND

[0002] The statements in this section merely provide background information related to the present application and do not necessarily constitute prior art.

[0003] With the development of ADAS and autonomous driving technology, the limited field of view of a single camera prompts automobile manufacturers to adopt a multi-camera system, especially a camera array arranged around the vehicle body, to achieve all-around and dead-angle-free environmental perception. The panoramic image system forms a continuous and seamless panoramic view by splicing and fusing the images collected by multiple cameras, providing more intuitive and comprehensive driving environment information for the driver, which is crucial for application scenarios such as parking assistance, blind area monitoring, panoramic parking, and traffic condition prediction.

[0004] Vehicle camera calibration is the basic work to ensure the accuracy and reliability of advanced driver assistance systems (ADAS), autonomous driving systems, and other visual perception functions. It involves determining the position and orientation of the vehicle-mounted camera relative to the vehicle coordinate system (external parameters) and determining the mathematical model of the camera's internal imaging characteristics (internal parameters).

[0005] The inventor found that the errors caused in determining internal and external parameters not only increase the complexity of image splicing, but also affect the accuracy of the panoramic view obtained after splicing. However, there are many calibration algorithms at present, each with its own advantages and disadvantages. Using a single optimization model in the calculation process of internal and external parameters has certain limitations and cannot evaluate its effectiveness, resulting in uneven quality of panoramic images. Therefore, how to fuse multiple calibration algorithms to achieve high-precision calibration of panoramic images has become a problem to be solved in the prior art. SUMMARY

[0006] In view of the deficiencies in the prior art, the present application aims to provide a panoramic image calibration method and system based on algorithm fusion, which combines expert network models and deep learning models, not only can evaluate multiple calibration algorithms, but also can learn the advantages of multiple algorithm calibration processes, and can obtain more accurate panoramic images.

[0007] To achieve the above-mentioned purpose, the present application is realized by the following technical solutions:

[0008] The present application provides a panoramic image calibration method based on algorithm fusion, comprising the following steps:

[0009] Obtain the image sequence to be calibrated, and pre-process the image sequence;

[0010] The parameter calibration model is constructed by using an expert network, a plurality of sub-networks are constructed according to different calibration algorithms, the image sequence is calibrated by using the sub-networks respectively, the calibration results of all the sub-networks are scored, and the image with the highest score is taken as the preliminary calibration image;

[0011] The parameter correction model is constructed by using a deep learning network, the preliminary calibration image is corrected according to the remaining calibration results except the preliminary calibration image by using the parameter correction model, and the final calibration image is obtained.

[0012] Further, the pre-processing of the image sequence includes image cropping, color conversion, normalization and noise reduction operations on the image.

[0013] Further, the parameter calibration model is composed of a gating network and three sub-networks, the three sub-networks respectively calibrate the image sequence by using different calibration algorithms, and the gating network scores according to the error size of the calibration, and the image with the highest score is taken as the preliminary calibration image.

[0014] Further, the three sub-networks are respectively a sub-network composed of an adversarial learning algorithm, a sub-network composed of a graph neural network and a sub-network composed of a grey wolf genetic algorithm.

[0015] Further, the specific steps of scoring the calibration results of all the sub-networks are as follows:

[0016] The gating network receives the calibration results and the corresponding errors of the three sub-networks.

[0017] The gating network calculates the confidence of each sub-network by using a softmax layer as a scoring result.

[0018] Further, the training steps of the parameter correction model are as follows:

[0019] Image sequences of different calibration station scene contents and diversified camera pose changes are collected to form a data set, and the image sequences are labeled according to the internal and external parameters of the camera;

[0020] The data set is expanded by using the output image and the parameters of each sub-network in the expert network;

[0021] The expanded data set is divided into a training set and a test set;

[0022] The deep learning network model is trained by using the training set, the training result is tested by using the test set, and the trained parameter correction model is obtained.

[0023] The second aspect of the present application provides a panoramic image calibration system based on algorithm fusion, comprising:

[0024] The data acquisition module is configured to acquire an image sequence to be calibrated and pre-process the image sequence;

[0025] The parameter calibration module is configured to construct a parameter calibration model by using an expert network, construct multiple sub-networks according to different calibration algorithms, calibrate the image sequence by using the sub-networks respectively, score the calibration results of all the sub-networks, and take the image with the highest score as a preliminary calibration image.

[0026] The parameter correction module is configured to construct a parameter correction model by using a deep learning network, correct the parameters of the preliminary calibration image according to the remaining calibration results except the preliminary calibration image by using the parameter correction model, and obtain a final calibration image.

[0027] The third aspect of the present application provides a medium having a program stored thereon, which, when executed by a processor, implements the steps in the panoramic image calibration method based on algorithm fusion according to the first aspect of the present application.

[0028] The fourth aspect of the present application provides a device including a memory, a processor, and a program stored on the memory and executable on the processor, wherein the processor implements the steps in the panoramic image calibration method based on algorithm fusion according to the first aspect of the present application when executing the program.

[0029] The fifth aspect of the present application provides a computer program product including a computer program, which, when executed by a processor, implements the steps in the panoramic image calibration method based on algorithm fusion according to the first aspect of the present application.

[0030] The above one or more technical solutions have the following beneficial effects:

[0031] The present application discloses a panoramic image calibration method and system based on algorithm fusion, in order to solve the problem of large calibration error caused by a single optimization model in the prior art, the present application adopts the way of expert network combined with deep learning model, adopts multiple calibration algorithms for fusion, and learns the advantages between different algorithms. After the screening and weight distribution of the expert network, as a reference for the deep learning model learning, the calculation amount is greatly reduced, and the learning of multiple algorithms is not blind, and the calibration efficiency is improved.

[0032] The advantages of the additional aspects of the present application will be partially given in the following description, partially will become obvious from the following description, or will be learned by the practice of the present application. BRIEF DESCRIPTION OF DRAWINGS

[0033] The drawings accompanying the specification of the present application form a part thereof, serve to provide further understanding of the present application, and together with the description of the exemplary embodiments of the present application and their description serve to explain the present application, and do not constitute improper limitations on the present application.

[0034] Figure 1 Figure 1 is a flowchart of a panoramic image calibration method based on algorithm fusion according to an embodiment of the present application. DETAILED DESCRIPTION

[0035] It should be noted that the following detailed description is exemplary in nature and is intended to provide further description of the application. Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs.

[0036] It is to be understood that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of example embodiments in accordance with the present application. As used herein, the singular forms "a", "an" and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms "comprises" and / or "comprising," when used in this specification, specify the presence of stated features, steps, operations, elements, components, and / or groups thereof, but do not preclude the presence or addition of one or more other features, steps, operations, elements, components, and / or groups thereof.

[0037] Embodiment One

[0038] Embodiment One of the present application provides a panoramic image calibration method based on algorithm fusion, as shown in Figure 1, comprising the following steps: Figure 1

[0039] Step 1: Obtain the image sequence to be calibrated, and pre-process the image sequence.

[0040] In a specific embodiment, pre-processing the image sequence includes image cropping, color conversion, normalization, and noise reduction operations.

[0041] Specifically, the image is adjusted to a uniform size, and the image is converted from RGB to grayscale or black and white, which can simplify the analysis and reduce noise. Using techniques such as Gaussian blur, median blur, and bilateral filtering can reduce noise and smooth the image. Normalizing the pixel values to a standard range, such as 0 to 1 or -1 to 1, helps the algorithm work better.

[0042] Step 2: Construct a parameter calibration model using an expert network, construct multiple sub-networks according to different calibration algorithms, use the sub-networks to calibrate the image sequence respectively, score the calibration results of all sub-networks, and the image with the highest score is the preliminary calibration image.

[0043] Step 2.1: Construct a parameter calibration model using an expert network, and construct multiple sub-networks according to different calibration algorithms.

[0044] ​In a specific embodiment, the parameter calibration model is composed of a gating network and three sub-networks. The three sub-networks respectively calibrate the image sequence using different calibration algorithms, and the gating network scores according to the calibration error size. The image with the highest score is the preliminary calibration image.

[0045] In this embodiment, the three sub-networks are respectively an adversarial learning algorithm sub-network, a graph neural network sub-network, and a grey wolf genetic algorithm sub-network.

[0046] Step 2.2: Calibrate the image sequence using the sub-networks respectively.

[0047] Step 2.2.1: First sub-network, this embodiment uses an adversarial network as the first sub-network. The adversarial network includes a generator and a discriminator. The generator is used to generate data close to the true calibration parameters, and the discriminator is used to distinguish between generated calibration parameters and true calibration parameters.

[0048] Generator structure: input layer: accepts random noise as input, which can be generated by a random number generator.

[0049] Hidden layer: a series of convolutional layers and activation functions, used to learn the mapping from noise to calibration parameters.

[0050] Output layer: output the predicted calibration parameters, such as camera intrinsic matrix, distortion coefficient, etc.

[0051] Discriminator structure: input layer: accepts the calibration parameters output by the generator and the true calibration parameters.

[0052] Hidden layer: a series of convolutional layers and activation functions, used to learn features that distinguish between true and false calibration parameters.

[0053] Output layer: output a probability value indicating the likelihood that the input calibration parameter is a true calibration parameter.

[0054] The training process for the first sub-network is as follows:

[0055] Data preparation: collect calibration board images and their corresponding true calibration parameters, which will be used as the training dataset for GAN.

[0056] Adversarial training: train the generator to generate increasingly realistic calibration parameters, while training the discriminator to distinguish between true and false calibration parameters.

[0057] Through adversarial training, the generator learns to generate more accurate calibration parameters, while the discriminator provides a feedback mechanism to guide the training of the generator.

[0058] Generator network optimization: Using the trained generator network, predict the calibration parameters for new calibration images.

[0059] Compare the generated calibration parameters with the true calibration parameters, calculate the loss function, and optimize the parameters of the generator network.

[0060] Discriminator network optimization: Train the discriminator network using the calibration parameters generated by the generator and the true calibration parameters.

[0061] Through the feedback of the discriminator network, further optimize the generator network to improve the accuracy of the calibration parameters.

[0062] Model evaluation and optimization: Evaluate the model performance on the validation set, measure the calibration effect through indicators such as re-projection error, calibration accuracy, etc.

[0063] Iterative optimization of the model, adjustment of hyperparameters, increase of training data, etc.

[0064] Step 2.2.2: Second sub-network, this embodiment uses the grey wolf genetic algorithm (GWO) to construct the second sub-network. The grey wolf population has a clear hierarchical system, and the GWO algorithm simulates this hierarchical system, dividing the individuals in the population into α (Alpha, leader), β (Beta, deputy leader), δ (Delta, follower), and ω (Omega, ordinary member).

[0065] The grey wolf genetic algorithm includes the following steps:

[0066] Initialize the population: randomly generate a certain number of grey wolves (solutions), each representing a potential solution, i.e. a set of calibration parameters.

[0067] Fitness evaluation: Calculate the fitness value of each grey wolf, which is usually related to the calibration error, the smaller the error, the higher the fitness value.

[0068] Update the alpha wolf: According to the fitness value, select α, β, and δ wolves, which represent the best three solutions in the current population.

[0069] Update the position of the wolf pack: other wolves (ω wolves) update their positions according to the positions of α, β, and δ wolves, simulating the surrounding and attacking behavior of the wolf pack during hunting.

[0070] In this embodiment, the specific steps of image calibration using the grey wolf genetic algorithm are as follows:

[0071] Initialize parameters: set the population size, iteration number, convergence precision, etc.

[0072] Initialize the position of the grey wolf: randomly initialize the position of the grey wolf individual according to the upper and lower bounds of the variables, which represents the candidate calibration parameters.

[0073] Calculate fitness value: Calculate the fitness value of each wolf (solution), evaluate the quality of the solution, and save the best, second best and third best solutions as α, β and δ wolves respectively.

[0074] Position Update: Based on the position information of α, β, and δ wolves, and the values ​​of parameters a, A, and C, the position of each ω wolf is updated. This position update reflects the gray wolves' hunting behavior of moving closer to the alpha wolf.

[0075] Here, parameter 'a' is a control factor that decreases linearly from 2 to 0 during iteration. This parameter affects the search behavior of the search agent (gray wolf) towards the location of the prey (optimal solution). As the value of 'a' decreases, the gray wolf gradually narrows its search range, approaching the optimal solution. Therefore, 'a' simulates the behavior of the wolf pack approaching its prey. Parameter A is a coefficient used to adjust the distance the gray wolf travels when searching for the prey's location. The value of A is given by the formula... The calculation shows that, among which A is a random number within the range [0,1]. The value of A determines the search range of the gray wolf during the search process. When |A|>1, the gray wolf tends to search in a dispersed manner, while when |A|<1, the gray wolf concentrates its search on a certain area. The parameter C is a random weight used to represent the randomness of the wolf's location's influence on its prey. The value of C is given by the formula... The calculation shows that, among which C is a random number within the range [0,1]. The randomness of C helps the algorithm avoid getting trapped in local optima and enhances the algorithm's global search capability, robustness, and convergence.

[0076] These parameters work together to enable the gray wolf optimization algorithm to effectively simulate the social behavior and hunting strategies of gray wolves in order to find the optimal solution to the problem.

[0077] Parameter update: As the iteration progresses, the value of parameter a is gradually decreased to simulate the behavior of a gray wolf gradually approaching its prey during the hunt, and A and C are updated according to the value of parameter a.

[0078] Iterative optimization: Repeat the fitness calculation, position update, and parameter update steps above until the maximum number of iterations is reached or other termination conditions are met.

[0079] Output the optimal solution: Output the position of wolf α as the optimal solution, that is, the best calibration parameters.

[0080] Through the above structure and steps, the improved gray wolf genetic algorithm can effectively search for optimal camera calibration parameters, thereby improving calibration accuracy. This method is particularly suitable for complex optimization problems because it has strong global search capabilities and a fast convergence speed.

[0081] Step 2.2.3: Third subnetwork, in this embodiment, a graph neural network is used to construct the third subnetwork.

[0082] At the calibration station, special equipment such as lasers are used to project light patterns of known shapes, sizes, or coded information onto the ground of the calibration site. These patterns can be regular grids (such as Gray code stripes, two-dimensional orthogonal stripes), randomly distributed speckles, or other designed coded patterns.

[0083] The images captured by the camera are processed using a graph convolutional neural network to extract key information and identify the key features of the projected patterns. Using the extracted key feature information and known physical properties of the projection (such as stripe spacing, coding meaning, etc.), the preliminary parameters are corrected, especially for specific types of calibration errors (such as lens distortion, nonlinear distortion, temperature-dependent effects, etc.). Combined with multiple sets of data, the camera's internal and external parameters and the scene's three-dimensional information are calculated.

[0084] Step 2.3: Score the calibration results of all subnetworks.

[0085] Step 2.3.1: The gating network receives the calibration results of the three subnetworks and the corresponding errors.

[0086] Step 3.3.2: The gating network uses the softmax layer to calculate the confidence of each subnetwork as the scoring result.

[0087] The gating network receives the calibration results from each subnetwork as input. These input features may include calibration errors, calculated position deviations, and other parameters that may affect the quality of calibration.

[0088] The gating network contains learnable weight parameters that are optimized during the training process through the backpropagation algorithm. The weight parameters determine the contribution of each subnetwork to the final decision.

[0089] The softmax layer in the gating network converts the input features into a probability distribution, with each output representing the confidence or score of the corresponding subnetwork. The output of the softmax layer is a probability distribution, with each probability value representing the score of the corresponding subnetwork. These scores reflect the gating network's evaluation of the quality of each subnetwork's calibration results.

[0090] According to the probability distribution output by the softmax layer, the subnetwork calibration result with the highest score is selected as the preliminary calibration image. This means that the subnetwork with the highest confidence is considered to be the network that provides the most reliable calibration results.

[0091] It should be particularly pointed out that the learning parameters of the gating network and the confidence of all sub-networks are reserved for reference in the learning process of different calibration algorithms of the deep learning model. Algorithms with high contribution will be focused on learning.

[0092] Step 3: Constructing a parameter correction model using a deep learning network, and correcting the parameters of the preliminary calibration image according to the remaining calibration results except the preliminary calibration image using the parameter correction model to obtain the final calibration image.

[0093] Step 3.1: Constructing a parameter correction model using a deep learning network. The deep learning network includes: an input layer: the input is the features of the calibration board image. Hidden layer: multiple convolutional layers and pooling layers are used to extract image features. Output layer: the output layer is a fully connected layer, which outputs the intrinsic and extrinsic parameters of the camera. Loss function: using mean square error (MSE) as the loss function to optimize the network weight.

[0094] Step 3.2: Correcting the parameters of the preliminary calibration image according to the remaining calibration results except the preliminary calibration image using the parameter correction model.

[0095] Step 3.2.1: Collecting image sequences of different calibration station scenes and diversified camera pose changes to form a data set, and labeling the image sequences according to the intrinsic and extrinsic parameters of the camera.

[0096] Step 3.2.2: Preprocessing the collected image sequences, including normalization and enhancement.

[0097] Step 3.2.3: Training the deep learning model using the collected data set to optimize the network parameters.

[0098] The data set is expanded using the output image and parameters of each sub-network in the expert network, including the parameters and confidence of each sub-network output image retained.

[0099] The expanded data set is divided into a training set and a test set;

[0100] The training set is used to train the deep learning network model, and the test set is used to test the training result to obtain the trained parameter correction model.

[0101] Through the above steps, the model can combine the advantages of different calibration algorithms to realize accurate calibration of the camera.

[0102] Embodiment two:

[0103] The embodiment two of the present application provides a panoramic image calibration system based on algorithm fusion, comprising:

[0104] The data acquisition module is configured to acquire an image sequence to be calibrated and preprocess the image sequence;

[0105] The parameter calibration module is configured to construct a parameter calibration model by using an expert network, construct multiple sub-networks according to different calibration algorithms, calibrate the image sequence by using the sub-networks respectively, score the calibration results of all the sub-networks, and take the image with the highest score as a preliminary calibration image.

[0106] The parameter correction module is configured to construct a parameter correction model by using a deep learning network, correct the preliminary calibration image according to the remaining calibration results except the preliminary calibration image by using the parameter correction model, and obtain a final calibration image.

[0107] Embodiment three:

[0108] Embodiment three of the present application provides a medium having a program stored thereon, the program being executed by a processor to implement the steps in the algorithm fusion-based panoramic image calibration method according to embodiment one of the present application.

[0109] Embodiment four:

[0110] Embodiment four of the present application provides a device including a memory, a processor, and a program stored on the memory and executable on the processor, and the processor implements the steps in the algorithm fusion-based panoramic image calibration method according to embodiment one of the present application when executing the program.

[0111] Embodiment five:

[0112] Embodiment five of the present application provides a computer program product including a computer program, the computer program being executed by a processor to implement the steps in the algorithm fusion-based panoramic image calibration method according to embodiment one of the present application.

[0113] The steps involved in the above embodiments two, three, four and five correspond to the method embodiment one, and the specific embodiments can be referred to the relevant description part of embodiment one. The term "computer readable storage medium" should be understood as including a single medium or multiple media of one or more instruction sets; it should also be understood as including any medium capable of storing, encoding or carrying instruction sets for execution by a processor and causing the processor to perform any method in the present application.

[0114] Those skilled in the art should understand that each module or step of the present application described above can be realized by a general computer device, alternatively, they can be realized by program codes executable by a computing device, so that they can be stored in a storage device for execution by a computing device, or they can be respectively manufactured into individual integrated circuit modules, or a plurality of modules or steps among them can be manufactured into a single integrated circuit module to realize. The present application is not limited to any specific combination of hardware and software.

[0115] The above describes the specific embodiments of the present application in conjunction with the drawings, but is not intended to limit the scope of protection of the present application. It should be understood by those skilled in the art that various modifications or variations can be made to the technical solutions of the present application without creative efforts, and still fall within the scope of protection of the present application.

Claims

1. A panoramic image calibration method based on algorithm fusion, characterized in that, Includes the following steps: Obtain the image sequence to be calibrated and preprocess the image sequence; A parameter calibration model is constructed using an expert network. Multiple sub-networks are built according to different calibration algorithms. The sub-networks are used to calibrate the image sequence respectively. The calibration results of all sub-networks are scored, and the image with the highest score is used as the initial calibrated image. The parameter calibration model consists of a gating network and three sub-networks. The three sub-networks are a sub-network constructed by an adversarial learning algorithm, a sub-network constructed by a graph neural network, and a sub-network constructed by a gray wolf genetic algorithm. The specific steps for scoring the calibration results of all sub-networks are as follows: The gated network receives the calibration results and corresponding errors from the three sub-networks; Gated networks use softmax layers to calculate the confidence score of each sub-network as the scoring result; A parameter correction model is constructed using a deep learning network. The parameter correction model is then used to correct the parameters of the initial calibration image based on the calibration results other than the initial calibration image, thus obtaining the final calibration image.

2. The panoramic image calibration method based on algorithm fusion as described in claim 1, characterized in that, Preprocessing of image sequences includes image cropping, color conversion, normalization, and noise reduction.

3. The panoramic image calibration method based on algorithm fusion as described in claim 1, characterized in that, The image sequence is calibrated by three sub-networks using different calibration algorithms. The gating network scores the images according to the magnitude of the calibration error, and the image with the lowest score is used as the initial calibrated image.

4. The panoramic image calibration method based on algorithm fusion as described in claim 1, characterized in that, The training steps for the parameter correction model are as follows: A dataset was formed by collecting image sequences of scene content and diverse camera pose changes from different calibration positions, and the image sequences were labeled according to the camera's intrinsic and extrinsic parameters. The dataset is augmented using the output images and parameters of each subnetwork in the expert network; The expanded dataset is divided into a training set and a test set; The deep learning network model is trained using the training set, and the training results are tested using the test set to obtain the trained parameter correction model.

5. A panoramic image calibration system based on algorithm fusion, employing the panoramic image calibration method based on algorithm fusion as described in any one of claims 1-4, characterized in that, include: The data acquisition module is configured to acquire the image sequence to be calibrated and preprocess the image sequence; The parameter calibration module is configured to construct a parameter calibration model using an expert network, build multiple sub-networks according to different calibration algorithms, calibrate the image sequence using the sub-networks respectively, score the calibration results of all sub-networks, and use the image with the highest score as the initial calibration image. The parameter correction module is configured to build a parameter correction model using a deep learning network, and then use the parameter correction model to correct the parameters of the initial calibration image based on the calibration results other than the initial calibration image, so as to obtain the final calibration image.

6. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the panoramic image calibration method based on algorithm fusion as described in any one of claims 1-4.

7. A computer-readable storage medium, characterized in that, It stores multiple instructions, which are adapted to be loaded and executed by the processor of the terminal device according to any one of claims 1-4, the panoramic image calibration method based on algorithm fusion.

8. A terminal device, characterized in that, The system includes a processor and a computer-readable storage medium, wherein the processor implements various instructions; and the computer-readable storage medium stores multiple instructions adapted to be loaded by the processor and executed by the processor for the panoramic image calibration method based on algorithm fusion as described in any one of claims 1-4.

Images