An AI vision camera extrinsic parameter calibration system and method
By combining an environmental perception sensor array with a deep learning model, dynamic and accurate calibration of the extrinsic parameters of an AI vision camera is achieved, solving the problems of calibration accuracy and stability in complex environments. This technology is applicable to intelligent transportation and industrial automation.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- JIANGSU SHIHAI INTELLIGENT TECH CO LTD
- Filing Date
- 2025-06-16
- Publication Date
- 2026-07-07
AI Technical Summary
Existing AI vision camera extrinsic calibration methods cannot adapt to complex and ever-changing environments, cannot fully utilize multi-source data fusion, and cannot achieve high-precision dynamic calibration.
Environmental parameters are collected in real time by an environmental perception sensor array. The feature alignment algorithm is combined with deep fusion of image data. The camera extrinsic parameters are predicted by a multi-layer deep feature extraction network and an LSTM-based deep learning model. The camera is then self-calibrated through a feedback evaluation module to achieve dynamic and accurate calibration.
It significantly improves the accuracy and stability of camera extrinsic calibration, maintains high-precision calibration results in complex and ever-changing environments, continuously optimizes camera extrinsic parameters, and improves the accuracy and stability of imaging.
Smart Images
Figure CN120765759B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of camera extrinsic parameter calibration technology, and in particular to an AI vision camera extrinsic parameter calibration system and method. Background Technology
[0002] With the development of computer vision technology, AI vision camera extrinsic calibration technology has gradually become a research hotspot. Traditional calibration methods mainly rely on manual feature extraction and fixed model calculation, such as using checkerboard patterns to extract corner points for calibration. These methods have achieved certain results in stable environments, and related technologies have been widely used in fields such as intelligent driving and industrial inspection, and the research interest continues to rise.
[0003] However, existing technologies have many shortcomings. They cannot adapt to complex and ever-changing environments; changes in environmental parameters have a significant impact on the accuracy of external parameters, leading to unstable calibration results. Furthermore, traditional methods utilize only a single image data source, neglecting the value of multi-source data fusion, resulting in insufficient information. In addition, the lack of a feedback optimization mechanism means the model cannot self-improve based on errors, making it difficult to meet the requirements of high-precision dynamic calibration. Unlike this invention, it cannot achieve high-precision dynamic calibration by deeply fusing environmental and image data and introducing a feedback self-calibration mechanism. Summary of the Invention
[0004] In view of the aforementioned existing problems, the present invention is proposed.
[0005] Therefore, this invention provides an AI vision camera extrinsic parameter calibration method to solve the problems of existing AI vision camera extrinsic parameter calibration methods, such as their inability to adapt to complex environments, their inability to fully utilize multi-source data fusion, and how to achieve high-precision dynamic calibration.
[0006] To solve the above-mentioned technical problems, the present invention provides the following technical solution:
[0007] In a first aspect, the present invention provides an AI vision camera extrinsic parameter calibration method, characterized in that it includes:
[0008] S1: Environmental parameters are collected in real time at high frequency using an environmental sensing sensor array, and then the environmental feature vector is formed after preliminary filtering.
[0009] S2: Deeply fuse environmental feature vectors with image data captured by AI vision cameras, and accurately align them in the spatiotemporal dimensions using feature alignment algorithms to obtain fused data.
[0010] S3: Input the fused data into a multi-layer deep feature extraction network to extract deep feature representations and form a high-dimensional feature matrix.
[0011] S4: Input the high-dimensional feature matrix into the deep learning-based camera extrinsic prediction model to predict the optimal camera extrinsic parameters and generate a feedback signal through the feedback evaluation module for quantitative evaluation.
[0012] S5: Calculate the environmental adaptation coefficient based on the feedback signal and environmental feature vector, and use this coefficient to self-calibrate the deep learning camera extrinsic prediction model to obtain the calibrated model parameters.
[0013] S6: The calibrated model parameters are used to determine the camera extrinsic parameters, enabling the system to run in a loop and complete the dynamic and accurate calibration of the camera extrinsic parameters.
[0014] As a preferred embodiment of the AI vision camera extrinsic parameter calibration method of the present invention, the step of using an environmental perception sensor array to collect environmental parameters in real time at high frequency, and forming an environmental feature vector after preliminary filtering processing, specifically includes the following steps:
[0015] Light, temperature, humidity, and color temperature sensors are installed around the AI vision camera according to a preset geometric arrangement to form an orderly sensor layout;
[0016] Start the sensor array, set the high-frequency acquisition frequency, and each sensor acquires ambient light intensity, temperature, humidity and color temperature values in real time to generate a raw environmental parameter data sequence;
[0017] The Kalman filter algorithm is used to smooth the original environmental parameter data sequence in the time domain. The filter state parameters are initialized with each sensor data sequence as input. After recursive processing through prediction and update steps, the filtered environmental parameter estimates are obtained.
[0018] The filtered light intensity, temperature, humidity, and color temperature values are integrated according to a predefined data structure to form an environmental feature vector.
[0019] As a preferred embodiment of the AI vision camera extrinsic parameter calibration method of the present invention, the following steps are described: Deeply fusing environmental feature vectors with image data captured by the AI vision camera, and precisely aligning them in the spatiotemporal dimension using a feature alignment algorithm to obtain fused data.
[0020] Based on the synchronization mechanism of the trigger signals of the camera and sensor array, the original environmental parameter data and image data are ensured to correspond in the time dimension.
[0021] By using the camera's intrinsic and extrinsic parameter matrices, the light intensity, temperature, humidity, and color temperature values in the environmental feature vector are projected onto their corresponding positions on the image plane, achieving spatial dimension alignment.
[0022] Bilinear interpolation is used to interpolate the distribution of environmental features in the image, so that the environmental features are continuously distributed in the image.
[0023] The processed environmental features are fused with the image pixel values pixel by pixel to obtain fused data.
[0024] As a preferred embodiment of the AI vision camera extrinsic calibration method of the present invention, the step of inputting fused data into a multi-layer deep feature extraction network to extract deep feature representations and form a high-dimensional feature matrix includes the following specific steps:
[0025] The fused data is normalized and organized into a four-dimensional tensor to obtain the input data;
[0026] The first layer of the convolutional neural network is used to perform convolution operations on the input data to extract local features, and the results are passed to the batch normalization layer for data normalization.
[0027] After introducing nonlinearity through the ReLU activation function, the second convolutional neural network further extracts deep local features, enhances the feature fusion effect, and performs max pooling to reduce the spatial size of the feature map.
[0028] After repeated convolution, batch normalization, activation, and pooling operations, the final feature map is flattened and input into a fully connected layer to integrate all features into a high-dimensional feature matrix.
[0029] As a preferred embodiment of the AI vision camera extrinsic calibration method of the present invention, the steps of inputting the high-dimensional feature matrix into the camera extrinsic prediction model based on deep learning, predicting the optimal camera extrinsic parameters, and generating a feedback signal through a feedback evaluation module are as follows:
[0030] The high-dimensional feature matrix is input into the LSTM-based camera extrinsic prediction model. The model receives and processes the input data to capture the temporal dependencies between features.
[0031] LSTM networks learn long-term dependencies through internal memory units and gating mechanisms, establish a mapping from input features to camera extrinsics, and output preliminary predicted camera extrinsics.
[0032] The predicted extrapolation is input into the actual image data feedback evaluation module to generate a feedback signal containing the evaluation score and error gradient.
[0033] As a preferred embodiment of the AI vision camera extrinsic parameter calibration method of the present invention, the steps of calculating the environment adaptation coefficient based on the feedback signal and the environment feature vector, and using this coefficient to self-calibrate the deep learning camera extrinsic parameter prediction model to obtain the calibrated model parameters are as follows:
[0034] The evaluation score and error gradient are extracted from the feedback signal and concatenated with the environmental feature vector to form a joint feature vector.
[0035] The joint feature vector is input into the multiple linear regression model to calculate the current environmental adaptability coefficient;
[0036] The weight matrix and bias terms of the deep learning camera extrinsic prediction model are adjusted using the environmental adaptation coefficient, and the calibrated model parameters are output.
[0037] As a preferred embodiment of the AI vision camera extrinsic parameter calibration method of the present invention, the step of using calibrated model parameters to finally determine the camera extrinsic parameters, realizing system cyclic operation, and completing the dynamic and accurate calibration of the camera extrinsic parameters, specifically includes the following steps:
[0038] The calibrated model parameters are applied to the camera extrinsic prediction model, replacing the original model parameters, thus completing the model update.
[0039] The updated model is used to process the new fused data to predict the optimal camera extrinsics for the current environment;
[0040] The predicted camera extrinsic parameters are applied to the camera system to perform real-time correction of the camera's imaging, ensuring the accuracy and stability of the camera's imaging.
[0041] The predicted camera extrinsic parameters are compared with the actual imaging results to generate new feedback signals, which are used to evaluate the process. This process is repeated to continuously optimize the camera extrinsic parameters and achieve dynamic and accurate calibration.
[0042] Secondly, this invention provides an AI vision camera extrinsic parameter calibration system, comprising,
[0043] The perception filtering model utilizes an array of environmental sensors to collect environmental parameters in real time at high frequency, and then performs preliminary filtering to form an environmental feature vector.
[0044] The fusion alignment model deeply fuses environmental feature vectors with image data captured by an AI vision camera, and precisely aligns them in the spatiotemporal dimensions using a feature alignment algorithm to obtain fused data.
[0045] The feature extraction model inputs the fused data into a multi-layer deep feature extraction network to extract deep feature representations and form a high-dimensional feature matrix.
[0046] The prediction and evaluation model takes the high-dimensional feature matrix as input to the deep learning-based camera extrinsic prediction model, predicts the optimal camera extrinsic parameters, and generates a feedback signal through the feedback evaluation module for quantitative evaluation.
[0047] The self-calibration model calculates the environmental adaptation coefficient based on the feedback signal and environmental feature vector. This coefficient is then used to self-calibrate the deep learning camera extrinsic prediction model, resulting in calibrated model parameters.
[0048] The dynamic calibration model uses the calibrated model parameters to determine the camera's extrinsic parameters, enabling the system to run cyclically and complete the dynamic and accurate calibration of the camera's extrinsic parameters.
[0049] Thirdly, the present invention provides a computer device including a memory and a processor, wherein the memory stores a computer program, wherein when the computer program is executed by the processor, it implements any step of the AI vision camera extrinsic calibration method as described in the first aspect of the present invention.
[0050] Fourthly, the present invention provides a computer-readable storage medium having a computer program stored thereon, wherein: when the computer program is executed by a processor, it implements any step of the AI vision camera extrinsic calibration method as described in the first aspect of the present invention.
[0051] The beneficial effects of this invention are as follows: By collecting environmental parameters in real time through an environmental perception sensor array and forming an environmental feature vector, and combining it with a feature alignment algorithm to deeply fuse environmental features with image data, the invention fully mines multi-source data information, significantly improving the accuracy of camera extrinsic parameter calibration compared to traditional single-data-source calibration methods. The introduction of an LSTM-based deep learning prediction model can capture complex temporal dependencies between features and accurately predict optimal camera extrinsic parameters. Simultaneously, a feedback evaluation module quantifies and evaluates the prediction results, and calculates an environmental adaptability coefficient based on the feedback signal and environmental feature vector to perform self-calibration of the model, achieving dynamic optimization of the model. This allows the system to maintain high-precision calibration results even in complex and changing environments, effectively solving the shortcomings of existing technologies in dynamic environment adaptability. Furthermore, the dynamic calibration model enables the system to run cyclically, continuously optimizing camera extrinsic parameters, further improving the accuracy and stability of camera imaging, and providing more reliable calibration technology support for the application of AI vision cameras in intelligent transportation, industrial automation, and other fields. Attached Figure Description
[0052] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0053] Figure 1 This is a flowchart of the AI vision camera extrinsic parameter calibration method in Example 1. Detailed Implementation
[0054] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings.
[0055] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and those skilled in the art can make similar extensions without departing from the spirit of the invention. Therefore, the invention is not limited to the specific embodiments disclosed below.
[0056] Secondly, the term "one embodiment" or "embodiment" as used herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The phrase "in one embodiment" appearing in different places in this specification does not necessarily refer to the same embodiment, nor is it a single or selective embodiment that is mutually exclusive with other embodiments.
[0057] Example 1, referring to Figure 1 This is the first embodiment of the present invention, which provides a method for calibrating extrinsic parameters of an AI vision camera, characterized by including:
[0058] S1: Environmental parameters are collected in real time at high frequency using an environmental sensing sensor array, and then the environmental feature vector is formed after preliminary filtering.
[0059] S2: Deeply fuse environmental feature vectors with image data captured by AI vision cameras, and accurately align them in the spatiotemporal dimensions using feature alignment algorithms to obtain fused data.
[0060] S3: Input the fused data into a multi-layer deep feature extraction network to extract deep feature representations and form a high-dimensional feature matrix.
[0061] S4: Input the high-dimensional feature matrix into the deep learning-based camera extrinsic prediction model to predict the optimal camera extrinsic parameters and generate a feedback signal through the feedback evaluation module for quantitative evaluation.
[0062] S5: Calculate the environmental adaptation coefficient based on the feedback signal and environmental feature vector, and use this coefficient to self-calibrate the deep learning camera extrinsic prediction model to obtain the calibrated model parameters.
[0063] S6: The calibrated model parameters are used to determine the camera extrinsic parameters, enabling the system to run in a loop and complete the dynamic and accurate calibration of the camera extrinsic parameters.
[0064] Specifically, the step of using an environmental sensing sensor array to collect environmental parameters in real time at high frequency, and then forming an environmental feature vector after preliminary filtering, involves the following steps:
[0065] Light, temperature, humidity, and color temperature sensors are installed around the AI vision camera according to a preset geometric arrangement to form an orderly sensor layout;
[0066] Start the sensor array, set the high-frequency acquisition frequency, and each sensor acquires ambient light intensity, temperature, humidity and color temperature values in real time to generate a raw environmental parameter data sequence;
[0067] The Kalman filter algorithm is used to smooth the original environmental parameter data sequence in the time domain. The filter state parameters are initialized with each sensor data sequence as input. After recursive processing through prediction and update steps, the filtered environmental parameter estimates are obtained.
[0068] The filtered light intensity, temperature, humidity, and color temperature values are integrated according to a predefined data structure to form an environmental feature vector.
[0069] It should be noted that light, temperature, humidity, and color temperature sensors are installed around the AI vision camera according to a preset geometric arrangement, forming an orderly sensor layout. This step, through precise sensor array placement, ensures that each sensor effectively covers the camera's field of view and does not interfere with each other, providing spatial rationality for subsequent data acquisition. The sensor array is then activated, and a high-frequency acquisition frequency is set. Each sensor acquires ambient light intensity, temperature, humidity, and color temperature values in real time, generating a raw environmental parameter data sequence. Each sensor operates based on its own physical principle: the light sensor is based on the photoelectric effect, the temperature sensor utilizes thermistor or semiconductor characteristics, the humidity sensor detects changes in capacitance or resistance, and the color temperature sensor is based on filter technology and a photodiode array. A high-frequency acquisition frequency of 1000 times per second is set to ensure that subtle changes in environmental parameters can be captured. A Kalman filter algorithm is used to perform temporal smoothing on the raw environmental parameter data sequence. Using the data sequence from each sensor as input, the filter state parameters are initialized, including state estimates and the estimation error covariance matrix. Data is recursively processed through prediction and update steps to remove high-frequency noise interference, resulting in filtered environmental parameter estimates. The filtered light intensity, temperature, humidity, and color temperature values are then integrated according to a predefined data structure to form an environmental feature vector. This vector has four dimensions, with each dimension corresponding to a filtered value of an environmental parameter, arranged in a fixed order to form a compact and ordered environmental feature vector.
[0070] By strategically arranging the sensor array, comprehensive coverage of environmental parameters within the camera's field of view was achieved. This orderly layout ensures the integrity and effectiveness of data acquisition, providing a solid foundation for subsequent data fusion and calibration. It enables real-time, high-frequency acquisition of environmental parameters, keenly capturing subtle environmental changes. The high-frequency acquisition ensures the timeliness and accuracy of the data, providing high-quality raw data for subsequent filtering and feature vector formation. The application of the Kalman filter algorithm effectively removes noise interference from the raw data, improving its purity and reliability. The filtered environmental parameter estimates are closer to the true values, providing accurate data support for subsequent feature vector formation. Integrating multiple environmental parameters into an ordered feature vector simplifies the data structure and facilitates subsequent processing and analysis. This integrated feature vector more intuitively reflects the environmental state, providing an efficient input format for camera extrinsic parameter calibration.
[0071] Specifically, the process involves deep fusing environmental feature vectors with image data captured by an AI vision camera, and precisely aligning them in the spatiotemporal dimensions using a feature alignment algorithm to obtain fused data. The specific steps are as follows:
[0072] Based on the synchronization mechanism of the trigger signals of the camera and sensor array, the original environmental parameter data and image data are ensured to correspond in the time dimension.
[0073] By using the camera's intrinsic and extrinsic parameter matrices, the light intensity, temperature, humidity, and color temperature values in the environmental feature vector are projected onto their corresponding positions on the image plane, achieving spatial dimension alignment.
[0074] Bilinear interpolation is used to interpolate the distribution of environmental features in the image, so that the environmental features are continuously distributed in the image.
[0075] The processed environmental features are fused with the image pixel values pixel by pixel to obtain fused data.
[0076] It should be noted that, based on the trigger signal synchronization mechanism of the camera and sensor array, the original environmental parameter data and image data are ensured to correspond in the time dimension. This step, through precise time synchronization, strictly aligns the environmental parameter data sequence with the image data sequence in time, ensuring that both are in the same time coordinate system. This time synchronization mechanism achieves accurate time matching between environmental parameter data and image data. This provides a unified temporal benchmark for subsequent data fusion, effectively avoiding data mismatch problems caused by time differences, and providing an accurate temporal data foundation for camera extrinsic parameter calibration. Using the camera's extrinsic and intrinsic parameter matrices, the light intensity, temperature, humidity, and color temperature values in the environmental feature vector are projected onto their corresponding positions on the image plane, achieving spatial dimension alignment. The camera's extrinsic and intrinsic parameter matrices play a crucial coordinate transformation role in this process, mapping three-dimensional environmental feature information to the two-dimensional image plane. This step aligns environmental features and image data in the spatial dimension, ensuring that each environmental parameter value accurately corresponds to a specific location on the image. This provides a spatial basis for subsequent feature fusion, enabling environmental and image features to be fused within the same spatial coordinate system. This provides accurate spatial positioning information for camera extrinsic calibration. Bilinear interpolation is used to interpolate the distribution of environmental features in the image, ensuring a continuous distribution. Bilinear interpolation smooths the distribution of environmental features on the image plane by calculating the weighted average of adjacent pixels, filling feature gaps caused by discrete sampling. This continuous distribution not only enhances the expressive power of environmental features but also provides richer details for subsequent feature fusion, making the fused data more realistically reflect the combined information of the environment and image. This provides a more refined feature representation for camera extrinsic calibration. The processed environmental features are then fused pixel-by-pixel with the image pixel values to obtain fused data. During the fusion process, the value of each pixel is composed of the image pixel value at that location and the corresponding environmental feature value, weighted according to certain weights, ultimately forming a fused dataset containing both environmental and image features.
[0077] This step achieves deep fusion of environmental and image features at the pixel level, making full use of two different types of data information. The fused data not only contains the visual information of the image but also incorporates multi-dimensional features of the environment, providing a more comprehensive and richer data foundation for camera extrinsic calibration and helping to improve the accuracy and reliability of the calibration.
[0078] Specifically, the process of inputting the fused data into a multi-layer deep feature extraction network to extract deep feature representations and form a high-dimensional feature matrix involves the following steps:
[0079] The fused data is normalized and organized into a four-dimensional tensor to obtain the input data;
[0080] The first layer of the convolutional neural network is used to perform convolution operations on the input data to extract local features, and the results are passed to the batch normalization layer for data normalization.
[0081] After introducing nonlinearity through the ReLU activation function, the second convolutional neural network further extracts deep local features, enhances the feature fusion effect, and performs max pooling to reduce the spatial size of the feature map.
[0082] After repeated convolution, batch normalization, activation, and pooling operations, the final feature map is flattened and input into a fully connected layer to integrate all features into a high-dimensional feature matrix.
[0083] It should be noted that the fused data is normalized, scaling it to the range [0, 1] to eliminate dimensional differences between different features. The normalized data is then organized into a four-dimensional tensor, containing the number of samples, image height, image width, and number of channels (including the RGB channels of the original image and the environmental feature channels), resulting in a data format suitable for deep learning network input. Through normalization and four-dimensional tensor organization, data standardization and structuring are achieved. This step provides a unified data input format for subsequent deep feature extraction, effectively improving data compatibility and processing efficiency, and laying the foundation for stable model training and feature extraction. The first-layer convolutional neural network performs convolution operations on the input data, with a kernel size of 3×3, a stride of 1, and a "same" padding method to maintain the feature map size being the same as the input image size. The convolution operation performs a linear transformation on the input data through a sliding window mechanism to extract local features. The convolution results are then passed to a batch normalization layer to normalize the mean and variance of each mini-batch of data, ensuring data stability and training speed. This step achieves preliminary extraction of local features from the fused data, while batch normalization improves the training efficiency and stability of the model. Local feature extraction captures basic features such as edges and textures in the image, providing the necessary information foundation for further feature fusion and abstraction. The normalized feature map is non-linearly transformed using the ReLU activation function, setting negative values to 0 and retaining positive values. A second convolutional neural network is then used to further convolve the activated feature map, extracting deeper local features and enhancing the feature fusion effect. The second convolutional operation also uses a 3×3 kernel with a stride of 1 and the same padding method. The ReLU activation function introduces non-linearity, enabling the model to learn more complex feature patterns. The second convolutional operation further abstracts features, extracting deeper local features and enhancing the model's ability to express features. This helps improve the discriminative power and representativeness of the features. Max pooling is then performed on the feature map after the second convolution, with a pooling window size of 2×2 and a stride of 2. Maximum sampling is used to reduce the spatial size of the feature map and decrease computational cost. After repeated convolution, batch normalization, activation, and pooling operations, the final feature map is flattened and input into a fully connected layer. The number of neurons in the fully connected layer is set according to actual needs, integrating all features to form a high-dimensional feature matrix.
[0084] Max pooling reduces the spatial dimensionality of the feature map, decreasing the computational cost of subsequent processing while preserving important information and enhancing the model's robustness to image scale changes. Fully connected layers integrate all local and global depth features, forming a high-dimensional feature matrix that provides a comprehensive feature representation for predicting camera extrinsic parameters, improving the model's prediction accuracy and generalization ability.
[0085] Specifically, the steps involve inputting the high-dimensional feature matrix into a deep learning-based camera extrinsic parameter prediction model to predict the optimal camera extrinsic parameters, and then quantifying and evaluating these parameters through a feedback evaluation module to generate a feedback signal.
[0086] The high-dimensional feature matrix is input into the LSTM-based camera extrinsic prediction model. The model receives and processes the input data to capture the temporal dependencies between features.
[0087] LSTM networks learn long-term dependencies through internal memory units and gating mechanisms, establish a mapping from input features to camera extrinsics, and output preliminary predicted camera extrinsics.
[0088] The predicted extrapolation is input into the actual image data feedback evaluation module to generate a feedback signal containing the evaluation score and error gradient.
[0089] It should be noted that the high-dimensional feature matrix is input into the LSTM-based camera extrinsic prediction model. After receiving the input data, the model processes the data through its internal structure. The core of the LSTM model lies in its ability to capture the temporal dependencies in sequential data, which is crucial for handling camera extrinsic calibration tasks with temporal continuity. The input high-dimensional feature matrix is processed step by step in the LSTM model. The model captures the changes in features over time by updating its internal state. By inputting the high-dimensional feature matrix into the LSTM model, the effective capture of the temporal dependencies of features is achieved. This step enables the model to understand the evolution pattern of features over time, providing a dynamic feature foundation for subsequent camera extrinsic prediction, improving the accuracy and adaptability of the prediction. The LSTM network learns long-term dependencies through internal memory units and gating mechanisms. The memory units are responsible for storing and transmitting long-term information, while the gating mechanisms (including input gates, forget gates, and output gates) control the flow of information. The input gate determines which new information is stored in the memory unit, the forget gate determines which information is forgotten, and the output gate determines which information is output. Through these mechanisms, the LSTM network can effectively learn the mapping relationship between input features and camera extrinsic parameters, outputting preliminary predicted camera extrinsic parameters. This step establishes a mapping relationship from input features to camera extrinsic parameters through the internal mechanism of the LSTM network. This mapping relationship can capture the long-term dependence between features and extrinsic parameters, enabling the model to predict camera extrinsic parameters more accurately, improving the accuracy and reliability of the calibration results. The predicted extrinsic parameters are then input into the feedback evaluation module using actual image data. The feedback evaluation module quantifies the accuracy of the prediction results by comparing the differences between the predicted extrinsic parameters and the actual image data. Specifically, the module calculates the reprojection error of image feature points under the predicted extrinsic parameters and the matching consistency of feature points under different viewpoints, generating a feedback signal containing the evaluation score and error gradient. The feedback evaluation module generates the feedback signal, realizing the quantitative evaluation of the prediction results. This step not only provides an accuracy indicator of the prediction results but also provides guidance for subsequent model optimization, enabling the model to self-adjust and improve based on the feedback signal, forming a closed-loop optimization system.
[0090] Specifically, the environmental adaptation coefficient is calculated based on the feedback signal and environmental feature vector. This coefficient is then used to self-calibrate the deep learning camera extrinsic prediction model to obtain the calibrated model parameters. The specific steps are as follows:
[0091] The evaluation score and error gradient are extracted from the feedback signal and concatenated with the environmental feature vector to form a joint feature vector.
[0092] The joint feature vector is input into the multiple linear regression model to calculate the current environmental adaptability coefficient;
[0093] The weight matrix and bias terms of the deep learning camera extrinsic prediction model are adjusted using the environmental adaptation coefficient, and the calibrated model parameters are output.
[0094] It should be noted that evaluation scores and error gradient information are extracted from the feedback signal, while environmental feature vectors are obtained simultaneously. The extracted evaluation scores and error gradients are concatenated with the environmental feature vectors to form a new joint feature vector. During the concatenation process, the order and correspondence of each feature element are ensured to be accurate for subsequent processing. By concatenating key information from the feedback signal with the environmental feature vector to form a joint feature vector, multi-source information integration is achieved. This step provides comprehensive feature input for the subsequent calculation of the environmental fitness coefficient, enabling the model to consider both feedback information and environmental features simultaneously, improving the comprehensiveness and accuracy of model calibration. The joint feature vector is then input into a multiple linear regression model. The multiple linear regression model uses each component of the joint feature vector as independent variables and the performance improvement rate of the calibration model in historical data as the dependent variable. The regression coefficients are estimated using the least squares method, establishing a quantitative relationship between the joint feature vector and the environmental fitness coefficient. Based on the input joint feature vector, the environmental fitness coefficient at the current moment is calculated. The multiple linear regression model is then used to calculate the environmental fitness coefficient, establishing a quantitative link between feedback information, environmental features, and model calibration parameters. This step enables the accurate calculation of the environmental adaptation coefficient, providing crucial parameter support for subsequent model calibration and enhancing the model's adaptability to environmental changes. The calculated environmental adaptation coefficient is used to adjust the weight matrix and bias terms of the deep learning camera extrinsic prediction model. Specifically, the environmental adaptation coefficient is multiplied element-wise with the model's current weight matrix to scale the weight values; simultaneously, the bias terms are scaled and translated based on the environmental adaptation coefficient. After these operations, the calibrated model parameters are output, completing the model's self-calibration process.
[0095] By using an environmental adaptation coefficient to self-calibrate the deep learning camera extrinsic prediction model, dynamic adjustment of model parameters is achieved. This step enables the model to automatically optimize its parameters based on current environmental characteristics and feedback information, improving the prediction accuracy and stability under different environmental conditions, and enhancing the model's generalization ability and adaptability.
[0096] Specifically, the process involves using calibrated model parameters to finalize the camera extrinsic parameters, enabling the system to run cyclically and achieve dynamic and accurate calibration of the camera extrinsic parameters. The specific steps are as follows:
[0097] The calibrated model parameters are applied to the camera extrinsic prediction model, replacing the original model parameters, thus completing the model update.
[0098] The updated model is used to process the new fused data to predict the optimal camera extrinsics for the current environment;
[0099] The predicted camera extrinsic parameters are applied to the camera system to perform real-time correction of the camera's imaging, ensuring the accuracy and stability of the camera's imaging.
[0100] The predicted camera extrinsic parameters are compared with the actual imaging results to generate new feedback signals, which are used to evaluate the process. This process is repeated to continuously optimize the camera extrinsic parameters and achieve dynamic and accurate calibration.
[0101] It should be noted that the calibrated model parameters obtained from the self-calibration step are applied to the camera extrinsic parameter prediction model, replacing the original model parameters and completing the model update. This step ensures that the model can reflect the latest environmental features and feedback information in a timely manner, providing optimized parameter support for subsequent predictions. By updating the model parameters, dynamic optimization of the model is achieved. This step enables the model to continuously adapt to environmental changes and new data feedback, improving the model's prediction accuracy and adaptability, and ensuring the accuracy and reliability of camera extrinsic parameter prediction. The updated model is used to process the new fused data, and through model calculation and analysis, the optimal camera extrinsic parameters for the current environment are predicted. The new fused data contains the latest environmental features and image information. The model uses this data for feature extraction and analysis, outputting the camera extrinsic parameters that best fit the current environmental conditions. This step achieves dynamic and accurate prediction of camera extrinsic parameters. By using the updated model parameters and the new fused data, the model can more accurately predict the camera extrinsic parameters adapted to the current environment, improving the accuracy and stability of camera imaging and providing a reliable foundation for subsequent imaging applications. The predicted camera extrinsic parameters are applied to the camera system for real-time correction of camera imaging. The camera system adjusts its imaging parameters based on the new extrinsic parameters to optimize imaging effects, ensuring accuracy and stability. By correcting camera imaging in real time, image quality is improved. This step effectively reduces imaging errors caused by inaccurate camera extrinsic parameters, improving image clarity and reliability. It provides high-quality image data for subsequent image analysis and applications, comparing the currently predicted camera extrinsic parameters with the actual imaging results to generate new feedback signals. These feedback signals contain evaluation scores and error gradient information, used to evaluate the system. The system then triggers the feedback evaluation module again based on the new feedback signal, enabling cyclical operation and continuous optimization of camera extrinsic parameters.
[0102] This step achieves closed-loop optimization and dynamic calibration of the system. By continuously generating new feedback signals and using them for further model optimization, the system can continuously improve the prediction accuracy of camera extrinsic parameters, adapt to environmental changes and new data inputs, forming a continuously optimizing dynamic system that ensures the long-term accuracy and stability of camera extrinsic parameter calibration.
[0103] This embodiment also provides an AI vision camera extrinsic parameter calibration system, including:
[0104] The perception filtering model utilizes an array of environmental sensors to collect environmental parameters in real time at high frequency, and then performs preliminary filtering to form an environmental feature vector.
[0105] The fusion alignment model deeply fuses environmental feature vectors with image data captured by an AI vision camera, and precisely aligns them in the spatiotemporal dimensions using a feature alignment algorithm to obtain fused data.
[0106] The feature extraction model inputs the fused data into a multi-layer deep feature extraction network to extract deep feature representations and form a high-dimensional feature matrix.
[0107] The prediction and evaluation model takes the high-dimensional feature matrix as input to the deep learning-based camera extrinsic prediction model, predicts the optimal camera extrinsic parameters, and generates a feedback signal through the feedback evaluation module for quantitative evaluation.
[0108] The self-calibration model calculates the environmental adaptation coefficient based on the feedback signal and environmental feature vector. This coefficient is then used to self-calibrate the deep learning camera extrinsic prediction model, resulting in calibrated model parameters.
[0109] The dynamic calibration model uses the calibrated model parameters to determine the camera's extrinsic parameters, enabling the system to run cyclically and complete the dynamic and accurate calibration of the camera's extrinsic parameters.
[0110] This embodiment also provides a computer device applicable to the AI vision camera extrinsic parameter calibration method, including: a memory and a processor; the memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions to implement the AI vision camera extrinsic parameter calibration method proposed in the above embodiment.
[0111] The computer device can be a terminal, comprising a processor, memory, communication interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, carrier networks, NFC (Near Field Communication), or other technologies. The display screen can be an LCD screen or an e-ink screen. The input devices can be a touch layer covering the display screen, buttons, a trackball, or a touchpad on the computer device's casing, or an external keyboard, touchpad, or mouse.
[0112] This embodiment also provides a storage medium storing a computer program, which, when executed by a processor, implements the AI vision camera extrinsic parameter calibration method proposed in the above embodiments. The storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as Static Random Access Memory (SRAM), Electrically Erasable Programmable Read-Only Memory (EEPROM), Erasable Programmable Read-Only Memory (EPROM), Programmable Red-Only Memory (PROM), Read-Only Memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.
[0113] In summary, this invention significantly improves the accuracy of camera extrinsic parameter calibration compared to traditional single-data source calibration methods by: real-time acquisition of environmental parameters and formation of environmental feature vectors using an environmental perception sensor array; deep fusion of environmental features and image data using a feature alignment algorithm; and full utilization of multi-source data information. The introduction of an LSTM-based deep learning prediction model captures complex temporal dependencies between features and accurately predicts optimal camera extrinsic parameters. Simultaneously, a feedback evaluation module quantifies the prediction results and calculates an environmental adaptability coefficient based on the feedback signal and environmental feature vector to perform self-calibration of the model, achieving dynamic optimization. This allows the system to maintain high-precision calibration even in complex and changing environments, effectively addressing the shortcomings of existing technologies in dynamic environmental adaptability. Furthermore, the dynamic calibration model enables the system to run cyclically, continuously optimizing camera extrinsic parameters, further improving the accuracy and stability of camera imaging and providing more reliable calibration technology support for the application of AI vision cameras in intelligent transportation, industrial automation, and other fields.
[0114] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.
Claims
1. A method for calibrating extrinsic parameters of an AI vision camera, characterized in that: include, S1: Environmental parameters are collected in real time at high frequency using an environmental sensing sensor array, and environmental feature vectors are formed after preliminary filtering. S2: Deeply fuse environmental feature vectors with image data captured by AI vision cameras, and accurately align them in the spatiotemporal dimension using feature alignment algorithms to obtain fused data; S3: Input the fused data into a multi-layer deep feature extraction network to extract deep feature representations and form a high-dimensional feature matrix; S4: Input the high-dimensional feature matrix into the deep learning-based camera extrinsic prediction model to predict the optimal camera extrinsic parameters and generate a feedback signal through the feedback evaluation module for quantitative evaluation. S5: Calculate the environmental adaptation coefficient based on the feedback signal and environmental feature vector. Use this coefficient to self-calibrate the deep learning camera extrinsic prediction model to obtain the calibrated model parameters. The specific steps are as follows: The evaluation score and error gradient are extracted from the feedback signal and concatenated with the environmental feature vector to form a joint feature vector. The joint feature vector is input into the multiple linear regression model to calculate the current environmental adaptability coefficient; The weight matrix and bias terms of the deep learning camera extrinsic prediction model are adjusted using the environmental adaptation coefficient, and the calibrated model parameters are output. S6: Use the calibrated model parameters to finally determine the camera extrinsic parameters, and implement system loop operation to complete the dynamic and accurate calibration of the camera extrinsic parameters. The specific steps are as follows: The calibrated model parameters are applied to the camera extrinsic prediction model, replacing the original model parameters, thus completing the model update. The updated model is used to process the new fused data to predict the optimal camera extrinsics for the current environment; The predicted camera extrinsic parameters are applied to the camera system to perform real-time correction of the camera's imaging, ensuring the accuracy and stability of the camera's imaging. The predicted camera extrinsic parameters are compared with the actual imaging results to generate new feedback signals, which are used to evaluate the process. This process is repeated to continuously optimize the camera extrinsic parameters and achieve dynamic and accurate calibration.
2. The AI vision camera extrinsic parameter calibration method as described in claim 1, characterized in that: The method involves using an environmental sensing sensor array to collect environmental parameters in real time at high frequency, and then performing preliminary filtering to form an environmental feature vector. The specific steps are as follows: Light, temperature, humidity, and color temperature sensors are installed around the AI vision camera according to a preset geometric arrangement to form an orderly sensor layout; Start the sensor array, set the high-frequency acquisition frequency, and each sensor acquires ambient light intensity, temperature, humidity and color temperature values in real time to generate a raw environmental parameter data sequence; The Kalman filter algorithm is used to smooth the original environmental parameter data sequence in the time domain. The filter state parameters are initialized with each sensor data sequence as input. After recursive processing through prediction and update steps, the filtered environmental parameter estimates are obtained. The filtered light intensity, temperature, humidity, and color temperature values are integrated according to a predefined data structure to form an environmental feature vector.
3. The AI vision camera extrinsic parameter calibration method as described in claim 2, characterized in that: The process involves deep fusing environmental feature vectors with image data captured by an AI vision camera, and precisely aligning them in the spatiotemporal dimensions using a feature alignment algorithm to obtain fused data. The specific steps are as follows: Based on the synchronization mechanism of the trigger signals of the camera and sensor array, the original environmental parameter data and image data are ensured to correspond in the time dimension. By using the camera's intrinsic and extrinsic parameter matrices, the light intensity, temperature, humidity, and color temperature values in the environmental feature vector are projected onto their corresponding positions on the image plane, achieving spatial dimension alignment. Bilinear interpolation is used to interpolate the distribution of environmental features in the image, so that the environmental features are continuously distributed in the image. The processed environmental features are fused with the image pixel values pixel by pixel to obtain fused data.
4. The AI vision camera extrinsic parameter calibration method as described in claim 3, characterized in that: The process involves inputting the fused data into a multi-layer deep feature extraction network to extract deep feature representations and form a high-dimensional feature matrix. The specific steps are as follows: The fused data is normalized and organized into a four-dimensional tensor to obtain the input data; The first layer of the convolutional neural network is used to perform convolution operations on the input data to extract local features, and the results are passed to the batch normalization layer for data normalization. After introducing nonlinearity through the ReLU activation function, the second convolutional neural network further extracts deep local features, enhances the feature fusion effect, and performs max pooling to reduce the spatial size of the feature map. After repeated convolution, batch normalization, activation, and pooling operations, the final feature map is flattened and input into a fully connected layer to integrate all features into a high-dimensional feature matrix.
5. The AI vision camera extrinsic parameter calibration method as described in claim 4, characterized in that: The specific steps are as follows: inputting the high-dimensional feature matrix into the deep learning-based camera extrinsic parameter prediction model to predict the optimal camera extrinsic parameters, and then quantifying and evaluating them through a feedback evaluation module to generate a feedback signal. The high-dimensional feature matrix is input into the LSTM-based camera extrinsic prediction model. The model receives and processes the input data to capture the temporal dependencies between features. LSTM networks learn long-term dependencies through internal memory units and gating mechanisms, establish a mapping from input features to camera extrinsics, and output preliminary predicted camera extrinsics. The predicted extrapolation is input into the actual image data feedback evaluation module to generate a feedback signal containing the evaluation score and error gradient.
6. An AI vision camera extrinsic parameter calibration system, based on the AI vision camera extrinsic parameter calibration method according to any one of claims 1 to 5, characterized in that: include, The perception filtering model utilizes an array of environmental sensors to collect environmental parameters in real time at high frequency, and then forms an environmental feature vector after preliminary filtering. The fusion alignment model deeply fuses environmental feature vectors with image data captured by an AI vision camera, and accurately aligns them in the spatiotemporal dimensions using a feature alignment algorithm to obtain fused data. The feature extraction model inputs the fused data into a multi-layer deep feature extraction network to extract deep feature representations and form a high-dimensional feature matrix. The prediction and evaluation model inputs the high-dimensional feature matrix into the deep learning-based camera extrinsic prediction model, predicts the optimal camera extrinsic parameters, and generates a feedback signal through the feedback evaluation module for quantitative evaluation. The self-calibration model calculates the environmental adaptation coefficient based on the feedback signal and environmental feature vector, and uses this coefficient to self-calibrate the deep learning camera extrinsic prediction model to obtain the calibrated model parameters. The dynamic calibration model uses the calibrated model parameters to determine the camera's extrinsic parameters, enabling the system to run cyclically and complete the dynamic and accurate calibration of the camera's extrinsic parameters.
7. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that: When the processor executes the computer program, it implements the steps of the AI vision camera extrinsic parameter calibration method according to any one of claims 1 to 5.
8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by the processor, it implements the steps of the AI vision camera extrinsic parameter calibration method according to any one of claims 1 to 5.