Camera attitude automatic calibration system and method based on multi-sensor fusion

Abstract

The invention belongs to the field of camera attitude control, and particularly relates to a camera attitude automatic calibration system and method based on multi-sensor fusion, and the system comprises the steps: obtaining video or image data of a target shot object and multi-dimensional deflection state information data of a corresponding camera through a data collection unit disposed on the camera, and carrying out the preprocessing; the data self-processing module calculates a dynamic camera attitude deviation angle by using a target calculation unit, an attitude space parameter unit and an adjustment prediction unit, obtains a camera attitude calibration parameter space, and carries out training to obtain a target attitude self-calibration model and a camera attitude calibration parameter space after secondary pre-calibration; the attitude self-calibration module constructs an enhanced adjustment model through a distributed model unit, and integrates the model into a camera attitude adjustment subsystem through an integrated self-calibration unit to realize real-time attitude calibration; according to the invention, through multi-sensor fusion and an intelligent algorithm, automatic calibration and accurate control of the attitude of the camera are realized.

Description

Technical Field

[0001] The present invention belongs to the field of camera attitude control, and particularly relates to an automatic camera attitude calibration system and method based on multi-sensor fusion. Background Art

[0002] Cameras play an important role in traffic monitoring, vehicle recognition, pedestrian detection, etc. However, during the installation and use of cameras, they may be affected by the external environment, resulting in changes in their attitudes, thus affecting the accuracy and application effects of video data. Traditional camera attitude calibration methods usually rely on manual calibration or the measurement of a single sensor (such as an inclinometer), and there are many deficiencies. For example, the method of a single sensor is greatly affected by environmental factors (such as temperature, humidity, electromagnetic interference, etc.), resulting in inaccurate measurement of attitude changes. In addition, the data of a single sensor cannot comprehensively reflect the attitude changes of the camera, and misjudgments are likely to occur. Moreover, the existing calibration methods are insufficient in terms of real-time performance and cannot meet the requirements of modern intelligent transportation systems for high real-time performance.

[0003] For example, Chinese Patent Application Publication No. CN118570307A discloses a method for recalibrating a camera perception system and a camera perception system, which are used to calibrate the current virtual world coordinate system adopted by the camera perception system based on the attitude change of the camera relative to the initial attitude. Among them, after the first calibration, the coordinates of the same object in the initial virtual world coordinate system and the real world coordinate system are the same, including: S1. Obtaining the attitude change of the camera based on the inclinometer installed on the camera; S2. Constructing a spatial coordinate system offset correction parameter based on the obtained attitude change of the camera, so that the coordinates of the same object in the current virtual world coordinate system and the real world coordinate system are the same, where the current virtual world coordinate system is obtained by correcting the initial virtual world coordinate system with the constructed spatial coordinate system offset correction parameter.

[0004] For example, Chinese Patent Publication No. CN101127128B discloses an annular camera array calibration system and its method. The method includes: all cameras in the annular field simultaneously photograph a calibration plate in each position and attitude, and collect an image of this position and attitude; multiple acquisition servers respectively perform corner detection on all the collected images and calculate the homography matrix Hij of each camera; the calibration server optimally groups the camera array according to the relationship weight of the number of common position and attitude that can be detected between two cameras; perform intra-group joint calibration on each group of cameras, and unify the external parameters of each group to the external parameters with the same camera as the origin of the world coordinate; optimize the calibration parameters; obtain the maximum likelihood estimation of the internal and external parameters of the camera array.

[0005] The above prior art has the following problems. In the existing tilt sensors or vision methods, the data acquisition frequency is relatively low, and the information on the minute attitude changes of the camera cannot be detected, resulting in a continuous increase in the integration error during camera attitude calibration. In addition, a single accelerometer or magnetometer is vulnerable to external vibration and shock, leading to inaccurate attitude estimation. For this reason, the present invention provides a camera attitude automatic calibration system and method based on multi-sensor fusion. Summary of the Invention

[0006] Aiming at the deficiencies of the prior art, the present invention proposes a camera attitude automatic calibration system and method based on multi-sensor fusion. The system obtains video or image data of the target object to be photographed and corresponding multi-dimensional deflection state information data of the camera through a data acquisition unit configured on the camera and preprocesses them. The data self-processing module calculates the dynamic camera attitude deviation angle by using a target calculation unit, an attitude space parameter unit, and an adjustment prediction unit, obtains the camera attitude calibration parameter space, and trains to obtain a target attitude self-calibration model and a camera attitude calibration parameter space after secondary pre-calibration. The attitude self-calibration module constructs a reinforcement adjustment model through a distributed model unit and integrates the model into the camera attitude adjustment subsystem through an integrated self-calibration unit to achieve real-time attitude calibration. The present invention realizes the automatic calibration and precise control of the camera attitude through multi-sensor fusion and intelligent algorithms.

[0007] To achieve the above object, the present invention provides the following technical solutions:

[0008] A camera attitude automatic calibration system based on multi-sensor fusion, comprising: a data acquisition module, a data self-processing module, an attitude self-calibration module, and a user interface module;

[0009] The data acquisition module includes a data acquisition unit; in the three-dimensional deflection coordinate system of the configured camera, video or image data of the target object to be photographed and corresponding multi-dimensional deflection state information data of the camera are obtained, and the acquired data is synchronously filtered and preprocessed through a configured distributed filtering model;

[0010] The data self-processing module includes a target calculation unit, an attitude space parameter unit, and an adjustment prediction unit; the preprocessed video or image data of the target object to be photographed is input into the target calculation unit to obtain the dynamic camera attitude deviation angle; at the same time, the multi-dimensional deflection state information data of the camera is input into the attitude space parameter unit to obtain the camera attitude calibration parameter space; the dynamic camera attitude deviation angle and the camera attitude calibration parameter space are input into the adjustment prediction unit to obtain a trained target attitude self-calibration model and a camera attitude calibration parameter space after secondary pre-calibration;

[0011] The attitude self-calibration module includes a distributed model unit and an integrated self-calibration unit; a reinforcement adjustment model is constructed and trained through the distributed model unit, and the reinforcement adjustment model and the target attitude self-calibration model are integrated into the attitude adjustment subsystem corresponding to each camera through the integrated self-calibration unit. At the same time, according to the target shooting video and multi-dimensional deflection state information data obtained in real time or the attitude adjustment parameters manually input through the user interface module, the camera attitude at the current moment is calibrated in real time.

[0012] Specifically, the camera multi-dimensional deflection state information data includes the X-axis angular velocity of the camera in the camera three-dimensional deflection coordinate system obtained by the gyroscope Y-axis angular velocity Z-axis angular velocity The X-axis acceleration ax, Y-axis acceleration ay, and Z-axis acceleration az of the camera in the camera three-dimensional deflection coordinate system obtained by the accelerometer; the X-axis magnetic field component mx, Y-axis magnetic field component my, and Z-axis magnetic field component mz of the camera in the camera three-dimensional deflection coordinate system obtained by the magnetometer.

[0013] Specifically, the steps for obtaining the camera attitude calibration parameter space include:

[0014] A 1. Construct a camera attitude parameter space according to the camera multi-dimensional deflection state information data

[0015]

[0016] A2. Input the in the camera attitude parameter space into a high-pass filter to remove the low-frequency drift band and obtain a high-pass filtered attitude parameter space;

[0017] A3. Integrate the parameters in the high-pass filtered attitude parameter space to obtain the angle offset of the camera in the camera three-dimensional deflection coordinate system within a fixed time period where successively represent the rotation angle of the i-th camera around the X-axis, the rotation angle around the Y-axis, and the rotation angle around the Z-axis obtained by the corresponding integral calculation;

[0018] A4. Input into a low-pass filter to remove the high-frequency noise band and obtain a low-pass filtered attitude parameter space, and use the low-pass filtered attitude parameter space to obtain and where represents the tilt angle of the i-th camera in the vertical plane within the camera three-dimensional deflection coordinate system, represents the tilt angle of the i-th camera in the horizontal plane within the camera three-dimensional deflection coordinate system, represents the orientation angle of the i-th camera in the horizontal plane, i.e., the rotation angle around the Z-axis;

[0019] A5. Input and into the complementary filter, and use the attitude parameter point corresponding to the minimum covariance in the complementary filter for fusion to obtain the fused camera attitude calibration parameter space

[0020] Specifically, the steps for obtaining the dynamic camera attitude deviation angle include:

[0021] B1. Obtain the target captured image, the corresponding reference image, and the corresponding camera attitude parameter space, and preprocess the obtained images;

[0022] B2. Input the preprocessed images into an image segmentation model with a built-in scale, obtain the visible state volume space of the target object in the image, and construct the invisible state volume space of the target object through the scale algorithm based on the current visible size ratio of the target object in the visible state volume space;

[0023] B3. According to the visible state volume space of the target object, obtain the corresponding Euler angles of the current camera shooting through the camera parameters and the inverse perspective projection model, combine the invisible state volume space of the target object and the visible state volume space of the target object, and use the combined target object state volume space to obtain the ideal Euler angles of the camera shooting and the corresponding ideal camera attitude parameter space;

[0024] B4. Use the corresponding Euler angles of the current camera shooting and the ideal Euler angles of the camera shooting to obtain the camera attitude deviation angle corresponding to the current moment.

[0025] Specifically, the steps for obtaining the dynamic camera attitude deviation angle further include:

[0026] B5. Configure the target tracking algorithm in the camera, obtain the trajectory video or image data of the target object corresponding to different moments within the monitoring range of the camera, repeat the B1 - B3 process, and obtain the corresponding Euler angles of the camera shooting and the ideal Euler angles of the camera shooting and the corresponding ideal camera attitude parameter space at each moment within the current camera monitoring range;

[0027] B6. Use the corresponding Euler angles of the camera shooting and the ideal Euler angles of the camera shooting at each moment within the current camera monitoring range to obtain the dynamic camera attitude deviation angle sequence θt, t = 1…T, where T represents the time length of the movement of the target object under the current camera;

[0028] B7. Construct a trajectory-attitude prediction model, input the trajectory videos or image data corresponding to the target objects at different times and the corresponding ideal camera attitude parameter spaces into the trajectory-attitude prediction model for training, and obtain the ideal camera attitude parameter space at the next moment;

[0029] B8. When the error between the ideal camera attitude parameter space at the next moment and the corresponding real ideal camera attitude parameter space is 0, obtain the trained trajectory-attitude prediction model.

[0030] Specifically, the specific steps for training the target attitude self-calibration model include:

[0031] C1. Input the dynamic camera attitude deviation angle sequence, the corresponding ideal camera attitude parameter space, and the camera attitude calibration parameter space into a linear function to obtain a quadratic calibration fitting function;

[0032] C2. Construct a target attitude self-calibration model based on the particle swarm algorithm, and use the quadratic calibration fitting function as the fitness function of the target attitude self-calibration model;

[0033] C3. Input the ideal camera attitude parameter space and the camera attitude calibration parameter space into the target attitude self-calibration model for training. When the camera attitude deviation angles corresponding to the ideal camera attitude parameter space and the camera attitude calibration parameter space are 0 for n consecutive training cycles, obtain the trained target attitude self-calibration model;

[0034] C4. Obtain the real-time camera attitude calibration parameter space through the A1-A5 process, and input the obtained real-time camera attitude calibration parameter space into the trained target attitude self-calibration model to obtain the quadratic calibrated camera attitude calibration parameter space corresponding to the current moment;

[0035] C5. Input the camera attitude calibration parameter space at the current moment and the quadratic calibrated camera attitude calibration parameter space corresponding to the current moment into the trajectory-attitude prediction model to obtain the quadratic pre-calibrated camera attitude calibration parameter space at the next moment.

[0036] Specifically, the steps for constructing the reinforcement adjustment model include:

[0037] D1. Construct the input state at the current moment based on the quadratic pre-calibrated camera attitude calibration parameter space at the next moment, the quadratic calibrated camera attitude calibration parameter space corresponding to the current moment, and the camera attitude deviation angle;

[0038] D2. Construct the trigger information at the current moment based on the input state at the current moment, including:

[0039] When no manual input instruction is received at the current moment and the camera attitude deviation angle is 0, the first execution action information is triggered, that is, the camera shooting is performed while maintaining the current camera attitude calibration parameter space;

[0040] When no manual input instruction is received at the current moment and the camera attitude deviation angle is not 0, the second execution action information is triggered, that is, the target attitude self-calibration model and the trajectory-attitude prediction model are called to perform secondary calibration on the current camera attitude calibration parameter space, and at the same time, pre-calibration is performed on the camera attitude calibration parameter space at the next moment, so that the camera attitude deviation angle is 0 at consecutive time points;

[0041] When the manually input attitude adjustment parameter is received, the third execution action information is triggered, that is, the manually input attitude adjustment parameter is used as the first-priority adjustment parameter, and the target attitude self-calibration model is called to perform secondary calibration on the current camera attitude calibration parameter space by using the manually input attitude adjustment parameter, and the manually secondary-calibrated camera attitude calibration parameter space is obtained to calibrate the camera attitude;

[0042] D3. Construct the current moment execution action at=(at1, at2, at3) based on the current moment trigger information, where at1 represents the action corresponding to the trigger of the first execution action information, at2 represents the action corresponding to the trigger of the second execution action information, and at3 represents the action corresponding to the trigger of the third execution action information.

[0043] The method for automatically calibrating the camera attitude based on multi-sensor fusion includes the steps of:

[0044] S1. Obtain the video or image data of the camera target object and the corresponding camera multi-dimensional deflection state information data in the constructed three-dimensional deflection coordinate system, and perform preprocessing through the configured distributed filtering model;

[0045] S2. Use the preprocessed camera image and the camera internal-external parameters to obtain the dynamic camera attitude deviation angle, and at the same time, use the preprocessed camera multi-dimensional deflection state information data to calculate the camera attitude calibration parameter space;

[0046] S3. Use the camera attitude calibration parameter space and the dynamic camera attitude deviation angle to obtain the camera attitude adjustment prediction function through linear function fitting, and configure the camera attitude adjustment prediction function into the target attitude self-calibration model constructed by the particle swarm algorithm for training, and perform secondary pre-calibration on the camera attitude calibration parameter space to obtain the trained target attitude self-calibration model;

[0047] S4. Build and train a reinforcement adjustment model, integrate the reinforcement adjustment model and the target pose self-calibration model into the camera pose adjustment subsystem, and at the same time, according to the target shooting video obtained in real time and the corresponding multi-dimensional deflection state information data or the manually input pose adjustment parameters, perform real-time calibration on the camera pose at the current moment, and perform real-time calibration on the camera pose at the current moment through the camera pose adjustment subsystem.

[0048] A computer-readable storage medium, on which computer instructions are stored, and when the computer instructions run, they execute the method for automatically calibrating the camera pose based on multi-sensor fusion.

[0049] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0050] Aiming at the deficiencies of the prior art, the present invention fuses and supplements the traditional single accelerometer or magnetometer data with the angular velocity data obtained by the gyroscope to construct a camera pose calibration parameter space, enhancing the system's capture of the camera's subtle movement information. Secondly, through the obtained target shooting object video data, the dynamic camera pose deviation angle is obtained, and the dynamic camera pose deviation angle is used to perform secondary pre-pose calibration on the camera pose calibration parameter space, enabling the camera to make real-time adjustments according to the movement trajectory of the target shooting object. While reducing the camera movement delay, the robustness of the camera pose adjustment is enhanced, enabling the camera to quickly and accurately perform real-time calibration of the camera pose in a complex and changeable external environment. Description of the Drawings

[0051] Figure 1 It is a module diagram of the camera pose automatic calibration system based on multi-sensor fusion in Embodiment 1 of the present invention;

[0052] Figure 2 It is a flowchart of the corresponding unit of the camera pose automatic calibration system based on multi-sensor fusion in Embodiment 2 of the present invention;

[0053] Figure 3 It is a flowchart of the method for automatically calibrating the camera pose based on multi-sensor fusion in Embodiment 3 of the present invention. Detailed Embodiments

[0054] Embodiment 1

[0055] Please refer to Figure 1 , an embodiment provided by the present invention: a camera pose automatic calibration system based on multi-sensor fusion, including: a data acquisition module, a data self-processing module, a pose self-calibration module, and a user interface module;

[0056] The data acquisition module is used to acquire video and image data from different perspectives and distances on the road and perform preprocessing; the data acquisition module includes a spatial coordinate unit and a data acquisition unit;

[0057] The spatial coordinate unit is used to construct a three-dimensional deflection coordinate system of the camera with the deflection support point of the camera installed on the road as the origin;

[0058] The data acquisition unit obtains video or image data of the target object to be photographed and corresponding multi-dimensional deflection state information data of the camera based on the three-dimensional deflection coordinate system of the camera, and performs synchronous filtering preprocessing on the camera image and multi-dimensional deflection state information data by using the distributed filtering model configured in each sensor;

[0059] Further, in this embodiment, the multi-dimensional deflection state information data of the camera includes: the angular velocity ωx of the camera on the X-axis in the three-dimensional deflection coordinate system of the camera obtained by the gyroscope representing the rotation rate of the camera around the X-axis, the angular velocity ωy representing the rotation rate of the camera around the Y-axis, and the angular velocity ωz representing the rotation rate of the camera around the Z-axis;

[0060] The acceleration ax of the camera on the X-axis in the three-dimensional deflection coordinate system of the camera obtained by the accelerometer: representing the acceleration of the camera along the X-axis, the acceleration ay of the Y-axis: representing the acceleration of the camera along the Y-axis, and the acceleration az of the Z-axis: representing the acceleration of the camera along the Z-axis;

[0061] The magnetic field component mx of the camera on the X-axis in the three-dimensional deflection coordinate system of the camera obtained by the magnetometer: representing the geomagnetic field intensity of the camera along the X-axis, the magnetic field component my of the Y-axis: representing the geomagnetic field intensity of the camera along the Y-axis, and the magnetic field component mz of the Z-axis: representing the geomagnetic field intensity of the camera along the Z-axis.

[0062] The data self-processing module is used to process the acquired image or video data to obtain the target shooting deviation angle and the corresponding attitude deviation adjustment space; the data self-processing module includes a target calculation unit, an attitude space parameter unit, and an adjustment prediction unit;

[0063] The target calculation unit is used to obtain the dynamic camera attitude deviation angle according to the acquired target shooting image, the target reference frame image, and the internal-external parameters of the camera;

[0064] Further, the steps for obtaining the dynamic camera attitude deviation angle in this embodiment include:

[0065] B1. Obtain the target shooting image, the corresponding reference image, and the corresponding camera attitude parameter space, and perform preprocessing on the acquired images;

[0066] B2. Input the preprocessed image into an image segmentation model with a built-in scale to obtain the visible state volume space of the target object in the image. Based on the current visible size ratio of the target object in the visible state volume space and through the scale algorithm, construct the invisible state volume space of the target object.

[0067] B3. According to the visible state volume space of the target object, obtain the Euler angles corresponding to the current camera shooting through the camera parameters and the inverse perspective projection model. Combine the invisible state volume space and the visible state volume space of the target object, and use the combined state volume space of the target object to obtain the ideal Euler angles for camera shooting and the corresponding ideal camera pose parameter space.

[0068] B4. Use the Euler angles corresponding to the current camera shooting and the ideal Euler angles for camera shooting to obtain the camera pose deviation angle corresponding to the current moment.

[0069] B5. Configure a target tracking algorithm for the camera to obtain the trajectory video or image data of the target object at different moments within the monitoring range of the camera. Repeat the processes of B1 - B3 to obtain the Euler angles corresponding to the current camera shooting, the ideal Euler angles for camera shooting, and the corresponding ideal camera pose parameter space at each moment within the current camera monitoring range.

[0070] B6. Use the Euler angles corresponding to the current camera shooting and the ideal Euler angles for camera shooting at each moment within the current camera monitoring range to obtain a sequence of dynamic camera pose deviation angles θt, where t = 1…T, and T represents the time length of the movement of the target object under the current camera.

[0071] B7. Construct a trajectory - pose prediction model, and input the trajectory video or image data of the target object at different moments and the corresponding ideal camera pose parameter space into the trajectory - pose prediction model for training, and obtain the ideal camera pose parameter space for the next moment.

[0072] B8. When the error between the ideal camera pose parameter space for the next moment and the corresponding real ideal camera pose parameter space is 0, obtain the trained trajectory - pose prediction model.

[0073] Furthermore, in this embodiment, the trajectory - pose prediction model is constructed through a trajectory tracking algorithm. It is used to predict the trajectory position point of the target object at the next moment based on the trajectory information of the target object at historical and current moments within the visible range of the current camera, so as to obtain the camera pose deviation angle between the camera pose calibration parameter space corresponding to the current moment of the camera and the ideal camera pose parameter space at each future moment, and perform pre - adjustment accordingly to further reduce the calculation delay time of the camera and improve the reaction rate of the camera.

[0074] This process ensures the image quality by obtaining the target captured image and the reference image and performing preprocessing. Then, using an image segmentation model with a built-in scale, the visible state volume space and the invisible state volume space of the target captured object are accurately calculated, and the Euler angles corresponding to the current camera capture and the ideal Euler angles are obtained through the inverse perspective projection model. Next, the trajectory data of the target captured object at different times is obtained through the target tracking algorithm, and a dynamic camera attitude deviation angle sequence is constructed. Finally, a trajectory-attitude prediction model is constructed, and the ideal attitude parameter space of the camera at the next moment is obtained through training to ensure that the attitude deviation angle is 0. This process significantly improves the accuracy and real-time performance of camera attitude estimation, reduces the attitude deviation, and enhances the robustness and stability of the system. Especially in a dynamic environment, it can adjust the camera attitude in real time to ensure the stable tracking and high-quality display of the target captured object.

[0075] The attitude space parameter unit is used to obtain the camera attitude parameter space according to the camera multi-dimensional deflection state information data, and construct the camera attitude calibration parameter space by using the camera target attitude parameter space;

[0076] Further, the steps of obtaining the camera attitude calibration parameter space in this embodiment include:

[0077] A1. Construct the camera attitude parameter space according to the camera multi-dimensional deflection state information data

[0078]

[0079] A2. Input the in the camera attitude parameter space into a high-pass filter to remove the low-frequency drift band and obtain the high-pass filtered attitude parameter space;

[0080] A3. Calculate the angle offset of the camera in the camera three-dimensional deflection coordinate system within a fixed time period through integral calculation of the parameters in the high-pass filtered attitude parameter space where successively represent the rotation angle of the i-th camera around the X-axis, the rotation angle around the Y-axis, and the rotation angle around the Z-axis obtained by corresponding integral calculation;

[0081] A4. Input into a low-pass filter to remove the high-frequency noise band, obtain the low-pass filtered attitude parameter space, and use the low-pass filtered attitude parameter space to obtain and where represents the tilt angle of the i-th camera in the vertical plane within the camera three-dimensional deflection coordinate system, represents the tilt angle of the i-th camera in the horizontal plane within the three-dimensional deflection coordinate system of the camera represents the orientation angle of the i-th camera in the horizontal plane, that is, the rotation angle around the Z-axis;

[0082] A5. Input and into the complementary filter, and use the attitude parameter point corresponding to the minimum covariance in the complementary filter for fusion to obtain the fused camera attitude calibration parameter space

[0083] This process uses the multi-dimensional deflection state information data obtained by the gyroscope, accelerometer, and magnetometer to construct the camera attitude parameter space. Then, the low-frequency drift is removed through a high-pass filter to obtain the high-pass filtered attitude parameter space; the angle offset within a fixed time period is obtained through integral calculation to ensure the dynamic accuracy of the attitude parameters; secondly, the high-frequency noise is removed using a low-pass filter to obtain the low-pass filtered attitude parameter space, and the tilt angle and orientation angle of the camera in the three-dimensional deflection coordinate system are calculated; finally, the results of the high-pass filter and the low-pass filter are input into the complementary filter, and the attitude parameter point corresponding to the minimum covariance is selected for fusion to obtain the fused camera attitude calibration parameter space; further improving the accuracy and stability of attitude estimation, effectively eliminating the influence of low-frequency drift and high-frequency noise, and the fusion of gyroscope data further improves the sensitivity of the camera attitude calibration parameter space constructed by the Euler angles calculated by the accelerometer and magnetometer to external vibrations and shocks, ensuring the stability and robustness of the camera in a dynamic environment.

[0084] The adjustment prediction unit is used to obtain the camera attitude adjustment prediction function by linear function fitting according to the camera attitude calibration parameter space and the dynamic camera attitude deviation angle, and configure the camera attitude adjustment prediction function into the target attitude self-calibration model constructed by the particle swarm algorithm for training to obtain the trained target attitude self-calibration model, and pre-calibrate the camera attitude calibration parameter space through the target attitude self-calibration model using the dynamic camera attitude deviation angle;

[0085] Furthermore, the specific steps of training the target attitude self-calibration model in this embodiment include:

[0086] C1. Input the dynamic camera attitude deviation angle sequence and the corresponding camera ideal attitude parameter space and camera attitude calibration parameter space into the linear function to obtain the secondary calibration fitting function;

[0087] C2. Construct the target attitude self-calibration model based on the particle swarm algorithm, and use the secondary calibration fitting function as the fitness function of the target attitude self-calibration model;

[0088] C3. Input the ideal camera pose parameter space and the camera pose calibration parameter space into the target pose self-calibration model for training. When the camera pose deviation angles corresponding to the ideal camera pose parameter space and the camera pose calibration parameter space are all 0 within n consecutive training cycles, obtain the trained target pose self-calibration model;

[0089] C4. Obtain the real-time camera pose calibration parameter space through processes A1 - A5, and input the obtained real-time camera pose calibration parameter space into the trained target pose self-calibration model to obtain the camera pose calibration parameter space after secondary calibration corresponding to the current moment;

[0090] C5. Input the camera pose calibration parameter space at the current moment and the camera pose calibration parameter space after secondary calibration corresponding to the current moment into the trajectory-pose prediction model to obtain the camera pose calibration parameter space after secondary pre-calibration at the next moment.

[0091] This process realizes the synchronous high-precision acquisition and filtering preprocessing of the multi-dimensional deflection state information data of the camera and the video or image data of the target object to be photographed through a highly integrated data acquisition and processing module, effectively reducing noise interference and improving data quality; at the same time, it uses the angular velocity data obtained by the gyroscope to fuse and enhance the data obtained by the accelerometer and magnetometer, further enhancing the anti-interference ability of the constructed camera pose calibration parameter space; secondly, the target calculation unit accurately calculates the dynamic camera pose deviation angle and constructs the ideal camera pose parameter space by using advanced image segmentation, inverse perspective projection and scale algorithms, providing an accurate target reference for subsequent pose calibration; at the same time, the pose space parameter unit effectively removes low-frequency drift and high-frequency noise through multi-stage filtering processes such as high-pass filtering, low-pass filtering and complementary filtering, and obtains a high-precision camera pose calibration parameter space; on this basis, the adjustment prediction unit constructs a target pose self-calibration model by using linear function fitting and particle swarm algorithm, realizing the real-time adjustment prediction and secondary pre-calibration of the camera pose, and significantly improving the accuracy and real-time performance of the camera pose.

[0092] The user interface module is used for the real-time display of the camera shooting results and the input of manual pose adjustment parameters.

[0093] The pose self-calibration module is used for self-calibrating the camera pose according to the real-time acquired image or input instruction data; the pose self-calibration module includes a distributed model unit and an integrated self-calibration unit;

[0094] The distributed model unit is used to construct a reinforcement adjustment model, and train the reinforcement adjustment model by using the obtained target captured image, target reference frame image, camera internal-external parameters, and manually input parameters, and deploy the trained reinforcement adjustment model to the pose adjustment subsystem corresponding to each camera;

[0095] Further, the steps for constructing the reinforcement adjustment model in this embodiment include:

[0096] D1. Construct the input state at the current moment based on the camera pose calibration parameter space after the secondary pre-calibration at the next moment, the camera pose calibration parameter space after the secondary calibration at the current moment, and the camera pose deviation angle where represents the camera pose calibration parameter space after the secondary calibration corresponding to the i-th camera at the current t moment, represents the camera pose calibration parameter space after the secondary pre-calibration of the i-th camera at the t+1 moment, represents the pose deviation angle corresponding to the i-th camera at the current t moment;

[0097] D2. Construct the trigger information at the current moment based on the input state at the current moment, including:

[0098] When no manual input instruction is received at the current moment and the camera pose deviation angle is 0, trigger the first execution action information, that is, keep the current camera pose calibration parameter space for camera shooting;

[0099] When no manual input instruction is received at the current moment and the camera pose deviation angle is not 0, trigger the second execution action information, that is, call the target pose self-calibration model and the trajectory-pose prediction model to perform secondary calibration on the current camera pose calibration parameter space, and at the same time perform pre-calibration on the camera pose calibration parameter space at the next moment, so that the camera pose deviation angle is 0 at consecutive time points;

[0100] When the manually input pose adjustment parameter is received, trigger the third execution action information, that is, use the manually input pose adjustment parameter as the first-priority adjustment parameter, and call the target pose self-calibration model to perform secondary calibration on the current camera pose calibration parameter space by using the manually input pose adjustment parameter to obtain the manually secondary-calibrated camera pose calibration parameter space and calibrate the camera pose;

[0101] D3. Construct the execution action at the current moment at = (at1, at2, at3) based on the trigger information at the current moment, where at1 represents the action corresponding to triggering the first execution action information, at2 represents the action corresponding to triggering the second execution action information, and at3 represents the action corresponding to triggering the third execution action information;

[0102] D4. Construct a reinforcement adjustment model based on the SAC algorithm, and input the current moment input state, the current moment trigger information, and the current moment execution action constructed into the reinforcement adjustment model for training to obtain a trained reinforcement adjustment model;

[0103] D5. Configure the trained reinforcement adjustment model into each camera, and automatically calibrate the camera pose according to the video data obtained by the camera in real time.

[0104] The integrated self-calibration unit is used to integrate the reinforcement adjustment model and the target pose self-calibration model, and use the target shooting video or image, multi-dimensional deflection state information data obtained in real time, or manually input pose adjustment parameters to calibrate the current moment camera pose in real time.

[0105] This process realizes the real-time display of the camera shooting results and the flexible input of manual pose adjustment parameters by integrating the user interface module and the pose self-calibration module, significantly improving the convenience of user operation and the accuracy of camera pose adjustment; the reinforcement adjustment model constructed by the distributed model unit can be efficiently trained based on real-time data and manually input parameters to ensure that each camera can obtain a personalized pose adjustment strategy; at the same time, the integrated self-calibration unit organically combines the reinforcement adjustment model with the target pose self-calibration model to realize the real-time and accurate calibration of the camera pose, effectively reducing the shooting error caused by pose deviation.

[0106] Embodiment 2

[0107] Please refer to Figure 2 , another embodiment provided by the present invention: The working process of the corresponding unit of the camera pose automatic calibration system based on multi-sensor fusion includes:

[0108] First, construct a three-dimensional deflection coordinate system of the camera through the space coordinate unit, and use the data acquisition unit to obtain the target shooting object video or image data and the corresponding camera multi-dimensional deflection state information data according to the three-dimensional deflection coordinate system of the camera, and perform real-time synchronous filtering preprocessing on the camera image and the multi-dimensional deflection state information data by using the distributed filtering model configured in each sensor;

[0109] Second, input the preprocessed camera image into the target calculation unit to obtain the dynamic camera pose deviation angle. At the same time, input the preprocessed multi-dimensional deflection state information data into the pose space parameter unit to obtain the camera pose calibration parameter space; input the camera pose calibration parameter space and the dynamic camera pose deviation angle into the adjustment prediction unit to obtain the target pose self-calibration model, and use the dynamic camera pose deviation angle to pre-calibrate the camera pose calibration parameter space through the target pose self-calibration model;

[0110] Thirdly, input the target captured image, the target reference frame image, the internal and external camera parameters, and the attitude adjustment parameters manually input through the user interface module into the distributed model unit to obtain the enhanced adjustment model. Then, integrate and build the enhanced adjustment model and the target attitude self-calibration model into the attitude adjustment subsystem of the corresponding camera through the integrated self-calibration unit. At the same time, use the target captured video or image obtained in real time, the corresponding multi-dimensional deflection state information data, or the manually input attitude adjustment parameters to perform real-time calibration on the camera attitude at the current moment, and display the calibrated video in real time through the user interface module.

[0111] The system constructs a three-dimensional deflection coordinate system for the camera through the space coordinate unit, and uses the data acquisition unit to obtain the video or image data of the target captured object and the multi-dimensional deflection state information data. Through real-time synchronous filtering preprocessing by the distributed filtering model, noise and drift are effectively removed, improving the accuracy and stability of the data. Secondly, input the preprocessed image into the target calculation unit to obtain the dynamic camera attitude deviation angle. At the same time, input the multi-dimensional deflection state information data into the attitude space parameter unit to obtain the camera attitude calibration parameter space. Through the adjustment prediction unit, use the dynamic camera attitude deviation angle and the attitude calibration parameter space to obtain the target attitude self-calibration model and perform pre-calibration, ensuring the accuracy and real-time performance of the attitude estimation. Finally, input the target captured image, the target reference frame image, the internal and external camera parameters, and the manually input attitude adjustment parameters into the distributed model unit to obtain the enhanced adjustment model, and integrate it with the target attitude self-calibration model into the attitude adjustment subsystem of the camera through the integrated self-calibration unit, realizing real-time calibration of the camera attitude at the current moment. The whole process significantly improves the accuracy and real-time performance of the camera attitude calibration, ensuring stability and robustness in a dynamic environment.

[0112] Embodiment 3

[0113] Please refer to Figure 3 , another embodiment provided by the present invention: a method for automatically calibrating the camera attitude based on multi-sensor fusion, the steps include:

[0114] S1. Obtain the video or image data of the camera target captured object and the corresponding camera multi-dimensional deflection state information data in the constructed three-dimensional deflection coordinate system, and perform preprocessing through the configured distributed filtering model;

[0115] S2. Use the preprocessed camera image and the internal and external camera parameters to obtain the dynamic camera attitude deviation angle, and at the same time use the preprocessed camera multi-dimensional deflection state information data to calculate the camera attitude calibration parameter space;

[0116] S3. Utilize the camera pose calibration parameter space and the dynamic camera pose deviation angle, obtain the camera pose adjustment prediction function through linear function fitting, configure the camera pose adjustment prediction function into the target pose self-calibration model constructed by the particle swarm algorithm for training, and perform secondary pre-calibration on the camera pose calibration parameter space to obtain the trained target pose self-calibration model;

[0117] S4. Construct and train the reinforcement adjustment model, integrate the reinforcement adjustment model and the target pose self-calibration model into the camera pose adjustment subsystem, and at the same time, according to the real-time obtained target shooting video and the corresponding multi-dimensional deflection state information data or the manually input pose adjustment parameters, perform real-time calibration on the camera pose at the current moment, and perform real-time calibration on the camera pose at the current moment through the camera pose adjustment subsystem.

[0118] The embodiments of the present invention have been described above in conjunction with the accompanying drawings. However, the present invention is not limited to the above specific embodiments. The above specific embodiments are merely illustrative and not restrictive. Under the inspiration of the present invention, those of ordinary skill in the art can also make changes, modifications, substitutions, and variations to the above embodiments without departing from the spirit and scope protected by the present invention and the claims. These all fall within the protection scope of the present invention.

[0119] If the technical solution of the present disclosure involves personal information, before the product applying the technical solution of the present disclosure processes personal information, it has clearly informed the personal information processing rules and obtained the personal's autonomous consent. If the technical solution of the present disclosure involves sensitive personal information, before the product applying the technical solution of the present disclosure processes sensitive personal information, it has obtained the personal's separate consent and at the same time meets the requirement of "express consent". For example, at a personal information collection device such as a camera, a clear and prominent sign is set to inform that the personal information collection range has been entered and personal information will be collected. If a person voluntarily enters the collection range, it is deemed that they consent to the collection of their personal information; or on the personal information processing device, when the personal information processing rules are informed by obvious signs / information, personal authorization is obtained through pop-up information or asking the person to upload their personal information by themselves; among them, the personal information processing rules may include information such as the personal information processor, the purpose of personal information processing, the processing method, and the types of personal information processed.

Claims

1. An automatic calibration system for camera pose based on multi-sensor fusion, characterized in that, Including: A data acquisition module, a data self - processing module, an attitude self - calibration module, and a user interface module; The data acquisition module includes a data acquisition unit; In the configured three - dimensional deflection coordinate system of the camera, the data acquisition unit acquires the video or image data of the target object to be photographed and the corresponding multi - dimensional deflection state information data of the camera, and performs synchronous filtering pre - processing on the acquired data through the configured distributed filtering model; The data self - processing module includes a target calculation unit, an attitude space parameter unit, and an adjustment prediction unit; the pre - processed video or image data of the target object to be photographed is input into the target calculation unit to obtain the dynamic camera attitude deviation angle; at the same time, the multi - dimensional deflection state information data of the camera is input into the attitude space parameter unit to obtain the camera attitude calibration parameter space; The dynamic camera attitude deviation angle and the camera attitude calibration parameter space are input into the adjustment prediction unit to obtain the trained target attitude self - calibration model and the camera attitude calibration parameter space after secondary pre - calibration; The attitude self - calibration module includes a distributed model unit and an integrated self - calibration unit; the distributed model unit is used to construct and train a reinforcement adjustment model, and the integrated self - calibration unit integrates the reinforcement adjustment model and the target attitude self - calibration model into the attitude adjustment subsystem corresponding to each camera. At the same time, according to the real - time acquired target shooting video and multi - dimensional deflection state information data or the attitude adjustment parameters manually input through the user interface module, the camera attitude at the current moment is calibrated in real time.

2. The automatic calibration system for camera pose based on multi-sensor fusion according to claim 1, characterized in that, The camera multi-dimensional deflection state information data includes the X-axis angular velocity of the camera in the camera three-dimensional deflection coordinate system obtained by the gyroscope Y-axis angular velocity Z-axis angular velocity The X-axis acceleration ax, Y-axis acceleration ay, and Z-axis acceleration az of the camera in the camera three-dimensional deflection coordinate system obtained by the accelerometer; the X-axis magnetic field component mx, Y-axis magnetic field component my, and Z-axis magnetic field component mz of the camera in the camera three-dimensional deflection coordinate system obtained by the magnetometer.

3. The camera attitude automatic calibration system based on multi-sensor fusion according to claim 2, characterized in that, The steps for obtaining the camera attitude calibration parameter space include: A1. Construct a camera attitude parameter space based on the camera multi-dimensional deflection state information data A2. Input the in the camera pose parameter space into a high-pass filter to remove the low-frequency drift band and obtain a high-pass filtered pose parameter space; A3. Integrate the parameters in the high-pass filtered attitude parameter space to obtain the angular offset of the camera in the three-dimensional deflection coordinate system of the camera over a fixed time period. Where Successively represent The i-th rotation angle of the camera around the X-axis, the rotation angle around the Y-axis, and the rotation angle around the Z-axis obtained by corresponding integral calculation; A4. Input into a low-pass filter to remove high-frequency noise bands, obtain a low-pass filtered attitude parameter space, and use the low-pass filtered attitude parameter space to obtain and where represents the tilt angle of the i-th camera in the vertical plane within the three-dimensional deflection coordinate system of the camera, represents the tilt angle of the i-th camera in the horizontal plane within the three-dimensional deflection coordinate system of the camera, represents the orientation angle of the i-th camera in the horizontal plane, that is, the rotation angle around the Z-axis; A5. Input and into a complementary filter, and use the attitude parameter point corresponding to the minimum covariance in the complementary filter for fusion to obtain the fused camera attitude calibration parameter space 4. The automatic calibration system for camera pose based on multi-sensor fusion according to claim 3, wherein The steps for obtaining the dynamic camera attitude deviation angle include: B1. Acquire the target shooting image, the corresponding reference image, and the corresponding camera attitude parameter space, and pre - process the acquired images; B2. Input the pre - processed images into an image segmentation model with a built - in scale to obtain the visible state volume space of the target object in the image. Through the current visible size ratio of the target object in the visible state volume space and the scale algorithm, construct the invisible state volume space of the target object; B3. According to the visible state volume space of the target object, through the camera parameters and the inverse perspective projection model, obtain the Euler angles corresponding to the current camera shooting. Combine the invisible state volume space and the visible state volume space of the target object, and use the combined target object state volume space to obtain the ideal Euler angles of the camera shooting and the corresponding ideal camera attitude parameter space; B4. Use the Euler angles corresponding to the current camera shooting and the ideal Euler angles of the camera shooting to obtain the camera attitude deviation angle corresponding to the current moment.

5. The camera attitude automatic calibration system based on multi-sensor fusion according to claim 4, characterized in that, The steps for obtaining the dynamic camera attitude deviation angle also include: B5. Configure a target tracking algorithm in the camera to acquire the trajectory video or image data of the target object to be photographed at different times within the monitoring range of the camera. Repeat the process of B1 - B3 to obtain the Euler angles corresponding to the camera shooting at each moment within the current camera monitoring range, the ideal Euler angles of the camera shooting, and the corresponding ideal camera attitude parameter space; B6. Monitor the Euler angles captured by the camera and the ideal Euler angles for camera capture at each moment within the monitoring range of the current camera, and obtain a sequence of dynamic camera attitude deviation angles θt, t = 1...T within the monitoring range of the current camera, where T represents the time length of the movement of the target object under the current camera; B7. Construct a trajectory-attitude prediction model, and input the trajectory video or image data corresponding to the target object at different moments and the corresponding ideal camera attitude parameter space into the trajectory-attitude prediction model for training, and obtain the ideal camera attitude parameter space for the next moment; B8. When the error between the ideal camera attitude parameter space for the next moment and the corresponding real ideal camera attitude parameter space is 0, obtain the trained trajectory-attitude prediction model.

6. The automatic camera attitude calibration system based on multi-sensor fusion according to claim 5, wherein The specific steps for training the target attitude self-calibration model include: C1. Input the sequence of dynamic camera attitude deviation angles, the corresponding ideal camera attitude parameter space, and the camera attitude calibration parameter space into a linear function to obtain a quadratic calibration fitting function; C2. Construct a target attitude self-calibration model based on the particle swarm algorithm, and use the quadratic calibration fitting function as the fitness function of the target attitude self-calibration model; C3. Input the ideal camera attitude parameter space and the camera attitude calibration parameter space into the target attitude self-calibration model for training. When the camera attitude deviation angles corresponding to the ideal camera attitude parameter space and the camera attitude calibration parameter space are 0 for n consecutive training cycles, obtain the trained target attitude self-calibration model; C4. Obtain the real-time camera attitude calibration parameter space through the A1 - A5 process, and input the obtained real-time camera attitude calibration parameter space into the trained target attitude self-calibration model to obtain the quadratic calibrated camera attitude calibration parameter space corresponding to the current moment; C5. Input the camera attitude calibration parameter space at the current moment and the quadratic calibrated camera attitude calibration parameter space corresponding to the current moment into the trajectory-attitude prediction model to obtain the quadratic pre-calibrated camera attitude calibration parameter space for the next moment.

7. The automatic calibration system for camera pose based on multi-sensor fusion according to claim 6, wherein, The steps for constructing the reinforcement adjustment model include: D1. Construct the input state at the current moment based on the quadratic pre-calibrated camera attitude calibration parameter space for the next moment, the quadratic calibrated camera attitude calibration parameter space corresponding to the current moment, and the camera attitude deviation angle; D2. Construct the trigger information at the current moment based on the input state at the current moment, including: When no manual input instruction is received at the current moment and the camera attitude deviation angle is 0, trigger the first execution action information, that is, keep the current camera attitude calibration parameter space for camera capture; When no manual input instruction is received at the current moment and the camera attitude deviation angle is not 0, trigger the second execution action information, that is, call the target attitude self-calibration model and the trajectory-attitude prediction model to perform quadratic calibration on the current camera attitude calibration parameter space, and at the same time perform pre-calibration on the camera attitude calibration parameter space for the next moment, so that the camera attitude deviation angles at consecutive time points are all 0; When receiving manually input attitude adjustment parameters, trigger the third execution action information, that is, use the manually input attitude adjustment parameters as the first-priority adjustment parameters, and call the target attitude self-calibration model to perform secondary calibration on the current camera attitude calibration parameter space using the manually input attitude adjustment parameters, obtain the manually secondary-calibrated camera attitude calibration parameter space, and calibrate the camera attitude; D3. Construct the current moment execution action at=(at1, at2, at3) based on the current moment trigger information, where at1 represents the action corresponding to the trigger of the first execution action information, at2 represents the action corresponding to the trigger of the second execution action information, and at3 represents the action corresponding to the trigger of the third execution action information.

8. The automatic calibration method for the camera pose based on multi-sensor fusion is implemented based on the automatic calibration system for the camera pose based on multi-sensor fusion described in any one of claims 1-7, and is characterized in that the steps Including: S1. Obtain the video or image data of the camera target shooting object and the corresponding multi-dimensional deflection state information data of the camera in the constructed three-dimensional deflection coordinate system, and perform preprocessing through the configured distributed filtering model; S2. Use the preprocessed camera image and the internal-external parameters of the camera to obtain the dynamic camera attitude deviation angle, and at the same time use the preprocessed multi-dimensional deflection state information data of the camera to calculate the camera attitude calibration parameter space; S3. Use the camera attitude calibration parameter space and the dynamic camera attitude deviation angle to obtain the camera attitude adjustment prediction function through linear function fitting, and configure the camera attitude adjustment prediction function into the target attitude self-calibration model constructed by the particle swarm algorithm for training, and perform secondary pre-calibration on the camera attitude calibration parameter space to obtain the trained target attitude self-calibration model; S4. Construct and train the reinforcement adjustment model, integrate the reinforcement adjustment model and the target attitude self-calibration model into the camera attitude adjustment subsystem, and at the same time perform real-time calibration on the current moment camera attitude according to the real-time obtained target shooting video and the corresponding multi-dimensional deflection state information data or the manually input attitude adjustment parameters, and perform real-time calibration on the current moment camera attitude through the camera attitude adjustment subsystem.

9. A computer-readable storage medium, characterized in that, It stores computer instructions, and when the computer instructions run, execute the method for automatically calibrating the camera attitude based on multi-sensor fusion described in claim 8.

Images