Camera pose automatic calibration system and method based on multi-sensor fusion

Abstract

The present application belongs to the field of camera posture control, and particularly relates to a camera posture automatic calibration system and method based on multi-sensor fusion, which acquires target shooting video or image data and corresponding camera multi-dimensional deflection state information data and pre-processes through a data acquisition unit arranged on the camera; a data self-processing module utilizes a target calculation unit, a posture space parameter unit and an adjustment prediction unit to calculate dynamic camera posture deviation angle, acquire camera posture calibration parameter space, and train to obtain a target posture self-calibration model and a camera posture calibration parameter space after secondary pre-calibration; a posture self-calibration module constructs a reinforced adjustment model through a distributed model unit, and integrates the model into a camera posture adjustment subsystem through an integrated self-calibration unit to realize real-time posture calibration; the present application realizes automatic calibration and accurate control of the camera posture through multi-sensor fusion and intelligent algorithm.

Description

TECHNICAL FIELD

[0001] The present application belongs to the field of camera pose control, and particularly relates to a camera pose automatic calibration system and method based on multi-sensor fusion. BACKGROUND

[0002] Cameras play an important role in traffic monitoring, vehicle recognition, pedestrian detection, etc. However, the camera may be affected by the external environment during installation and use, resulting in changes in its pose, thereby affecting the accuracy of video data and application effect. Traditional camera pose calibration methods usually rely on manual calibration or measurement by a single sensor (such as a tilt sensor), and have many shortcomings. For example, the single sensor method is greatly affected by environmental factors (such as temperature, humidity, electromagnetic interference, etc.), resulting in inaccurate measurement of pose changes. In addition, the data of the single sensor cannot fully reflect the changes in the pose of the camera, and misjudgment is likely to occur. Furthermore, existing calibration methods have deficiencies in real-time performance and cannot meet the requirements of modern intelligent traffic systems for high real-time performance.

[0003] A camera perception system recalibration method and a camera perception system are disclosed in Chinese Patent Application No. CN118570307A, which is used to calibrate a current virtual world coordinate system adopted by a camera perception system based on changes in the pose of the camera relative to an initial pose, wherein the coordinates of the same object in the initial virtual world coordinate system and the real world coordinate system are the same after the first calibration, comprising: S1, obtaining the pose changes of the camera based on the tilt sensor installed on the camera; S2, constructing a spatial coordinate system offset correction parameter based on the obtained pose changes of the camera, so that the coordinates of the same object in the current virtual world coordinate system and the real world coordinate system are the same, wherein the current virtual world coordinate system is obtained by correcting the initial virtual world coordinate system by the constructed spatial coordinate system offset correction parameter.

[0004] A ring camera array calibration system and method are disclosed in Chinese Patent No. CN101127128B, which comprises: simultaneously capturing images of each position and pose of a calibration plate by all cameras in a ring field; performing corner point detection on all captured images by multiple acquisition servers, and calculating a homography matrix Hij for each camera; optimizing grouping of the camera array by a calibration server according to the number of common position and poses that can be detected between each pair of cameras as the relationship weight therebetween; performing joint calibration within each group of cameras to unify the external parameters of each group as external parameters with the same camera as the world coordinate origin; optimizing the calibration parameters; and obtaining the maximum likelihood estimation of the internal and external parameters of the camera array.

[0005] The existing technologies have the following problems: existing tilt sensors or vision methods have low data acquisition frequency and cannot detect small changes in camera attitude, which makes the integration error continuously increase when calibrating camera attitude. In addition, a single accelerometer or magnetometer is easily affected by external vibration and impact, resulting in inaccurate attitude estimation. To address this, the present invention provides an automatic camera attitude calibration system and method based on multi-sensor fusion. Summary of the Invention

[0006] To address the shortcomings of existing technologies, this invention proposes an automatic camera attitude calibration system and method based on multi-sensor fusion. The system acquires and preprocesses video or image data of the target object and corresponding multi-dimensional camera deflection state information through a data acquisition unit configured on the camera. A data self-processing module uses a target calculation unit, an attitude space parameter unit, and an adjustment prediction unit to calculate the dynamic camera attitude deviation angle, obtain the camera attitude calibration parameter space, and train a target attitude self-calibration model and a camera attitude calibration parameter space after secondary pre-calibration. The attitude self-calibration module constructs an enhanced adjustment model through a distributed model unit and integrates the model into the camera attitude adjustment subsystem through an integrated self-calibration unit to achieve real-time attitude calibration. This invention achieves automatic calibration and precise control of camera attitude through multi-sensor fusion and intelligent algorithms.

[0007] To achieve the above objectives, the present invention provides the following technical solution:

[0008] The camera attitude automatic calibration system based on multi-sensor fusion includes: a data acquisition module, a data self-processing module, an attitude self-calibration module, and a user interface module.

[0009] The data acquisition module includes a data acquisition unit; the data acquisition unit acquires video or image data of the target object and corresponding multi-dimensional deflection state information data of the camera in the configured three-dimensional deflection coordinate system of the camera, and performs synchronous filtering preprocessing on the acquired data through the configured distributed filtering model;

[0010] The data self-processing module includes a target calculation unit, a posture space parameter unit, and an adjustment prediction unit. It inputs pre-processed target video or image data into the target calculation unit to obtain the dynamic camera posture deviation angle. Simultaneously, it inputs multi-dimensional camera deflection state information into the posture space parameter unit to obtain the camera posture calibration parameter space. Finally, it inputs the dynamic camera posture deviation angle and the camera posture calibration parameter space into the adjustment prediction unit to obtain the trained target posture self-calibration model and the camera posture calibration parameter space after secondary pre-calibration.

[0011] The attitude self-calibration module comprises a distributed model unit and an integrated self-calibration unit; a reinforced adjustment model is constructed and trained through the distributed model unit, and the reinforced adjustment model and a target attitude self-calibration model are integrated into an attitude adjustment subsystem corresponding to each camera through the integrated self-calibration unit, and the camera attitude at the current time is calibrated in real time according to the target shooting video and multi-dimensional deflection state information data acquired in real time or the attitude adjustment parameters manually input through a user interface module.

[0012] Specifically, the camera multi-dimensional deflection state information data comprises X-axis angular velocity Y-axis angular velocity Z-axis angular velocity X-axis acceleration ax, Y-axis acceleration ay and Z-axis acceleration az of the camera in the camera three-dimensional deflection coordinate system acquired by an accelerometer, and X-axis magnetic field component mx, Y-axis magnetic field component my and Z-axis magnetic field component mz of the camera in the camera three-dimensional deflection coordinate system acquired by a magnetometer.

[0013] Specifically, the step of obtaining the camera attitude calibration parameter space comprises:

[0014] A1, constructing a camera attitude parameter space according to the camera multi-dimensional deflection state information data

[0015]

[0016] A2, inputting the camera attitude parameter space into a high-pass filter to remove a low-frequency drift wave band and obtain a high-pass filtered attitude parameter space;

[0017] A3, obtaining the angle offset of the camera in the camera three-dimensional deflection coordinate system in a fixed time period through integral calculation of the parameters in the high-pass filtered attitude parameter space wherein represents the rotation angle of the i-th camera around the X-axis, the rotation angle of the i-th camera around the Y-axis and the rotation angle of the i-th camera around the Z-axis in sequence

[0018] A4, inputting into a low-pass filter to remove a high-frequency noise wave band, obtain a low-pass filtered attitude parameter space, and obtain and wherein represents the tilt angle of the i-th camera in the vertical plane in the camera three-dimensional deflection coordinate system, represents the tilt angle of the i-th camera in the horizontal plane in the camera three-dimensional deflection coordinate system,​ This represents the orientation angle of the i-th camera in the horizontal plane, i.e., the rotation angle around the Z-axis;

[0019] A5, will and The input is fed into a complementary filter, and the attitude parameter points corresponding to the minimum covariance in the complementary filter are fused to obtain the fused camera attitude calibration parameter space.

[0020] Specifically, the steps for obtaining the attitude deviation angle of a dynamic camera include:

[0021] B1. Acquire the target image and the corresponding reference image and the corresponding camera pose parameter space, and preprocess the acquired image;

[0022] B2. Input the preprocessed image into the image segmentation model with a built-in scale bar to obtain the visible state volume space of the target object in the image. Based on the current visible size ratio of the target object in the visible state volume space, construct the invisible state volume space of the target object through the scale bar algorithm.

[0023] B3. Based on the visible state volume space of the target object, obtain the Euler angles corresponding to the current camera shooting through camera parameters and inverse perspective projection model. Combine the invisible state volume space of the target object with the visible state volume space of the target object, and use the combined target object state volume space to obtain the ideal Euler angles for camera shooting and the corresponding ideal camera pose parameter space.

[0024] B4. Using the Euler angles captured by the current camera and the ideal Euler angles captured by the camera, obtain the camera attitude deviation angle at the current moment.

[0025] Specifically, the steps for obtaining the attitude deviation angle of the dynamic camera also include:

[0026] B5. Configure a target tracking algorithm on the camera to obtain the trajectory video or image data of the target object at different times within the camera's monitoring range. Repeat the process B1-B3 to obtain the Euler angles corresponding to the camera's shooting at each time within the current camera's monitoring range, the ideal Euler angles of the camera's shooting, and the corresponding ideal camera attitude parameter space.

[0027] B6. Using the Euler angles corresponding to the camera's capture at each moment within the current camera's monitoring range and the ideal Euler angle captured by the camera, obtain the dynamic camera attitude deviation angle sequence θt within the current camera's monitoring range, t=1…T, where T represents the length of time the target object moves under the current camera.

[0028] B7, construct a trajectory-pose prediction model, and input the trajectory video or image data corresponding to the target object at different time and the corresponding camera ideal pose parameter space into the trajectory-pose prediction model for training, and obtain the camera ideal pose parameter space at the next time;

[0029] B8, when the camera ideal pose parameter space at the next time and the corresponding real camera ideal pose parameter space have an error of 0, a trained trajectory-pose prediction model is obtained.

[0030] Specifically, the specific steps of training the target pose self-calibration model include:

[0031] C1, input the dynamic camera pose deviation angle sequence, the corresponding camera ideal pose parameter space and the camera pose calibration parameter space into a linear function to obtain a quadratic calibration fitting function;

[0032] C2, construct a target pose self-calibration model based on a particle swarm algorithm, and take the quadratic calibration fitting function as a fitness function of the target pose self-calibration model;

[0033] C3, input the camera ideal pose parameter space and the camera pose calibration parameter space into the target pose self-calibration model for training, and when the camera ideal pose parameter space and the camera pose calibration parameter space correspond to the camera pose deviation angle for n continuous training periods, a trained target pose self-calibration model is obtained;

[0034] C4, obtain a real-time camera pose calibration parameter space through the processes A1-A5, and input the obtained real-time camera pose calibration parameter space into the trained target pose self-calibration model to obtain a camera pose calibration parameter space after secondary calibration corresponding to the current time;

[0035] C5, input the camera pose calibration parameter space at the current time and the camera pose calibration parameter space after secondary calibration corresponding to the current time into the trajectory-pose prediction model to obtain a camera pose calibration parameter space after secondary pre-calibration at the next time.

[0036] Specifically, the steps of constructing the reinforcement adjustment model include:

[0037] D1, construct an input state at the current time based on the camera pose calibration parameter space after secondary pre-calibration at the next time, the camera pose calibration parameter space after secondary calibration corresponding to the current time and the camera pose deviation angle;

[0038] D2, construct a trigger information at the current time based on the input state at the current time, including:

[0039] When the current time does not receive a manual input instruction and the camera pose deviation angle is 0, the first execution action information is triggered, that is, the current camera pose calibration parameter space is kept for camera shooting;

[0040] When the current time does not receive a manual input instruction and the camera pose deviation angle is not 0, the second execution action information is triggered, that is, the target pose self-calibration model and the trajectory-pose prediction model are called to perform secondary calibration on the current camera pose calibration parameter space, and the camera pose calibration parameter space at the next time is pre-calibrated, so that the camera pose deviation angle at the continuous time point is 0;

[0041] When the manually input pose adjustment parameter is received, the third execution action information is triggered, that is, the manually input pose adjustment parameter is used as the first priority adjustment parameter, and the target pose self-calibration model is called to perform secondary calibration on the current camera pose calibration parameter space using the manually input pose adjustment parameter, to obtain the manually secondary calibrated camera pose calibration parameter space, and the camera pose is calibrated;

[0042] D3, based on the current time trigger information, the current time execution action at=(at1, at2, at3) is constructed, wherein at1 represents the action corresponding to the trigger of the first execution action information, at2 represents the action corresponding to the trigger of the second execution action information, and at3 represents the action corresponding to the trigger of the third execution action information.

[0043] The camera pose automatic calibration method based on multi-sensor fusion, the steps comprising:

[0044] S1, in the constructed three-dimensional deflection coordinate system, the camera target shooting object video or image data and the corresponding camera multi-dimensional deflection state information data are obtained, and the configured distributed filtering model is used for preprocessing;

[0045] S2, using the preprocessed camera image and camera internal-external parameter, the dynamic camera pose deviation angle is obtained, and using the preprocessed camera multi-dimensional deflection state information data, the camera pose calibration parameter space is calculated;

[0046] S3, using the camera pose calibration parameter space and the dynamic camera pose deviation angle, the camera pose adjustment prediction function is obtained by linear function fitting, and the camera pose adjustment prediction function is configured into the target pose self-calibration model constructed by the particle swarm algorithm for training, and the camera pose calibration parameter space is pre-calibrated, to obtain the trained target pose self-calibration model;

[0047] S4, the reinforcement adjustment model is constructed and trained, and the reinforcement adjustment model and the target attitude self-calibration model are integrated into the camera attitude adjustment subsystem, and the camera attitude at the current time is calibrated in real time according to the target shooting video and the corresponding multi-dimensional deflection state information data acquired in real time or the manually input attitude adjustment parameters, and the camera attitude at the current time is calibrated in real time through the camera attitude adjustment subsystem.

[0048] A computer readable storage medium, having stored thereon computer instructions, when executed, perform a camera attitude automatic calibration method based on multi-sensor fusion.

[0049] Compared with the prior art, the beneficial effects of the present application are:

[0050] The present application aims at the deficiencies of the prior art, and the angular velocity data obtained by the gyroscope is fused and supplemented to the traditional single accelerometer or magnetometer data to construct a camera attitude calibration parameter space, thereby enhancing the capture of subtle camera movement information by the system. Secondly, the dynamic camera attitude deviation angle is obtained through the target shooting object video data, and the camera attitude calibration parameter space is subjected to secondary pre-attitude calibration by using the dynamic camera attitude deviation angle, so that the camera can be adjusted in real time according to the target shooting object movement track. In the case of reducing the camera movement delay, the robustness of the camera attitude adjustment is enhanced, so that the camera can quickly and accurately calibrate the camera attitude in real time under complex and variable external environments. BRIEF DESCRIPTION OF DRAWINGS

[0051] Figure 1 Figure 1 is a module diagram of the camera attitude automatic calibration system based on multi-sensor fusion of the embodiment 1 of the present application;

[0052] Figure 2 Figure 2 is a corresponding unit flow chart of the camera attitude automatic calibration system based on multi-sensor fusion of the embodiment 2 of the present application;

[0053] Figure 3 Figure 3 is a flow chart of the camera attitude automatic calibration method based on multi-sensor fusion of the embodiment 3 of the present application. DETAILED DESCRIPTION

[0054] Embodiment 1

[0055] Please refer to Figure 1 An embodiment provided by the present application is a camera attitude automatic calibration system based on multi-sensor fusion, which comprises a data acquisition module, a data self-processing module, an attitude self-calibration module and a user interface module.

[0056] The data acquisition module is configured to acquire video and image data of different angles and distances on the highway and to perform preprocessing; the data acquisition module comprises a spatial coordinate unit and a data acquisition unit;

[0057] The spatial coordinate unit is configured to construct a three-dimensional deflection coordinate system of the camera based on a deflection support point of the camera installed on the highway as an origin;

[0058] The data acquisition unit is configured to acquire video or image data of a target object and corresponding camera multi-dimensional deflection state information data based on the three-dimensional deflection coordinate system of the camera, and to perform synchronous filtering preprocessing on the camera image and the multi-dimensional deflection state information data by using a distributed filtering model configured in each sensor;

[0059] Further, in the embodiment, the camera multi-dimensional deflection state information data comprises: X-axis angular velocity ωx of the camera in the three-dimensional deflection coordinate system of the camera acquired by a gyroscope, Y-axis angular velocity ωy of the camera in the three-dimensional deflection coordinate system of the camera acquired by the gyroscope, and Z-axis angular velocity ωz of the camera in the three-dimensional deflection coordinate system of the camera acquired by the gyroscope. ωx represents a rotation rate of the camera around the X-axis, ωy represents a rotation rate of the camera around the Y-axis, and ωz represents a rotation rate of the camera around the Z-axis.

[0060] X-axis acceleration ax of the camera in the three-dimensional deflection coordinate system of the camera acquired by an accelerometer, Y-axis acceleration ay of the camera in the three-dimensional deflection coordinate system of the camera acquired by the accelerometer, and Z-axis acceleration az of the camera in the three-dimensional deflection coordinate system of the camera acquired by the accelerometer.

[0061] X-axis magnetic field component mx of the camera in the three-dimensional deflection coordinate system of the camera acquired by a magnetometer, Y-axis magnetic field component my of the camera in the three-dimensional deflection coordinate system of the camera acquired by the magnetometer, and Z-axis magnetic field component mz of the camera in the three-dimensional deflection coordinate system of the camera acquired by the magnetometer.

[0062] The data self-processing module is configured to process the acquired image or video data to obtain a target shooting deflection angle and a corresponding attitude deflection adjustment space; the data self-processing module comprises a target calculation unit, an attitude space parameter unit, and an adjustment prediction unit.

[0063] The target calculation unit is configured to obtain a dynamic camera attitude deflection angle based on acquired target shooting image, target reference frame image, and camera internal-external parameters.

[0064] Further, the step of obtaining the dynamic camera attitude deflection angle in the embodiment comprises:

[0065] B1, acquiring target shooting image, corresponding reference image, and corresponding camera attitude parameter space, and preprocessing the acquired image;

[0066] ​​B2, input the pre-processed image into the image segmentation model with a scale, obtain the target object visible state volume space in the image, and through the current visible size ratio of the target object in the visible state volume space, the invisible state volume space of the target object is constructed through the scale algorithm;

[0067] B3, according to the target object visible state volume space, the current camera shooting corresponding Euler angle is obtained through the camera parameter and the inverse perspective projection model, the invisible state volume space of the target object and the target object visible state volume space are combined, and the camera ideal pose parameter space is obtained by using the combined target object state volume space;

[0068] B4, using the current camera shooting corresponding Euler angle and the camera ideal Euler angle, the current camera pose deviation angle corresponding to the time is obtained;

[0069] B5, configure the target tracking algorithm in the camera, obtain the corresponding trajectory video or image data of the target object in the camera monitoring range at different times, repeat the process of B1-B3, and obtain the corresponding Euler angle and the camera ideal pose parameter space of the camera shooting at each time in the current camera monitoring range;

[0070] B6, using the current camera monitoring range at each time, the camera shooting corresponding Euler angle and the camera ideal Euler angle are obtained, and the dynamic camera pose deviation angle sequence θt, t = 1…T is obtained in the current camera monitoring range, wherein T represents the time length of the target object moving under the current camera;

[0071] B7, construct a trajectory-pose prediction model, input the corresponding trajectory video or image data of the target object at different times and the corresponding camera ideal pose parameter space into the trajectory-pose prediction model for training, and obtain the camera ideal pose parameter space at the next time;

[0072] B8, when the next time camera ideal pose parameter space and the corresponding real camera ideal pose parameter space error is 0, the trained trajectory-pose prediction model is obtained.

[0073] Further, the trajectory-pose prediction model in the embodiment is constructed by a trajectory tracking algorithm, which is used to predict the trajectory position point of the target object at the next time according to the trajectory information of the target object at the historical time and the current time in the current camera visible range, so as to obtain the camera pose deviation angle of the camera pose calibration parameter space corresponding to the current time of the camera at each future time and the camera ideal pose parameter space, so as to make pre-adjustment, further reduce the time delay of camera calculation, and improve the reaction rate of the camera.

[0074] This process involves acquiring and preprocessing both the target image and a reference image to ensure image quality. Then, using an image segmentation model with a built-in scale, the visible and invisible volumetric spaces of the target are accurately calculated. An inverse perspective projection model is used to obtain the Euler angles corresponding to the current camera movement and the ideal Euler angles. Next, a target tracking algorithm acquires the target trajectory data at different times, constructing a dynamic camera attitude deviation angle sequence. Finally, a trajectory-attitude prediction model is built and trained to obtain the ideal camera attitude parameter space for the next moment, ensuring that the attitude deviation angle is zero. This process significantly improves the accuracy and real-time performance of camera attitude estimation, reduces attitude deviation, and enhances the robustness and stability of the system. Especially in dynamic environments, it enables real-time adjustment of the camera attitude, ensuring stable tracking and high-quality display of the target.

[0075] The attitude space parameter unit is used to obtain the camera attitude parameter space based on the camera's multidimensional deflection state information data, and to construct the camera attitude calibration parameter space using the camera target attitude parameter space.

[0076] Furthermore, in this embodiment, the step of obtaining the camera attitude calibration parameter space includes:

[0077] A 1. Construct the camera attitude parameter space based on the camera's multi-dimensional deflection state information data.

[0078]

[0079] A2. Transfer the camera pose parameter space to... The input is fed into a high-pass filter to remove the low-frequency drift band, thus obtaining the high-pass filter attitude parameter space;

[0080] A3. By integrating the parameters in the high-pass filter attitude parameter space, the angular offset of the camera in the camera's three-dimensional deflection coordinate system within a fixed time period is obtained. in In order to represent The rotation angles around the X-axis, Y-axis, and Z-axis of the i-th camera are obtained by corresponding integral calculations.

[0081] A4, will The input is fed into a low-pass filter to remove high-frequency noise bands, obtaining a low-pass filtered attitude parameter space. Then, using this low-pass filtered attitude parameter space, the following is obtained: and in This represents the tilt angle of the i-th camera in the vertical plane within the camera's three-dimensional tilt coordinate system. represents the tilt angle of the i-th camera in the horizontal plane in the camera three-dimensional deflection coordinate system, represents the orientation angle of the i-th camera in the horizontal plane, i.e. the rotation angle around the Z axis;

[0082] A5, the and input into the complementary filter, and the attitude parameter point corresponding to the minimum covariance in the complementary filter is fused to obtain the fused camera attitude calibration parameter space

[0083] The process uses multi-dimensional deflection state information data obtained by the gyroscope, accelerometer and magnetometer to construct the camera attitude parameter space. Then, the low-frequency drift is removed by the high-pass filter to obtain the high-pass filtered attitude parameter space; the angle offset in a fixed time period is calculated by integration to ensure the dynamic accuracy of the attitude parameter; secondly, the high-frequency noise is removed by the low-pass filter to obtain the low-pass filtered attitude parameter space, and the tilt angle and the orientation angle of the camera in the three-dimensional deflection coordinate system are calculated; finally, the results of the high-pass filtering and the low-pass filtering are input into the complementary filter, and the attitude parameter point corresponding to the minimum covariance is selected for fusion to obtain the fused camera attitude calibration parameter space; the accuracy and stability of the attitude estimation are further improved, the influence of low-frequency drift and high-frequency noise is effectively eliminated, and the fusion of gyroscope data further improves the sensitivity of the camera attitude calibration parameter space constructed by the Euler angle calculated by the accelerometer and the magnetometer to external vibration and impact, ensuring the stability and robustness of the camera in a dynamic environment.

[0084] The adjustment prediction unit is configured to obtain a camera attitude adjustment prediction function through linear function fitting according to the camera attitude calibration parameter space and the dynamic camera attitude deviation angle, configure the camera attitude adjustment prediction function into a target attitude self-calibration model constructed by a particle swarm algorithm for training, obtain a trained target attitude self-calibration model, and use the dynamic camera attitude deviation angle to pre-calibrate the camera attitude calibration parameter space through the target attitude self-calibration model;

[0085] Further, the specific steps of training the target attitude self-calibration model in the embodiment include:

[0086] C1, input the dynamic camera attitude deviation angle sequence and the corresponding camera ideal attitude parameter space and camera attitude calibration parameter space into a linear function to obtain a secondary calibration fitting function;

[0087] C2, construct a target attitude self-calibration model based on a particle swarm algorithm, and use the secondary calibration fitting function as the fitness function of the target attitude self-calibration model;

[0088] C3, input the camera ideal pose parameter space and the camera pose calibration parameter space into the target pose self-calibration model for training, when the camera ideal pose parameter space and the camera pose calibration parameter space correspond to the camera pose deviation angle for 0 in the continuous n training periods, the training completed target pose self-calibration model is obtained;

[0089] C4, the real-time camera pose calibration parameter space is obtained through the A1-A5 process, and the obtained real-time camera pose calibration parameter space is input into the training completed target pose self-calibration model, and the corresponding secondary calibrated camera pose calibration parameter space at the current time is obtained;

[0090] C5, the camera pose calibration parameter space at the current time and the corresponding secondary calibrated camera pose calibration parameter space at the current time are input into the trajectory-pose prediction model, and the secondary pre-calibrated camera pose calibration parameter space at the next time is obtained.

[0091] The process realizes the synchronous high-precision acquisition and filtering preprocessing of the camera multi-dimensional deflection state information data and the target shooting object video or image data through the highly integrated data acquisition and processing module, effectively reduces the noise interference, and improves the data quality; at the same time, the data obtained by the accelerometer and the magnetometer is fused and enhanced by using the angular velocity data obtained by the gyroscope, and the anti-interference ability of the camera pose calibration parameter space constructed is further enhanced; secondly, the target calculation unit uses advanced image segmentation, inverse perspective projection and scale algorithm to accurately calculate the dynamic camera pose deviation angle, and constructs the camera ideal pose parameter space, which provides an accurate target reference for subsequent pose calibration; at the same time, the pose space parameter unit removes the low-frequency drift and high-frequency noise through multi-level filtering processing such as high-pass filtering, low-pass filtering and complementary filtering, and obtains the high-precision camera pose calibration parameter space; on this basis, the adjustment and prediction unit uses linear function fitting and particle swarm algorithm to construct the target pose self-calibration model, realizes the real-time adjustment and prediction of the camera pose and the secondary pre-calibration, and significantly improves the accuracy and real-time performance of the camera pose.

[0092] The user interface module is used for real-time display of the camera shooting result and input of the manual pose adjustment parameter.

[0093] The pose self-calibration module is used for self-calibration of the camera pose according to the real-time acquired image or input instruction data; the pose self-calibration module includes a distributed model unit and an integrated self-calibration unit.

[0094] The distributed model unit is configured to construct a reinforcement adjustment model, and train the reinforcement adjustment model by using the obtained target shooting image and target reference frame image, camera internal-external parameters and manually input parameters, and deploy the trained reinforcement adjustment model to the pose adjustment subsystem corresponding to each camera.

[0095] Further, the step of constructing the reinforcement adjustment model in the embodiment includes:

[0096] D1, constructing an input state at a current time based on a camera pose calibration parameter space after secondary pre-position calibration at a next time, a camera pose calibration parameter space after secondary calibration corresponding to the current time, and a camera pose deviation angle, wherein wherein represents a camera pose calibration parameter space after secondary calibration corresponding to the i-th camera at the current t time, represents a camera pose calibration parameter space after secondary pre-position calibration of the i-th camera at t+1 time, represents a pose deviation angle of the i-th camera at the current t time.

[0097] D2, constructing trigger information at the current time based on the input state at the current time, including:

[0098] When no manual input instruction is received at the current time and the camera pose deviation angle is 0, a first execution action information is triggered, i.e., maintaining the current camera pose calibration parameter space for camera shooting;

[0099] When no manual input instruction is received at the current time and the camera pose deviation angle is not 0, a second execution action information is triggered, i.e., calling a target pose self-calibration model and a trajectory-pose prediction model to perform secondary calibration on the current camera pose calibration parameter space, and pre-position calibration on the camera pose calibration parameter space at the next time, so that the camera pose deviation angle is 0 at continuous time points;

[0100] When a manually input pose adjustment parameter is received, a third execution action information is triggered, i.e., taking the manually input pose adjustment parameter as a first priority adjustment parameter, and calling the target pose self-calibration model to perform secondary calibration on the current camera pose calibration parameter space by using the manually input pose adjustment parameter, to obtain a manually calibrated camera pose calibration parameter space after secondary calibration, and calibrate the camera pose;

[0101] D3, constructing an execution action at the current time at=(at1, at2, at3) based on the trigger information at the current time, wherein at1 represents an action executed by triggering the first execution action information, at2 represents an action executed by triggering the second execution action information, and at3 represents an action executed by triggering the third execution action information.

[0102] D4, construct a reinforcement adjustment model based on the SAC algorithm, and input the constructed current moment input state, current moment trigger information and current moment execution action into the reinforcement adjustment model for training to obtain the trained reinforcement adjustment model;

[0103] D5, configure the trained reinforcement adjustment model into each camera, and automatically calibrate the camera pose according to the video data acquired by the camera in real time.

[0104] The integrated self-calibration unit is used for integrating the reinforcement adjustment model and the target pose self-calibration model, and real-time calibration of the current moment camera pose is performed on the basis of the real-time acquired target shooting video or image, multi-dimensional deflection state information data or manually input pose adjustment parameters.

[0105] The process realizes real-time display of the camera shooting result and flexible input of the manual pose adjustment parameter by integrating the user interface module and the pose self-calibration module, and significantly improves the convenience of user operation and the accuracy of camera pose adjustment; the reinforcement adjustment model constructed by the distributed model unit can be efficiently trained on the basis of real-time data and manual input parameters, so that each camera can obtain a personalized pose adjustment strategy; meanwhile, the integrated self-calibration unit organically combines the reinforcement adjustment model and the target pose self-calibration model, realizes real-time and accurate calibration of the camera pose, and effectively reduces the shooting error caused by the pose deviation.

[0106] Embodiment 2

[0107] Please refer to Figure 2 Another embodiment provided by the application: the camera pose automatic calibration system based on multi-sensor fusion corresponds to a unit workflow, which comprises:

[0108] First, a three-dimensional deflection coordinate system of the camera is constructed by the space coordinate unit, and video or image data of a target shooting object and corresponding multi-dimensional deflection state information data of the camera are acquired by the data acquisition unit on the basis of the three-dimensional deflection coordinate system of the camera, and the camera image and the multi-dimensional deflection state information data are subjected to real-time synchronous filtering pretreatment by using the distributed filtering model configured in each sensor;

[0109] Second, the pretreated camera image is input into the target calculation unit to obtain a dynamic camera pose deviation angle, and the pretreated multi-dimensional deflection state information data is input into the pose space parameter unit to obtain a camera pose calibration parameter space; the camera pose calibration parameter space and the dynamic camera pose deviation angle are input into the adjustment prediction unit to obtain a target pose self-calibration model, and the camera pose calibration parameter space is pre-calibrated by the target pose self-calibration model through the dynamic camera pose deviation angle.

[0110] Thirdly, the target shooting image and the target reference frame image, the camera internal-external parameter and the posture adjustment parameter manually input through the user interface module are input into the distributed model unit, a reinforced adjustment model is obtained, the reinforced adjustment model and the target posture self-calibration model are integrated into the posture adjustment subsystem of the corresponding camera through the integrated self-calibration unit, and the posture of the camera at the current time is calibrated in real time by using the target shooting video or image and the corresponding multi-dimensional deflection state information data acquired in real time or the posture adjustment parameter manually input, and the calibrated video is displayed in real time through the user interface module.

[0111] The system constructs a three-dimensional deflection coordinate system of the camera through the space coordinate unit, acquires video or image data and multi-dimensional deflection state information data of a target shooting object through the data acquisition unit, and performs real-time synchronous filtering preprocessing through the distributed filtering model, so that noise and drift are effectively removed, and the accuracy and stability of the data are improved. Secondly, the preprocessed image is input into the target calculation unit to obtain a dynamic camera posture deviation angle, and the multi-dimensional deflection state information data is input into the posture space parameter unit to obtain a camera posture calibration parameter space. Through the adjustment prediction unit, the dynamic camera posture deviation angle and the posture calibration parameter space are used to obtain a target posture self-calibration model, and pre-calibration is performed to ensure the accuracy and real-time performance of the posture estimation. Finally, the target shooting image, the target reference frame image, the camera internal-external parameter and the manually input posture adjustment parameter are input into the distributed model unit to obtain a reinforced adjustment model, and the reinforced adjustment model and the target posture self-calibration model are integrated into the posture adjustment subsystem of the camera through the integrated self-calibration unit, so that real-time calibration of the posture of the camera at the current time is realized. The whole process significantly improves the accuracy and real-time performance of the camera posture calibration, and ensures the stability and robustness in a dynamic environment.

[0112] Embodiment 3

[0113] Please refer to Figure 3 Another embodiment provided by the application is a camera posture automatic calibration method based on multi-sensor fusion, and the steps include:

[0114] S1, acquiring camera target shooting object video or image data and corresponding camera multi-dimensional deflection state information data in the constructed three-dimensional deflection coordinate system, and preprocessing through a configured distributed filtering model;

[0115] S2, using the preprocessed camera image and the camera internal-external parameter to obtain a dynamic camera posture deviation angle, and using the preprocessed camera multi-dimensional deflection state information data to calculate a camera posture calibration parameter space;

[0116] S3, using the camera pose calibration parameter space and the dynamic camera pose deviation angle, a camera pose adjustment prediction function is obtained through linear function fitting, and the camera pose adjustment prediction function is configured into a target pose self-calibration model constructed by a particle swarm algorithm for training, and the camera pose calibration parameter space is subjected to secondary pre-calibration to obtain a trained target pose self-calibration model;

[0117] S4, the reinforcement adjustment model is constructed and trained, and the reinforcement adjustment model and the target pose self-calibration model are integrated into the camera pose adjustment subsystem, and the camera pose is calibrated in real time according to the real-time acquired target shooting video and the corresponding multi-dimensional deflection state information data or the manually input pose adjustment parameters, and the camera pose is calibrated in real time by the camera pose adjustment subsystem.

[0118] The embodiments of the present application are described above in combination with the drawings, but the present application is not limited to the above-mentioned specific embodiments, and the above-mentioned specific embodiments are only illustrative and not restrictive, and those skilled in the art can make changes, modifications, replacements and variations to the above-mentioned embodiments without departing from the purpose of the present application and the scope protected by the claims under the inspiration of the present application, which are all within the protection of the present application.

[0119] If the technical solution of the present disclosure involves personal information, the product applying the technical solution of the present disclosure has been explicitly informed of the personal information processing rules before processing the personal information and has obtained the personal independent consent. If the technical solution of the present disclosure involves sensitive personal information, the product applying the technical solution of the present disclosure has obtained the personal independent consent before processing the sensitive personal information and at the same time meets the requirement of "explicit consent". For example, at the personal information collection device such as camera, a clear and prominent mark is set to inform that the personal information collection range has been entered and the personal information will be collected, and if the individual voluntarily enters the collection range, it is considered to agree to collect the personal information; or on the device for processing personal information, the personal information processing rules are informed through obvious marks / information, and the personal authorization is obtained through pop-up information or by uploading the personal information by the individual; wherein, the personal information processing rules can include personal information processor, personal information processing purpose, processing method and personal information type, etc.

Claims

1. A camera pose auto-calibration system based on multi-sensor fusion, characterized in that, include: The system comprises a data acquisition module, a data self-processing module, an attitude self-calibration module, and a user interface module. The data acquisition module includes a data acquisition unit; The data acquisition unit acquires video or image data of the target object and corresponding multi-dimensional deflection status information data of the camera in the configured three-dimensional deflection coordinate system of the camera, and performs synchronous filtering preprocessing on the acquired data through the configured distributed filtering model. The data self-processing module includes a target calculation unit, an attitude space parameter unit, and an adjustment prediction unit; it inputs the pre-processed target video or image data into the target calculation unit to obtain the dynamic camera attitude deviation angle; at the same time, it inputs the camera multi-dimensional deflection state information data into the attitude space parameter unit to obtain the camera attitude calibration parameter space. The dynamic camera attitude deviation angle and the camera attitude calibration parameter space are input into the adjustment prediction unit to obtain the trained target attitude self-calibration model and the camera attitude calibration parameter space after secondary pre-calibration. The posture self-calibration module comprises a distributed model unit and an integrated self-calibration unit; a reinforced adjustment model is constructed and trained through the distributed model unit, and the reinforced adjustment model and a target posture self-calibration model are integrated into a posture adjustment subsystem corresponding to each camera through the integrated self-calibration unit, while the posture of the camera at the current time is calibrated in real time according to target shooting video and multi-dimensional deflection state information data acquired in real time or posture adjustment parameters manually input through a user interface module; the multi-dimensional deflection state information data of the camera comprises X-axis angular velocity , Y-axis angular velocity , and Z-axis angular velocity of the camera in a three-dimensional deflection coordinate system of the camera acquired by a gyroscope; X-axis acceleration , Y-axis acceleration , and Z-axis acceleration of the camera in the three-dimensional deflection coordinate system of the camera acquired by an accelerometer; X-axis magnetic field component , Y-axis magnetic field component , and Z-axis magnetic field component of the camera in the three-dimensional deflection coordinate system of the camera acquired by a magnetometer; The specific steps for training the target attitude self-calibration model include: C1. Input the dynamic camera attitude deviation angle sequence and the corresponding ideal camera attitude parameter space and camera attitude calibration parameter space into the linear function to obtain the quadratic calibration fitting function. C2. Construct a target attitude self-calibration model based on particle swarm optimization algorithm, and use the quadratic calibration fitting function as the fitness function of the target attitude self-calibration model. C3. Input the ideal camera attitude parameter space and the camera attitude calibration parameter space into the target attitude self-calibration model for training. When the camera attitude deviation angle corresponding to the ideal camera attitude parameter space and the camera attitude calibration parameter space is 0 for n consecutive training cycles, the trained target attitude self-calibration model is obtained. C4. Obtain the real-time camera attitude calibration parameter space and input the obtained real-time camera attitude calibration parameter space into the trained target attitude self-calibration model to obtain the camera attitude calibration parameter space after secondary calibration at the current moment. C5. Input the current camera attitude calibration parameter space and the corresponding secondary calibration camera attitude calibration parameter space into the trajectory-attitude prediction model to obtain the secondary pre-calibration camera attitude calibration parameter space at the next moment. The steps for constructing the reinforcement adjustment model include: D1. Construct the current input state based on the camera attitude calibration parameter space after the second pre-calibration at the next time step, the camera attitude calibration parameter space after the second calibration at the current time step, and the camera attitude deviation angle. D2. Construct the trigger information for the current moment based on the input state at the current moment, including: When no manual input command is received at the current moment and the camera attitude deviation angle is 0, the first execution action information is triggered, that is, the camera is shooting while maintaining the current camera attitude calibration parameter space; When the current time does not receive manual input instruction and the camera pose deviation angle is not 0, the second execution action information is triggered, that is, the target pose self-calibration model and the trajectory-pose prediction model are called to perform secondary calibration on the current camera pose calibration parameter space, and the next time camera pose calibration parameter space is pre-calibrated, so that the camera pose deviation angle is 0 at the continuous time point; When the manually input pose adjustment parameter is received, the third execution action information is triggered, that is, the manually input pose adjustment parameter is used as the first priority adjustment parameter, and the target pose self-calibration model is called to perform secondary calibration on the current camera pose calibration parameter space by using the manually input pose adjustment parameter, so as to obtain the manually secondary calibrated camera pose calibration parameter space and calibrate the camera pose; D3, constructing a current time execution action based on current time trigger information wherein, indicates a triggered action corresponding to the first execution action information, indicates a triggered action corresponding to the second execution action information, indicates a triggered action corresponding to the third execution action information.

2. The multi-sensor fusion based camera pose auto-calibration system of claim 1, wherein, The step of obtaining the camera pose calibration parameter space comprises: A1, constructing the camera pose parameter space according to the camera multi-dimensional deflection state information data ； A2, the camera pose parameter space in Input into a high-pass filter, remove the low-frequency drift band, get high-pass filter pose parameter space; A3, the parameters in the high-pass filtered attitude parameter space are calculated by integration to obtain the angular displacement of the camera in the three-dimensional camera deflection coordinate system within a fixed time period wherein denote in sequence , , the rotation angle of the i-th camera around the X-axis, the rotation angle of the i-th camera around the Y-axis, and the rotation angle of the i-th camera around the Z-axis obtained by integration A4、will be input to a low-pass filter to remove high-frequency noise bands, obtain a low-pass filtered attitude parameter space, and use the low-pass filtered attitude parameter space to obtain 、 and wherein represents the tilt angle of the i-th camera in the vertical plane in the camera three-dimensional deflection coordinate system, represents the tilt angle of the i-th camera in the horizontal plane in the camera three-dimensional deflection coordinate system, represents the orientation angle of the i-th camera in the horizontal plane, i.e. the rotation angle around the Z axis. A5、will and Input into the complementary filter, and use the minimum covariance corresponding to the attitude parameter point in the complementary filter to fuse, obtain the fused camera attitude calibration parameter space .

3. The multi-sensor fusion based camera pose auto-calibration system of claim 2, wherein, The step of obtaining the dynamic camera pose deviation angle comprises: B1, obtaining the target shooting image, the corresponding reference image and the corresponding camera pose parameter space, and preprocessing the obtained image; B2, inputting the preprocessed image into the image segmentation model with the built-in scale to obtain the target shooting object visible state volume space, and constructing the invisible state volume space of the target shooting object by using the current visible size ratio of the target shooting object in the visible state volume space and the scale algorithm; B3, obtaining the current camera shooting corresponding Euler angle according to the target shooting object visible state volume space, the camera parameters and the inverse perspective projection model, combining the invisible state volume space of the target shooting object and the visible state volume space of the target shooting object, and obtaining the ideal Euler angle of the camera shooting and the corresponding ideal camera pose parameter space by using the combined target shooting object state volume space; B4, obtaining the corresponding camera pose deviation angle at the current time by using the current camera shooting corresponding Euler angle and the ideal Euler angle of the camera shooting.

4. The multi-sensor fusion based camera pose auto-calibration system of claim 3, wherein, The step of obtaining the dynamic camera pose deviation angle further comprises: B5, configuring the target tracking algorithm for the camera, obtaining the corresponding trajectory video or image data of the target shooting object at different times in the camera monitoring range, repeating the processes of B1-B3 to obtain the corresponding Euler angle of the camera shooting at each time in the current camera monitoring range, the ideal Euler angle of the camera shooting and the corresponding ideal camera pose parameter space; B6, using the current camera monitoring range each time camera shooting corresponding Euler angle and camera shooting ideal Euler angle, obtain the current camera monitoring range dynamic camera pose deviation angle sequence , wherein T represents the time length of the target object moving under the current camera; B7, constructing the trajectory-pose prediction model, inputting the corresponding trajectory video or image data of the target shooting object at different times and the corresponding ideal camera pose parameter space into the trajectory-pose prediction model for training, and obtaining the next time ideal camera pose parameter space; B8, when the next time ideal camera pose parameter space and the corresponding real ideal camera pose parameter space have an error of 0, the trained trajectory-pose prediction model is obtained.

5. A camera pose automatic calibration method based on multi-sensor fusion, implemented based on the camera pose automatic calibration system based on multi-sensor fusion in any one of claims 1-4, characterized in that the steps of Comprise: S1, obtain camera target shooting video or image data and corresponding camera multi-dimensional deflection state information data in the constructed three-dimensional deflection coordinate system, and pre-process through the configured distributed filtering model; S2, obtain dynamic camera pose deviation angle by using the pre-processed camera image and camera internal-external parameter, and calculate camera pose calibration parameter space by using the pre-processed camera multi-dimensional deflection state information data; S3, obtain camera pose adjustment prediction function by linear function fitting by using the camera pose calibration parameter space and the dynamic camera pose deviation angle, configure the camera pose adjustment prediction function into the target pose self-calibration model constructed by the particle swarm algorithm for training, and perform secondary pre-position calibration on the camera pose calibration parameter space to obtain the trained target pose self-calibration model; S4, construct and train the reinforcement adjustment model, integrate the reinforcement adjustment model and the target pose self-calibration model into the camera pose adjustment subsystem, and calibrate the camera pose in real time according to the real-time obtained target shooting video and corresponding multi-dimensional deflection state information data or manually input pose adjustment parameter, and calibrate the camera pose in real time through the camera pose adjustment subsystem.

6. A computer-readable storage medium, characterized in that, The computer instructions are stored thereon, and when the computer instructions are executed, the method for automatically calibrating camera pose based on multi-sensor fusion in claim 5 is executed.

Images