A multi-mode radar and visual integrated machine dynamic verification automatic calibration method and device

Abstract

This application provides a method and apparatus for dynamic verification and automatic calibration of a multimodal radar-visual integrated machine, relating to the field of intelligent transportation technology. The method includes: real-time acquisition of feature points of a target reference point at a fixed location. The feature points are extracted from the target reference point's feature information, signal strength information, and position information of the target device relative to the target reference point. The target device can be any one of a high-definition camera, millimeter-wave radar, or thermal imaging camera in the multimodal radar-visual integrated machine; determining whether the target device needs calibration based on the feature points; and when calibration is required, acquiring first data, second data, and third data, where the first data is displacement data, the second data is attitude data, and the third data is temperature data; determining the calibration type of the target device based on the first, second, and third data, and calibrating the target device according to the calibration type. This application can improve the accuracy and efficiency of calibration.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of intelligent transportation, and in particular to a multi-modal radar and vision integrated machine dynamic verification automatic calibration method and device. BACKGROUND

[0002] Currently, in the expressway video monitoring system, the radar and vision integrated machine is challenged by complex traffic environment. Such as rain, snow, fog and haze weather, large diurnal temperature difference or mechanical vibration, etc. All of these are prone to cause the monitoring data of the radar and vision integrated machine to deviate. Once the monitoring data deviates, the radar and vision integrated machine must be maintained to ensure the accuracy of subsequent monitoring data. However, in the current maintenance process, the calibration parameters are prone to drift, which requires frequent manual calibration. Moreover, the radar and vision integrated machine lacks self-diagnosis function, which makes the maintenance efficiency very low. In addition, the existing radar and vision integrated machine also has the problem of lacking device state automatic monitoring function, which cannot actively warn of faults. SUMMARY

[0003] The present application provides a multi-modal radar and vision integrated machine dynamic verification automatic calibration method and device to solve the problems of low accuracy and low efficiency of the calibration results of the radar and vision integrated machine in the prior art.

[0004] In a first aspect, the present application provides a multi-modal radar and vision integrated machine dynamic verification automatic calibration method, comprising:

[0005] Real-time acquisition of feature points of a target reference point at a fixed position, the feature points being feature information, signal strength information and position information of the target reference point acquired by a target device, the target device being any device of a high-definition camera, a millimeter wave radar and a thermal imaging camera in a multi-modal radar and vision integrated machine;

[0006] According to the feature points, it is determined whether the target device needs to be calibrated, and when the target device needs to be calibrated, first data, second data and third data are acquired, the first data being displacement data collected by a displacement sensor, the second data being attitude data collected by an attitude sensor, and the third data being temperature data collected by a temperature sensor;

[0007] According to the first data, the second data and the third data, the calibration type of the target device is determined, and according to the calibration type, the calibration of the target device is realized.

[0008] In a second aspect, the present application provides a multi-modal radar and vision integrated machine dynamic verification automatic calibration device, comprising:

[0009] The feature point acquisition module is configured to acquire feature points of a target reference point at a fixed position in real time, the feature points being feature information, signal strength information and position information of the target device relative to the target reference point, which are acquired by the target device, the target device being any one of a high-definition camera, a millimeter wave radar and a thermal imaging camera in a multimodal radar and visual integrated machine.

[0010] The calibration judgment and data acquisition module is configured to judge whether the target device needs to be calibrated according to the feature points, and acquire first data, second data and third data when the target device needs to be calibrated, the first data being displacement data collected by a displacement sensor, the second data being attitude data collected by an attitude sensor, and the third data being temperature data collected by a temperature sensor.

[0011] The calibration module is configured to judge a calibration type of the target device according to the first data, the second data and the third data, and to realize calibration of the target device according to the calibration type.

[0012] The application provides a multi-modal radar and vision integrated machine dynamic verification automatic calibration method and device, feature points of a target reference point at a fixed position are acquired in real time, the feature points are feature information, signal strength information and position information of the target reference point acquired by a target device, and the target device is any device of a high-definition camera, a millimeter wave radar and a thermal imaging camera in the multi-modal radar and vision integrated machine; whether the target device needs to be calibrated is determined according to the feature points, and when the target device needs to be calibrated, first data, second data and third data are acquired, the first data is displacement data collected by a displacement sensor, the second data is attitude data collected by an attitude sensor, and the third data is temperature data collected by a temperature sensor; the calibration type of the target device is determined according to the first data, the second data and the third data, and calibration of the target device is realized according to the calibration type. The application acquires the feature points of the target reference point in real time, and determines whether the target device needs to be calibrated based on the feature points, the determination method based on the position information of the target reference point in the actual captured image can directly reflect the current working state and imaging or detection accuracy of the device, compared with the traditional periodic calibration or calibration based on experience, the method can more accurately capture the performance change of the device and timely find the situation that needs to be calibrated, thereby improving the pertinence and accuracy of calibration; and after it is determined that the target device needs to be calibrated, the displacement data collected by the displacement sensor, the attitude data collected by the attitude sensor and the temperature data collected by the temperature sensor are acquired, and the calibration type is determined by comprehensively considering the data, the displacement, the attitude and the temperature and other factors will affect the performance of each device (the high-definition camera, the millimeter wave radar and the thermal imaging camera) in the radar and vision integrated machine, by considering these multi-dimensional data, the reasons and degrees of the performance change of the device can be more accurately understood, and then the most suitable calibration type is selected, and the accuracy of calibration is further improved; meanwhile, the application can perform calibration determination and adjustment in real time during the operation of the device, and the device does not need to be frequently disassembled and transported, the maintenance cost and time are greatly reduced, and the usability and working efficiency of the device are improved. BRIEF DESCRIPTION OF DRAWINGS

[0013] In order to more clearly illustrate the technical solutions in the embodiments of the application, the following will briefly introduce the drawings needed to be used in the embodiments or prior art description. Obviously, the drawings in the following description only some embodiments of the application, and for those skilled in the art, other drawings can also be obtained without creative labor.

[0014] Figure 1 It is a flowchart of the multi-modal radar and vision integrated machine dynamic verification automatic calibration method provided by the embodiments of the application.

[0015] Figure 2 It is a structural schematic diagram of the multi-modal radar and vision integrated machine dynamic verification automatic calibration device provided by the embodiments of the application. Detailed Implementation

[0016] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods have been omitted so as not to obscure the description of this application with unnecessary detail.

[0017] To make the objectives, technical solutions, and advantages of this application clearer, the following description will be provided in conjunction with the accompanying drawings and specific embodiments.

[0018] To address the issues of low accuracy and inefficient calibration results of existing radar-visual integrated machines, this application provides an automatic dynamic calibration method for intelligent sensing and monitoring multimodal radar-visual integrated machines for smart transportation. Through multimodal feature point extraction, temperature drift compensation model, and vibration compensation model compensation mechanisms, it achieves fully automatic high-precision calibration of sensors such as high-definition cameras, millimeter-wave radar, and thermal imaging cameras.

[0019] Specifically, sensor feature points are extracted and matched based on the same coordinate system. Then, a temperature drift compensation model is used to correct calibration errors caused by changes in ambient temperature in real time, and a vibration compensation model is used to dynamically adjust calibration errors caused by mechanical vibration. In complex traffic environments (such as rain, snow, fog, haze, day and night temperature differences, and mechanical vibration), horizontal / pitch offset compensation, adaptive temperature drift correction, and real-time vibration suppression can be achieved. This solves the problems of insufficient calibration accuracy and frequent maintenance required by existing technologies, achieving high-precision continuous calibration of ±0.01°, and significantly improving the working performance and reliability of the multimodal radar-visual integrated machine in harsh environments.

[0020] Figure 1 The implementation flowchart of the automatic calibration method for dynamic verification of multimodal radar-visual integrated machines provided in this application embodiment is described in detail below:

[0021] In step 101, feature points of a target reference point at a fixed location are acquired in real time. The feature points are the feature information, signal strength information, and position information of the target device relative to the target reference point obtained by the target device. The target device can be any one of the high-definition camera, millimeter-wave radar, or thermal imaging camera in the multimodal radar-visual integrated machine.

[0022] Among them, the multi-modal radar and vision all-in-one machine includes a high-definition camera, a millimeter wave radar and a thermal imaging camera. Among them, the high-definition camera can clearly capture the detailed information of the surrounding environment due to its high-resolution imaging capability; the millimeter wave radar uses millimeter wave frequency electromagnetic waves to realize accurate measurement of target distance, speed and angle parameters, and has strong penetration and anti-interference ability in complex environment; the thermal imaging camera generates images by sensing the infrared radiation emitted by objects, and can detect and identify targets in low light or even no light conditions.

[0023] When any one camera or radar in the multi-modal radar and vision all-in-one machine deforms or displaces, the highway monitoring video data obtained by the multi-modal radar and vision all-in-one machine will be abnormal. Therefore, in the embodiments of the present application, the information of a target reference point with a fixed position photographed by any device in the multi-modal radar and vision all-in-one machine is obtained in real time, and the feature points of the target reference point are extracted from the information, and the purpose is to provide basic judgment data for subsequent calibration.

[0024] In addition, the multi-modal radar and vision all-in-one machine also includes a blind-filling radar, which aims to supplement the data acquisition of the blind area position of the millimeter wave radar, and is also a millimeter wave radar. Generally, the blind-filling radar will not deform, but if it deforms, the calibration method is the same as that of the millimeter wave radar.

[0025] The multi-modal radar and vision all-in-one machine in the embodiments of the present application integrates high-definition cameras, millimeter wave radars, thermal imaging cameras and other multiple different types of sensors. The data fusion of these sensors is crucial for accurate target detection, recognition and tracking. If the calibration of each sensor is inaccurate or inconsistent, it will cause deviation in data fusion and affect the system's perception and judgment ability of the target. This method accurately calibrates each target device to ensure that they provide accurate data in the same coordinate system, thereby optimizing the effect of multi-sensor data fusion and improving the target detection and recognition accuracy of the entire multi-modal radar and vision all-in-one machine system, providing more reliable information support for subsequent decision-making and control.

[0026] In a possible implementation, the feature points of the target reference point at the fixed position are obtained in real time, which can include:

[0027] When the target device is a high-definition camera, the scale-invariant feature transform and ORB algorithm are used to extract the pixel information and position information of the target reference point photographed by the high-definition camera, and the pixel information and position information of the target reference point photographed by the high-definition camera are used as the feature points of the target reference point obtained by the high-definition camera in real time;

[0028] When the target device is a millimeter wave radar, the echo amplitude and noise point position of the target reference point in the detection area of the millimeter wave radar are extracted by using Doppler velocity constraint, and the echo amplitude and noise point position are taken as the feature points of the target reference point obtained by the millimeter wave radar in real time.

[0029] When the target device is a thermal imaging camera, pixel information, position information and thermal radiation gradient in an image of the target reference point shot by the thermal imaging camera are extracted by fusing the thermal radiation gradient and the space-time feature, and the pixel information, position information and thermal radiation gradient in the image of the target reference point shot by the thermal imaging camera are taken as the feature points of the target reference point obtained by the thermal imaging camera in real time.

[0030] Optionally, when the target device is a high-definition camera, the feature point extraction method is a scale-invariant feature transform (SIFT) and an ORB algorithm.

[0031] The scale-invariant feature transform (SIFT) algorithm has strong invariance to rotation, scaling, brightness change and the like of an image, and can stably detect and describe feature points in the image under different viewing angles and illumination conditions. It first detects extreme points in the image by constructing a Gaussian pyramid, and the extreme points are potential feature point positions. Then, each extreme point is precisely positioned, and a main direction thereof is calculated to ensure that the feature point has rotation invariance. Finally, a feature descriptor is generated, which contains gradient information of a region around the feature point and is used for subsequent feature matching.

[0032] The ORB algorithm is a fast feature point extraction and description algorithm, which combines the advantages of FAST key point detection and BRIEF descriptor. The FAST key point detection can quickly find points with obvious feature changes in the image, and the BRIEF descriptor generates a binary descriptor by comparing the gray values of pixel pairs around the feature point, and has the characteristics of fast calculation and small storage space.

[0033] In the embodiments of the application, the SIFT and ORB algorithms are used simultaneously to extract feature points of the target reference point image shot by the high-definition camera, and corresponding pixel information and position information are obtained, that is, corresponding feature points are extracted. The stability of the SIFT algorithm and the rapidity of the ORB algorithm are used to comprehensively improve the accuracy and efficiency of feature point extraction.

[0034] When the target device is a millimeter wave radar, the feature point extraction method is Doppler velocity constraint. The echo amplitude and noise point position of the target reference point in the detection area of the millimeter wave radar are extracted by using Doppler velocity constraint, as the feature points of the corresponding target reference point of the millimeter wave radar.

[0035] When the target device is a thermal imaging camera, the feature point extraction method is the fusion of thermal radiation gradient and spatio-temporal features. The thermal infrared image generated by the thermal imaging camera reflects the temperature distribution of the target reference point, and the thermal radiation gradient can describe the intensity of temperature change in the image. By calculating the thermal radiation gradient of each pixel point in the thermal infrared image, areas with large temperature changes can be found, which often contain important target feature information. At the same time, considering that the thermal radiation characteristics of the target reference point may change at different times, spatio-temporal feature fusion is also needed. The spatio-temporal feature fusion method can combine the information of multiple thermal infrared images, and by analyzing the position, shape and temperature change of the target at different times, the feature points with stability and representativeness are extracted.

[0036] For example, the image of the target reference point taken by the thermal imaging camera is obtained, and then the thermal radiation gradient and the spatio-temporal feature fusion are used to extract the pixel information, position information and thermal radiation gradient of the target reference point of the image as the feature points.

[0037] In a possible implementation, after obtaining the feature points of the target reference point at a fixed position in real time, the method can further include:

[0038] Mapping the feature points to a spatio-temporal coordinate system of the multi-modal sensor based on Lie group SE(3);

[0039] Correspondingly, judging whether the target device needs to be calibrated according to the feature points can include:

[0040] Judging whether the target device needs to be calibrated according to the feature points mapped to the spatio-temporal coordinate system of the multi-modal sensor based on Lie group SE(3).

[0041] Optionally, the multi-modal sensor system usually integrates multiple different types of sensors, such as high-definition cameras, millimeter wave radars, thermal imaging cameras, etc., and each sensor has its own coordinate system and measurement characteristics. In order to be able to fuse and process these data from different sensors in a unified framework, it is crucial to construct a unified spatio-temporal coordinate system. Lie group SE(3) as a powerful mathematical tool can well describe the rigid body motion in three-dimensional space, including rotation and translation, providing an ideal mathematical basis for constructing the spatio-temporal coordinate system of the multi-modal sensor.

[0042] Specifically, constructing the spatio-temporal coordinate system of the multi-modal sensor based on Lie group SE(3) needs to consider multiple factors. First, a global reference coordinate system needs to be determined, which is usually a fixed and easy-to-describe and understand coordinate system, such as a geographic coordinate system. The geographic coordinate system takes the Earth as a reference and can provide absolute position and direction information. According to the actual application scenario and requirements, a suitable global reference coordinate system is selected.

[0043] Next, the initial pose of each sensor relative to the global reference coordinate system needs to be determined. This can be obtained through the installation parameters of the sensor, calibration data, and external measurement equipment such as total stations, laser trackers, etc.

[0044] After determining the initial pose of each sensor, the Lie group SE(3) is used to describe the motion of the sensor in space-time. The Lie group SE(3) consists of a rotation matrix and a translation vector, which can completely describe the motion of a rigid body in three-dimensional space. By establishing a sensor motion model, combined with the measurement data of the sensor (such as the data of inertial sensors such as accelerometers and gyroscopes), the pose of the sensor relative to the global reference coordinate system can be updated in real time.

[0045] At the same time, time synchronization needs to be considered. The sampling frequency and data transmission delay of different sensors may be different, in order to ensure that the data of different sensors can be accurately aligned in the space-time coordinate system, time synchronization processing is needed. Common time synchronization methods include hardware synchronization and software synchronization. Hardware synchronization triggers the sampling of sensors using a unified clock signal to ensure that the data of different sensors is collected at the same time; software synchronization adds timestamps to the data of the sensor, and aligns and interpolates the data according to the timestamps in the data processing stage.

[0046] After constructing the space-time coordinate system of the multi-modal sensor based on the Lie group SE(3), the feature points of the target reference points obtained by the real-time target device are mapped into this coordinate system. The purpose is to unify the feature points obtained by different sensors into a common coordinate system, so as to perform subsequent data fusion, target recognition and tracking operations.

[0047] Because the imaging principles and measurement methods of different sensors are different, the representation of the feature points obtained by them in their respective sensor coordinate systems is also different. For example, the feature points obtained by a high-definition camera are usually represented in the form of image pixel coordinates, the feature points obtained by a millimeter wave radar are represented in the form of distance, angle and speed, and the feature points obtained by a thermal imaging camera are represented in the form of temperature distribution and position information. Therefore, it is necessary to convert these different forms of feature points into the space-time coordinate system of the multi-modal sensor based on the Lie group SE(3).

[0048] For the feature points obtained by a high-definition camera, first, the image pixel coordinates need to be converted to three-dimensional coordinates in the camera coordinate system through the camera's intrinsic matrix. The camera's intrinsic matrix contains parameters such as the camera's focal length and principal point coordinates, which can be obtained through camera calibration experiments. Then, according to the pose of the camera relative to the global reference coordinate system (described by the Lie group SE(3)), the three-dimensional coordinates in the camera coordinate system are converted to the global reference coordinate system.

[0049] For the feature points acquired by the millimeter wave radar, the distance, angle, and velocity parameters can be directly converted into target position and velocity information in the radar coordinate system. Then, according to the pose of the radar relative to the global reference coordinate system, the target position and velocity information in the radar coordinate system is converted into the global reference coordinate system.

[0050] For the feature points acquired by the thermal imaging camera, it is usually necessary to combine the imaging model and calibration data of the thermal imaging camera to convert the temperature distribution and position information into three-dimensional coordinates in the thermal imaging camera coordinate system. Then, according to the pose of the thermal imaging camera relative to the global reference coordinate system, the three-dimensional coordinates in the thermal imaging camera coordinate system are converted into the global reference coordinate system.

[0051] After mapping the feature points of different sensors to the global reference coordinate system, it is also necessary to fuse and remove the feature points. Due to the different measurement accuracy and coverage of different sensors, multiple sensors may detect the same target reference point, resulting in repeated feature points in the global reference coordinate system. By setting appropriate fusion algorithm and threshold, the feature points from different sensors about the same target reference point can be fused, and the repeated feature points can be removed, improving the accuracy and reliability of the feature points. For example, a weighted average method can be used to fuse the position information of the repeated feature points, and the weight coefficients are determined according to the measurement accuracy and reliability of different sensors, so that the position of the fused feature points is more accurate.

[0052] In step 102, it is determined whether the target device needs to be calibrated according to the feature points, and when the target device needs to be calibrated, the first data, the second data and the third data are acquired, the first data is displacement data collected by a displacement sensor, the second data is attitude data collected by an attitude sensor, and the third data is temperature data collected by a temperature sensor.

[0053] In the embodiment of the present application, the feature points extracted in step 101 are compared with the standard feature points of the corresponding target device to determine whether the parameters of the target device need to be calibrated at the current time. When it is determined that calibration is needed, the first data (i.e. displacement data) of the target device is collected by the displacement sensor of the target device, the second data (i.e. attitude data) of the target device is collected by the attitude sensor of the target device, and the third data (i.e. temperature data) of the target device is collected by the temperature sensor of the target device.

[0054] The process of collecting the first data, the second data and the third data is as follows:

[0055] 1) Collecting the first data: Displacement sensors can accurately measure the position changes of target devices in space, and the displacement data collected is crucial for evaluating the mechanical motion accuracy of target devices. When obtaining displacement data, choose the appropriate type and accuracy of displacement sensors, such as laser displacement sensors, inductive displacement sensors, etc., according to the motion characteristics and measurement range of the target device, and install them reasonably. Ensure that the measurement axis of the displacement sensor is consistent with the motion direction of the target device to avoid measurement errors. During the operation of the target device, real-time collection of displacement data at a certain sampling frequency is carried out, and the collected data is stored in a special data storage system. At the same time, pre-process the displacement data, including filtering, denoising, etc., to improve the quality and accuracy of the data.

[0056] 2) Collecting the second data: Attitude sensors are used to measure the attitude information of the target device, such as pitch angle, yaw angle, and roll angle, etc. These attitude data can reflect the rotation state of the target device in three-dimensional space, and are of great significance for evaluating the spatial positioning accuracy and motion stability of the target device. Select high-precision inertial measurement unit (IMU) as the attitude sensor, which usually integrates accelerometer, gyroscope and magnetometer, etc. multiple sensors can provide comprehensive attitude information. When installing the attitude sensor, ensure that its coordinate system is aligned with the body coordinate system of the target device to ensure the accuracy of the measurement results. During the operation of the target device, real-time collection of attitude data is carried out, and data fusion processing is carried out. Due to the errors and noises in the measurement data of different sensors, Kalman filtering and other data fusion algorithms are used to fuse the data of accelerometers, gyroscopes and magnetometers to obtain more accurate and reliable attitude data.

[0057] 3) Collecting the third data: Temperature is one of the important factors affecting the performance and measurement accuracy of the target device. During the operation of the target device, heat will be generated due to mechanical friction, current thermal effect, etc., causing the temperature of the device to rise. Changes in temperature may cause thermal expansion and thermal deformation of the parts of the target device, affecting its measurement accuracy and performance. Therefore, it is necessary to collect temperature data of the target device to evaluate the impact of temperature on the device. Install temperature sensors such as thermocouples, thermal resistors, etc. at key parts of the target device, such as motors, sensors, mechanical connections, etc. According to the running characteristics and environmental conditions of the target device, set the sampling frequency of the temperature sensor reasonably, and collect temperature data in real time. At the same time, monitor and analyze the temperature data, establish a correlation model between temperature and the performance of the target device, so as to consider the influence of temperature factors in the calibration process.

[0058] The embodiment of the present application can more accurately reflect the current working state and measurement accuracy of the device by acquiring the feature points of the target reference point in real time and judging whether the target device needs to be calibrated based on the feature points. Compared with the traditional periodic calibration or calibration method based on a fixed time interval, the embodiment of the present application can trigger the calibration process in time according to the performance change of the device in the actual operation process, so as to ensure that the calibration is performed when the device is really needed, thereby improving the pertinence and accuracy of the calibration.

[0059] In a possible implementation, judging whether the target device needs to be calibrated according to the feature points can include:

[0060] judging whether the position information of the feature points extracted by the target device and mapped to the spatiotemporal coordinate system of the multi-modal sensor based on Lie group SE(3) is the same as preset position information, the preset position information being position information of the target reference point acquired in the spatiotemporal coordinate system of the multi-modal sensor based on Lie group SE(3);

[0061] if the position information of the feature points extracted by the target device and mapped to the spatiotemporal coordinate system of the multi-modal sensor based on Lie group SE(3) is the same as the preset position information, it is indicated that the target device does not need to be calibrated;

[0062] if the position information of the feature points extracted by the target device and mapped to the spatiotemporal coordinate system of the multi-modal sensor based on Lie group SE(3) is not the same as the preset position information, it is indicated that the target device needs to be calibrated.

[0063] Optionally, when the target device is in an initial ideal state, that is, the device has been accurately calibrated and debugged, and the performance and position of each component are in the best state, the position information of the feature points of the target reference point acquired by the target device and mapped to the spatiotemporal coordinate system of the multi-modal sensor based on Lie group SE(3) is consistent with the position information of the target reference point acquired in the spatiotemporal coordinate system of the multi-modal sensor based on Lie group SE(3) in the embodiment of the present application.

[0064] Therefore, after determining the preset position information of the target reference point in the spatiotemporal coordinate system of the multi-modal sensor based on Lie group SE(3), the embodiment of the present application judges whether the position information of the feature points extracted by the target device and mapped to the spatiotemporal coordinate system of the multi-modal sensor based on Lie group SE(3) is the same as the preset position information, if the same, it is indicated that the target device does not need to be calibrated; if not the same, it is indicated that the target device needs to be calibrated.

[0065] In step 103, the calibration type of the target device is judged according to the first data, the second data and the third data, and the calibration of the target device is implemented according to the calibration type.

[0066] In the embodiments of the present application, after the first data, the second data and the third data are preprocessed, the preprocessed first data, the second data and the third data are compared with the corresponding standard data to determine the calibration type required by the target device at present. Then, according to the corresponding calibration type, the corresponding compensation model is selected for compensation calibration.

[0067] Among them, the preprocessing includes data cleaning, removing noise and outliers in the data to improve the quality and reliability of the data; data alignment, ensuring that the three kinds of data are consistent in time and space dimensions, facilitating subsequent comparison and analysis; data normalization, converting data of different dimensions to the same scale range to avoid the influence of dimension difference on the analysis result.

[0068] After determining that the target device needs calibration, the displacement data collected by the displacement sensor, the attitude data collected by the attitude sensor and the temperature data collected by the temperature sensor are obtained, and the calibration type is determined by comprehensively considering these data. Among them, the displacement data can reflect the change of the device in the space position, the attitude data can reflect the change of the angle and direction of the device, and the temperature data considers the influence of the environmental temperature on the performance of the device. By comprehensively considering these multi-dimensional factors, the reasons for the change of the device performance can be more comprehensively and accurately analyzed, and the most suitable calibration type is determined, so that the calibration process is more scientific and reasonable, and the measurement accuracy of the calibrated device is effectively improved.

[0069] In a possible implementation manner, the calibration type includes temperature deformation automatic calibration and mechanical deformation automatic calibration; the automatic calibration type required by the target device is determined according to the first data, the second data and the third data, including:

[0070] It is determined whether the first data exceeds a preset displacement threshold value, and whether the second data exceeds a preset attitude threshold value, and whether the third data exceeds a preset temperature threshold value;

[0071] If the first data exceeds the preset displacement threshold value, and / or the second data exceeds the preset attitude threshold value, and the third data does not exceed the preset temperature threshold value, it is determined that the calibration type required by the target device is mechanical deformation automatic calibration;

[0072] If the first data does not exceed the preset displacement threshold value, and the second data does not exceed the preset attitude threshold value, and the third data exceeds the preset temperature threshold value, it is determined that the calibration type required by the target device is temperature deformation automatic calibration;

[0073] If the first data exceeds the preset displacement threshold value, and / or the second data exceeds the preset attitude threshold value, and the third data exceeds the preset temperature threshold value, it is determined that the automatic calibration type required by the target device is mechanical deformation automatic calibration and temperature deformation automatic calibration.

[0074] Optionally, in the calibration process of the target device, the explicit calibration type is the key prerequisite to ensure the accurate and effective development of the calibration work. The calibration types in the embodiments of the present application mainly cover temperature deformation automatic calibration and mechanical deformation automatic calibration. Among them, the temperature deformation automatic calibration mainly corrects the deformation of the target device under the influence of temperature change, aiming to eliminate the interference of temperature factors on the measurement accuracy and performance stability of the device; the mechanical deformation automatic calibration focuses on the deformation of the device caused by mechanical factors such as mechanical structure stress, wear, assembly error, etc., and improves the accuracy and reliability of the device through correction of mechanical deformation.

[0075] Among them, the preset displacement threshold value represents the maximum displacement deviation allowed by the target device in the normal working state, and exceeding the value means that the device may have abnormal movement or deformation of the mechanical structure.

[0076] The preset posture threshold value specifies the maximum allowed deviation of the device maintaining the correct posture in space, and once it is exceeded, it may imply that the mechanical support structure, transmission components, etc. of the device have problems, affecting the posture stability of the device.

[0077] The preset temperature threshold value is determined according to the material properties used by the device, the working temperature range of electronic components, and the sensitivity of the device as a whole to temperature changes. When the temperature exceeds the threshold value, the device may produce an inconsiderable deformation due to thermal expansion and contraction, etc.

[0078] The specific judgment process is:

[0079] Judge whether the first data exceeds the preset displacement threshold value, and judge whether the second data exceeds the preset posture threshold value, and judge whether the third data exceeds the preset temperature threshold value.

[0080] 1) Mechanical deformation automatic calibration judgment

[0081] If the first data exceeds the preset displacement threshold value, it indicates that the target device has a displacement deviation exceeding the normal range in the actual running process, which may be caused by reasons such as loosening, wear, deformation or uneven stress of mechanical parts. And / or, the second data also exceeds the preset posture threshold value, which means that the attitude angle of the device has changed abnormally, further indicating that the mechanical structure of the device has problems, affecting its posture stability. And the third data does not exceed the preset temperature threshold value, i.e. the temperature environment of the device is within the normal range, and temperature change is not the main factor causing the displacement and posture abnormality of the device. In combination with the above situations, it can be determined that the calibration type required by the target device is mechanical deformation automatic calibration. At this time, it is necessary to focus on checking and adjusting the mechanical structure of the device, such as tightening loose bolts, replacing worn parts, correcting deformed parts, etc., to eliminate the influence of mechanical deformation on the performance of the device.

[0082] 2) Temperature deformation automatic calibration judgment

[0083] If the first data does not exceed the preset displacement threshold, it indicates that the displacement of the target device is within a normal range, and the mechanical structure does not have obvious abnormal movement. In addition, the second data also does not exceed the preset posture threshold, indicating that the posture angle of the device remains stable, and the mechanical support and transmission system work normally. Then, the third data exceeds the preset temperature threshold, which means that the temperature environment of the device exceeds its normal working range, and the thermal expansion and contraction of the key components of the device may have occurred due to temperature changes, thereby causing deformation and affecting the accuracy and performance of the device. In this case, the calibration type required by the target device is determined to be temperature deformation automatic calibration. For temperature deformation automatic calibration, the deformation law of the device at different temperatures needs to be analyzed, a mathematical model between temperature and deformation is established, and the measurement data of the device is corrected through a corresponding compensation algorithm to eliminate the error caused by temperature deformation.

[0084] 3) Mechanical deformation and temperature deformation joint calibration judgment

[0085] If the first data exceeds the preset displacement threshold, and / or, the second data exceeds the preset posture threshold, and the third data also exceeds the preset temperature threshold, it indicates that the target device has both mechanical structure deformation problems and is significantly affected by temperature changes. Mechanical deformation may cause deviations in the displacement and posture of the device, and temperature deformation will further exacerbate this deviation or produce new deformation patterns under different temperature conditions. Therefore, in this case, the calibration type required by the target device is determined to be mechanical deformation automatic calibration and temperature deformation automatic calibration. A comprehensive inspection and analysis of the mechanical structure and temperature characteristics of the device is required, the mechanical deformation of the device is corrected through the mechanical deformation automatic calibration method first, and then the comprehensive influence of temperature factors on the deformation of the device is considered by combining the temperature deformation automatic calibration technology to establish a more complex compensation model for accurate correction of the measurement data of the device, so as to ensure that the device can meet the expected accuracy and performance requirements under various working conditions.

[0086] In one possible implementation, when the calibration type is temperature deformation automatic calibration, according to the calibration type, the calibration of the target device can include:

[0087] A long short-term memory artificial neural network combined with an attention mechanism is used to construct a temperature drift compensation model, and the temperature drift compensation model takes the temperature data collected by the temperature sensor as input and outputs the axial offset.

[0088] The third data is input into the temperature drift compensation model to output the corresponding axial offset, so as to calibrate the target device.

[0089] Optionally, in the temperature deformation automatic calibration, considering that the temperature change of the target device during actual operation will have a complex impact on the structure of the device, thereby causing deformation problems such as axial offset.

[0090] As a special recurrent neural network, Long Short-Term Memory (LSTM) has strong sequence data processing capability and can effectively capture long-term dependencies in data. During device operation, temperature data is a sequence that changes over time, and LSTM can learn the pattern of temperature change over time and the potential impact of such changes on device axial offset. The attention mechanism can enable the model to automatically focus on key information that has a greater impact on the output when processing input data, improving the model's ability to extract important features. Combining LSTM and attention mechanism to build a temperature drift compensation model can more accurately simulate the complex nonlinear relationship between temperature change and device axial offset, thereby providing reliable compensation basis for temperature deformation automatic calibration.

[0091] The construction process of the temperature drift compensation model is as follows:

[0092] Collect a large amount of running data of the target device under different temperature conditions, including temperature data collected by temperature sensors and corresponding axial offset obtained by high-precision measuring instruments. These data should cover various temperature ranges and working conditions that the device may encounter to ensure the generalization ability of the model.

[0093] Design the structure of the LSTM network, determine the number of hidden layers, the number of neurons in each hidden layer, and the specific implementation of the attention mechanism. Generally, the selection of the number of hidden layers and the number of neurons needs to be repeated and optimized according to the complexity of the data and the performance of the model. The attention mechanism can use common self-attention mechanism or soft attention mechanism, which assigns different weights to temperature data at different time steps according to the feature distribution of input temperature data, focusing on key temperature change areas.

[0094] Divide the prepared data set into training set, validation set and test set. Use the training set to train the constructed temperature drift compensation model, and continuously adjust the parameters of the model through the backpropagation algorithm to minimize the error between the output of the model (i.e. the predicted axial offset) and the true axial offset. During training, use the validation set to monitor the performance of the model to prevent overfitting. When the performance of the model on the validation set no longer improves significantly, stop training. Finally, use the test set to evaluate the trained model and verify the accuracy and generalization ability of the model.

[0095] After the temperature drift compensation model is trained, the third data, i.e., the temperature data of the target device in the current working state collected by the temperature sensor in real time, is input into the temperature drift compensation model, and the corresponding axial displacement prediction value is output through a series of calculations and processing by using the complex relationship between the temperature and the axial displacement learned by the temperature drift compensation model. The prediction value reflects the axial displacement of the target device caused by the deformation due to temperature change under the current temperature condition. According to the output axial displacement, the measurement result or the control parameter of the target device is adjusted correspondingly.

[0096] For example, when the thermal imaging millimeter wave radar in the multimodal radar and vision integrated machine, the axial displacement output by the temperature drift compensation model is deducted from the measurement result in the measurement process, so as to eliminate the influence of the deformation caused by temperature on the measurement accuracy.

[0097] The embodiments of the present application compensate parameters by the temperature drift compensation model, realize automatic calibration of the temperature deformation of the target device, and improve the performance and accuracy of the device.

[0098] In a possible implementation, when the calibration type is mechanical deformation automatic calibration, according to the calibration type, the calibration of the target device can include:

[0099] The angular velocity and linear acceleration are acquired in real time by the six-axis inertial measurement unit;

[0100] The vibration compensation model is constructed by using the multimodal frequency domain analysis and adaptive filtering algorithm, the vibration compensation model takes the angular velocity and linear acceleration as input and takes the displacement offset as output;

[0101] The angular velocity and linear acceleration are input into the vibration compensation model, and the corresponding displacement offset is output to calibrate the target device.

[0102] Optionally, when it is determined that the mechanical deformation automatic calibration is performed, in the process of device operation, the vibration and other dynamic factors have a complex influence on the measurement accuracy and operation stability of the device. In order to realize accurate calibration, the vibration compensation is considered comprehensively. Specifically:

[0103] Firstly, data acquisition is performed.

[0104] By installing a six-axis inertial measurement unit (IMU) on the target device, which integrates three mutually perpendicular gyroscopes and three mutually perpendicular accelerometers, the angular velocity and linear acceleration information of the target device in three-dimensional space can be obtained in real time and accurately. Among them, the coordinate system of the IMU needs to be strictly aligned with the body coordinate system of the target device, so as to avoid affecting the accuracy of the data due to coordinate conversion error. At the same time, according to the working environment and motion characteristics of the target device, the appropriate sampling frequency is selected to ensure that the dynamic changes of angular velocity and linear acceleration can be captured completely.

[0105] Then, a vibration compensation model is constructed.

[0106] The collected angular velocity and linear acceleration are subjected to multi-modal frequency domain analysis. First, the fast Fourier transform is used to convert the time domain signal into the frequency domain signal, and the frequency spectrum distribution of the signal is obtained. By analyzing the frequency spectrum, the vibration modes corresponding to different frequency components can be identified, for example, low-frequency vibration may be caused by the mechanical structure resonance of the device, and high-frequency vibration may be caused by external environmental interference or small vibration of internal parts of the device. At the same time, combined with multi-modal analysis methods such as wavelet transform, the signal is subjected to multi-scale decomposition, further extracting the local features of different frequency bands and time domains, and more comprehensively describing the characteristics of the vibration. Multi-modal frequency domain analysis can provide rich feature information for subsequent adaptive filtering algorithms, which helps to more accurately separate and compensate the mechanical displacement offset caused by vibration.

[0107] Based on the results of multi-modal frequency domain analysis, an adaptive filtering algorithm is designed to construct a vibration compensation model. The adaptive filtering algorithm can automatically adjust the parameters of the filter according to the statistical characteristics of the input signal to effectively suppress different vibration modes. Common adaptive filtering algorithms such as least mean square algorithm, recursive least squares algorithm, etc. can be selected according to the actual application scene and performance requirements. In the algorithm design process, the angular velocity and linear acceleration are taken as the input signal, and the displacement offset is taken as the expected output signal. By continuously iterating and updating the weight of the filter, the error between the output of the filter and the expected output is minimized. At the same time, in order to improve the adaptability and robustness of the model, neural network and other intelligent algorithms can be introduced to optimize the adaptive filtering algorithm, so that the model can better cope with complex and variable vibration environments.

[0108] Then, the model is trained and verified.

[0109] A large amount of actual data is collected, including the angular velocity, linear acceleration and corresponding displacement offset of the target device under different working conditions, which are obtained by high-precision measuring instruments. The data is divided into a training set, a validation set and a test set. The trained vibration compensation model is trained using the training set, and the parameters of the model are continuously adjusted through algorithms such as back propagation, so that the performance indicators (such as mean square error) of the model on the training set are optimized. During the training process, the validation set is used to monitor the performance of the model to prevent overfitting. When the performance of the model on the validation set no longer improves significantly, training is stopped. Finally, the test set is used to independently evaluate the trained model to verify the accuracy and generalization ability of the model. If the performance of the model does not meet the requirements, the model structure or parameters need to be adjusted again, and the training and verification are performed again.

[0110] Finally, after the vibration compensation model is trained, the angular velocity and linear acceleration of the target device are input into the vibration compensation model. The model outputs the corresponding displacement offset prediction value according to the input angular velocity and linear acceleration information and the relationship between vibration and displacement offset learned by the model. According to the output displacement offset, the measurement results or control parameters of the target device are adjusted accordingly.

[0111] For example, for a millimeter wave radar in a radar and vision integrated machine, the displacement offset output by the vibration compensation model is deducted from the measurement results to eliminate the measurement error of the millimeter wave radar caused by vibration.

[0112] The embodiments of the present application use the vibration compensation model for compensation adjustment, realize automatic calibration of the mechanical deformation of the target device, effectively compensate the influence of vibration on the performance of the device, and improve the overall precision and stability of the device.

[0113] In one possible implementation, the feature points are composed of horizontal coordinates and vertical coordinates. Before the angular velocity and linear acceleration are obtained in real time by the six-axis inertial measurement unit, the method can further include:

[0114] The coordinate difference between the feature points and the initial feature points is calculated to obtain a coordinate difference value, which includes a horizontal coordinate difference value and a vertical coordinate difference value.

[0115] According to the coordinate difference value, the deformation displacement type of the target device when the mechanical deformation occurs is determined.

[0116] Optionally, before the automatic calibration of mechanical deformation, the application needs to calculate the coordinate difference between the feature point and the initial feature point. The initial feature point is the feature point position determined when the target device is in the initial state (i.e. no mechanical deformation, in the standard working condition or static state, etc.). By real-time acquisition of the coordinate value of the feature point at the current time, and subtracting the coordinate value of the initial feature point, the coordinate difference can be obtained. Specifically, let the horizontal coordinate of the feature point at the current time be , the vertical coordinate be , the horizontal coordinate of the initial feature point be , and the vertical coordinate be , then the horizontal coordinate difference is , and the vertical coordinate difference is . The coordinate difference obtained here is an important basis for subsequent analysis of the mechanical deformation displacement type.

[0117] Then, according to the calculated coordinate difference, the deformation displacement type of the target device when the mechanical deformation occurs is determined. The mechanical deformation displacement type can usually be divided into several cases, such as simple translation deformation along the horizontal axis direction, simple translation deformation along the vertical axis direction, translation deformation along the horizontal and vertical axes, rotation deformation around a point, and more complex composite deformation (including the combination of translation and rotation), etc.

[0118] In one possible implementation, the deformation displacement type includes horizontal displacement and pitch displacement; according to the coordinate difference, the deformation displacement type of the target device when the mechanical deformation occurs can include:

[0119] determining whether the absolute value of the horizontal coordinate difference is greater than 0 and whether the absolute value of the vertical coordinate difference is greater than 0;

[0120] if the absolute value of the horizontal coordinate difference is greater than 0 and the absolute value of the vertical coordinate difference is equal to 0, it indicates that the deformation displacement type of the target device when the mechanical deformation occurs is horizontal displacement;

[0121] if the absolute value of the horizontal coordinate difference is equal to 0 and the absolute value of the vertical coordinate difference is greater than 0, it indicates that the deformation displacement type of the target device when the mechanical deformation occurs is pitch displacement;

[0122] if the absolute value of the horizontal coordinate difference is greater than 0 and the absolute value of the vertical coordinate difference is greater than 0, it indicates that the deformation displacement type of the target device when the mechanical deformation occurs is horizontal displacement and pitch displacement.

[0123] Optionally, in the mechanical deformation monitoring and analysis of the target device, the explicit deformation displacement type is a key prerequisite for accurately evaluating the device state and formulating effective compensation strategies. In the embodiments of the present application, the deformation displacement type mainly covers two types with typical characteristics, namely horizontal displacement and pitch displacement. Among them, the horizontal displacement mainly embodies the overall translation movement of the target device in a specific direction (i.e. the horizontal axis direction) in the horizontal plane. This displacement will cause the positions of different points on the device to change in the horizontal direction, while the positions perpendicular to the horizontal direction (i.e. the vertical axis direction) remain relatively unchanged. The pitch displacement refers to the rotation of the target device around a certain horizontal axis. During the rotation process, the points at different positions on the device will have different degrees of displacement change in the vertical direction (i.e. the vertical axis direction), while the positions in the horizontal direction (i.e. the horizontal axis direction) may remain unchanged or show regular changes related to the rotation.

[0124] The specific determination process of the deformation displacement type is as follows:

[0125] First, absolute value operations are performed on the horizontal coordinate difference and the vertical coordinate difference respectively. The purpose of the absolute value operation is to eliminate the positive and negative signs of the coordinate difference and only focus on the numerical value, because both positive and negative coordinate differences represent that the feature points have occurred displacement relative to the initial position, and the size of the displacement is the key factor for judging the deformation type. Specifically, let the horizontal coordinate difference be , and the vertical coordinate difference be , then the absolute value of the horizontal coordinate difference is , and the absolute value of the vertical coordinate difference is .

[0126] Next, it is judged whether the absolute value of the horizontal coordinate difference is greater than 0, and whether the absolute value of the vertical coordinate difference is greater than 0. This judgment process is based on the inherent correlation between the coordinate difference and the deformation displacement type. In an ideal case, if the target device only occurs horizontal displacement, since the horizontal displacement only affects the position of the device in the horizontal direction (i.e. the horizontal axis direction), and the position in the vertical direction (i.e. the vertical axis direction) is not affected, at this time the absolute value of the vertical coordinate difference should be equal to 0, and the absolute value of the horizontal coordinate difference is greater than 0. Conversely, if the target device only occurs pitch displacement, since the pitch displacement mainly causes the position of the device to change in the vertical direction (i.e. the vertical axis direction), the position in the horizontal direction (i.e. the horizontal axis direction) may remain unchanged or change very little (which can be ignored in an ideal case), therefore the absolute value of the horizontal coordinate difference is equal to 0, and the absolute value of the vertical coordinate difference is greater than 0.

[0127] That is, if the absolute value of the horizontal coordinate difference greater than 0, and the absolute value of the vertical coordinate difference is greater than 0, which indicates that the target device has moved in both the horizontal and vertical directions when mechanical deformation occurs. This situation may be caused by the target device being subjected to forces in both the horizontal and vertical directions, or by a complex compound motion, resulting in both horizontal displacement and pitch displacement. equal to 0, which indicates that the target device has only moved in the horizontal direction when mechanical deformation occurs, and there is no significant change in position in the vertical direction. Therefore, it can be determined that the deformation displacement type of the target device is horizontal displacement.

[0128] greater than 0, and the absolute value of the vertical coordinate difference is greater than 0, which indicates that the target device has moved in both the horizontal and vertical directions when mechanical deformation occurs. This situation may be caused by the target device being subjected to forces in both the horizontal and vertical directions, or by a complex compound motion, resulting in both horizontal displacement and pitch displacement. equal to 0, which indicates that the target device has only moved in the vertical direction when mechanical deformation occurs, and there is no significant change in position in the horizontal direction. Therefore, it can be determined that the deformation displacement type of the target device is pitch displacement. greater than 0, and the absolute value of the vertical coordinate difference is greater than 0, which indicates that the target device has moved in both the horizontal and vertical directions when mechanical deformation occurs. This situation may be caused by the target device being subjected to forces in both the horizontal and vertical directions, or by a complex compound motion, resulting in both horizontal displacement and pitch displacement.

[0129] greater than 0, and the absolute value of the vertical coordinate difference is greater than 0, which indicates that the target device has moved in both the horizontal and vertical directions when mechanical deformation occurs. This situation may be caused by the target device being subjected to forces in both the horizontal and vertical directions, or by a complex compound motion, resulting in both horizontal displacement and pitch displacement. greater than 0, which indicates that the target device has moved in both the horizontal and vertical directions when mechanical deformation occurs. This situation may be caused by the target device being subjected to forces in both the horizontal and vertical directions, or by a complex compound motion, resulting in both horizontal displacement and pitch displacement.

[0130] ​The application provides a multi-modal radar and vision integrated machine dynamic verification automatic calibration method, feature points of a target reference point at a fixed position are acquired in real time, the feature points are feature information, signal strength information and position information of the target reference point acquired by a target device, and the target device is any device of a high-definition camera, a millimeter wave radar and a thermal imaging camera in the multi-modal radar and vision integrated machine; whether the target device needs to be calibrated is determined according to the feature points, and when the target device needs to be calibrated, first data, second data and third data are acquired, the first data is displacement data collected by a displacement sensor, the second data is attitude data collected by an attitude sensor, and the third data is temperature data collected by a temperature sensor; the calibration type of the target device is determined according to the first data, the second data and the third data, and calibration of the target device is realized according to the calibration type. The application acquires the feature points of the target reference point in real time, and determines whether the target device needs to be calibrated based on the feature points. This determination method based on the position information of the target reference point in an actual captured image can directly reflect the current working state and imaging or detection accuracy of the device, compared with the traditional periodic calibration or calibration based on experience, can more accurately capture the performance change of the device, and timely find the situation that needs to be calibrated, thereby improving the pertinence and accuracy of calibration. Furthermore, after it is determined that the target device needs to be calibrated, the displacement data collected by the displacement sensor, the attitude data collected by the attitude sensor and the temperature data collected by the temperature sensor are acquired, and the calibration type is determined by comprehensively considering these data. Displacement, attitude and temperature and other factors will affect the performance of each device (the high-definition camera, the millimeter wave radar and the thermal imaging camera) in the radar and vision integrated machine. By considering these multi-dimensional data, the reasons and degrees of the performance change of the device can be more accurately understood, and then the most suitable calibration type is selected, and the accuracy of calibration is further improved. Meanwhile, the application can perform calibration determination and adjustment in real time during the operation of the device, without frequent disassembly and transportation of the device, thereby greatly reducing the maintenance cost and time, and improving the availability and working efficiency of the device.

[0131] It should be understood that the size of the serial number of each step in the above embodiment does not mean the order of execution, and the execution order of each process should be determined according to its function and inherent logic, and should not constitute any limitation on the implementation process of the embodiments of the application.

[0132] The following is a device embodiment of the application, and for details not described in detail, reference can be made to the corresponding method embodiments described above.

[0133] Figure 2 A structure schematic diagram of a multi-modal radar and vision integrated machine dynamic verification automatic calibration device provided by an embodiment of the application is shown, only parts related to the embodiment of the application are shown for the convenience of description, and the details are as follows:

[0134] As Figure 2As shown, the multi-modal radar and visual integrated machine dynamic verification automatic calibration device 2 comprises:

[0135] The feature point acquisition module 21 is configured to acquire feature points of a target reference point at a fixed position in real time, the feature points being feature information, signal strength information and position information of the target reference point acquired by a target device, and the target device being any one of a high-definition camera, a millimeter wave radar and a thermal imaging camera in the multi-modal radar and visual integrated machine;

[0136] The calibration judgment and data acquisition module 22 is configured to judge whether the target device needs to be calibrated according to the feature points, and acquire first data, second data and third data when the target device needs to be calibrated, the first data being displacement data collected by a displacement sensor, the second data being attitude data collected by an attitude sensor, and the third data being temperature data collected by a temperature sensor.

[0137] The calibration module 23 is configured to judge a calibration type of the target device according to the first data, the second data and the third data, and implement calibration of the target device according to the calibration type.

[0138] The application provides a multi-modal radar and vision integrated machine dynamic verification automatic calibration device. The feature points of the target reference points at a fixed position are acquired in real time. The feature points are feature information, signal strength information and position information of the target reference points acquired by a target device. The target device is any one of a high-definition camera, a millimeter wave radar and a thermal imaging camera in the multi-modal radar and vision integrated machine. Whether the target device needs to be calibrated is determined according to the feature points. When the target device needs to be calibrated, first data, second data and third data are acquired. The first data is displacement data collected by a displacement sensor. The second data is attitude data collected by an attitude sensor. The third data is temperature data collected by a temperature sensor. The calibration type of the target device is determined according to the first data, the second data and the third data. The calibration of the target device is realized according to the calibration type. The feature points of the target reference points are acquired in real time, and whether the target device needs to be calibrated is determined according to the feature points. The determination method based on the position information of the target reference points in the actual captured image can directly reflect the current working state and imaging or detection accuracy of the device. Compared with the traditional periodic calibration or calibration method based on experience, the device performance change can be more accurately captured, and the situation needing calibration can be found in time, so that the pertinence and accuracy of calibration are improved. After it is determined that the target device needs to be calibrated, the displacement data collected by the displacement sensor, the attitude data collected by the attitude sensor and the temperature data collected by the temperature sensor are acquired, and the calibration type is determined by comprehensively considering the data. The displacement, attitude and temperature and other factors will affect the performance of each device (the high-definition camera, the millimeter wave radar and the thermal imaging camera) in the radar and vision integrated machine. By considering the multi-dimensional data, the reasons and degrees of the device performance change can be more accurately understood, and the most suitable calibration type is selected, so that the accuracy of calibration is further improved. Meanwhile, the calibration determination and adjustment can be performed in real time during the operation of the device, the device does not need to be frequently disassembled and transported, the maintenance cost and time are greatly reduced, and the usability and working efficiency of the device are improved.

[0139] In a possible implementation, the feature point acquisition module can be configured to:

[0140] When the target device is a high-definition camera, the pixel information and position information of the target reference points captured by the high-definition camera are extracted by using the scale-invariant feature transform and the ORB algorithm, and the pixel information and position information of the target reference points captured by the high-definition camera are taken as the feature points of the target reference points acquired by the high-definition camera in real time.

[0141] When the target device is a millimeter wave radar, the echo amplitude and noise point position of the target reference points in the detection region of the millimeter wave radar are extracted by using the Doppler velocity constraint, and the echo amplitude and noise point position are taken as the feature points of the target reference points acquired by the millimeter wave radar in real time.

[0142] When the target device is a thermal imaging camera, the thermal radiation gradient is fused with the spatio-temporal features, the pixel information, the position information and the thermal radiation gradient of the target reference point in the image shot by the thermal imaging camera are extracted, and the pixel information, the position information and the thermal radiation gradient of the target reference point in the image shot by the thermal imaging camera are taken as the feature points of the target reference point acquired by the thermal imaging camera in real time.

[0143] In a possible implementation, the apparatus can further include a spatio-temporal alignment module, which can be configured to:

[0144] map the feature points to the spatio-temporal coordinate system of the multi-modal sensor based on the Lie group SE(3) that has been constructed;

[0145] Correspondingly, the calibration judgment and data acquisition module can be configured to:

[0146] determine whether the target device needs to be calibrated according to the feature points mapped to the spatio-temporal coordinate system of the multi-modal sensor based on the Lie group SE(3).

[0147] In a possible implementation, the calibration judgment and data acquisition module can be specifically configured to:

[0148] determine whether the position information of the feature points extracted by the target device and mapped to the spatio-temporal coordinate system of the multi-modal sensor based on the Lie group SE(3) is the same as preset position information, the preset position information being the position information of the target reference point acquired in the spatio-temporal coordinate system of the multi-modal sensor based on the Lie group SE(3).

[0149] If the position information of the feature points extracted by the target device and mapped to the spatio-temporal coordinate system of the multi-modal sensor based on the Lie group SE(3) is the same as the preset position information, it is indicated that the target device does not need to be calibrated.

[0150] If the position information of the feature points extracted by the target device and mapped to the spatio-temporal coordinate system of the multi-modal sensor based on the Lie group SE(3) is not the same as the preset position information, it is indicated that the target device needs to be calibrated.

[0151] In a possible implementation, the calibration types include temperature deformation automatic calibration and mechanical deformation automatic calibration; and the calibration module can be configured to:

[0152] determine whether the first data exceeds a preset displacement threshold, whether the second data exceeds a preset attitude threshold, and whether the third data exceeds a preset temperature threshold.

[0153] If the first data exceeds the preset displacement threshold, and / or the second data exceeds the preset attitude threshold, and the third data does not exceed the preset temperature threshold, it is determined that the calibration type required by the target device is mechanical deformation automatic calibration.

[0154] If the first data does not exceed the preset displacement threshold, the second data does not exceed the preset posture threshold, and the third data exceeds the preset temperature threshold, it is determined that the calibration type required by the target device is temperature deformation automatic calibration.

[0155] If the first data exceeds the preset displacement threshold, and / or, the second data exceeds the preset posture threshold, and the third data exceeds the preset temperature threshold, it is determined that the automatic calibration type required by the target device is mechanical deformation automatic calibration and temperature deformation automatic calibration.

[0156] In a possible implementation, when the calibration type is temperature deformation automatic calibration, the calibration module can be configured to:

[0157] A temperature drift compensation model is constructed by using a long short-term memory artificial neural network combined with an attention mechanism, the temperature drift compensation model takes temperature data collected by a temperature sensor as input and outputs an axial offset;

[0158] The third data is input into the temperature drift compensation model, and a corresponding axial offset is output to calibrate the target device.

[0159] In a possible implementation, when the calibration type is mechanical deformation automatic calibration, the calibration module can be configured to:

[0160] Angular velocity and linear acceleration are obtained in real time by a six-axis inertial measurement unit;

[0161] A vibration compensation model is constructed by using a multi-modal frequency domain analysis and an adaptive filtering algorithm, the vibration compensation model takes angular velocity and linear acceleration as input and outputs a displacement offset;

[0162] The angular velocity and the linear acceleration are input into the vibration compensation model, and a corresponding displacement offset is output to calibrate the target device.

[0163] In a possible implementation, the feature points are composed of horizontal coordinates and vertical coordinates; the device can further include a deformation type determination module, which can be configured to:

[0164] The coordinate difference between the feature points and the initial feature points is calculated to obtain a coordinate difference, the coordinate difference including a horizontal coordinate difference and a vertical coordinate difference;

[0165] According to the coordinate difference, a deformation displacement type of the target device when mechanical deformation occurs is determined.

[0166] In a possible implementation, the deformation displacement type includes horizontal displacement and pitch displacement; the deformation type determination module can be specifically configured to:

[0167] It is determined whether the absolute value of the horizontal coordinate difference is greater than 0, and whether the absolute value of the vertical coordinate difference is greater than 0;

[0168] If the absolute value of the horizontal coordinate difference value is greater than 0 and the absolute value of the vertical coordinate difference value is equal to 0, it is indicated that the deformation displacement type of the target device when mechanical deformation occurs is horizontal displacement.

[0169] If the absolute value of the horizontal coordinate difference value is equal to 0 and the absolute value of the vertical coordinate difference value is greater than 0, it is indicated that the deformation displacement type of the target device when mechanical deformation occurs is pitching displacement.

[0170] If the absolute value of the horizontal coordinate difference value is greater than 0 and the absolute value of the vertical coordinate difference value is greater than 0, it is indicated that the deformation displacement type of the target device when mechanical deformation occurs is horizontal displacement and pitching displacement.

[0171] In the above embodiments, the description of each embodiment has its own emphasis, and the parts not described or recorded in a certain embodiment can be referred to the related description of other embodiments.

[0172] Those of ordinary skill in the art can realize that the template, unit and algorithm steps of each example described in combination with the embodiments disclosed herein can be realized by electronic hardware or a combination of computer software and electronic hardware. Whether the functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. A person skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0173] The modules / units, if realized in the form of software function units and sold or used as independent products, can be stored in a computer readable storage medium. Based on such understanding, all or part of the processes in the above-mentioned embodiment methods can also be completed by a computer program instructing related hardware, and the computer program can be stored in a computer readable storage medium. The computer program can implement the steps of each multi-modal radar and visual integrated machine dynamic verification and automatic calibration method embodiment described above when executed by a processor. The computer program includes computer program code, which can be in the form of source code, object code, executable files or some intermediate forms, etc. The computer readable medium can include any entity or device capable of carrying the computer program code, recording medium, U disk, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory, random access memory, electrical carrier signal, telecommunication signal and software distribution medium, etc.

[0174] The above-described embodiments are only used to illustrate the technical solutions of the present application, and are not intended to limit the present application; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: it can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement for part of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application, and should be included in the protection scope of the present application.

Claims

1. A multi-modal radar and visual integrated machine dynamic verification automatic calibration method, characterized in that, The application relates to a method for real-time automatic calibration of a target device. The method comprises the following steps: real-time acquisition of feature points of a target reference point at a fixed position, wherein the feature points are feature information, signal strength information and position information of the target reference point acquired by a target device, and the target device is any one of a high-definition camera, a millimeter wave radar and a thermal imaging camera in a multimodal radar and visual integrated machine; determination of whether the target device needs to be calibrated according to the feature points, and acquisition of first data, second data and third data when the target device needs to be calibrated, wherein the first data is displacement data collected by a displacement sensor, the second data is attitude data collected by an attitude sensor, and the third data is temperature data collected by a temperature sensor; determination of a calibration type of the target device according to the first data, the second data and the third data, and calibration of the target device according to the calibration type; wherein the calibration type comprises temperature deformation automatic calibration and mechanical deformation automatic calibration, and the determination of the calibration type of the target device according to the first data, the second data and the third data comprises: determination of whether the first data exceeds a preset displacement threshold value, whether the second data exceeds a preset attitude threshold value and whether the third data exceeds a preset temperature threshold value; if the first data exceeds the preset displacement threshold value, and / or the second data exceeds the preset attitude threshold value, and the third data does not exceed the preset temperature threshold value, it is determined that the calibration type required by the target device is the mechanical deformation automatic calibration; if the first data does not exceed the preset displacement threshold value, and the second data does not exceed the preset attitude threshold value, and the third data exceeds the preset temperature threshold value, it is determined that the calibration type required by the target device is the temperature deformation automatic calibration; 2. The multi-modal radar-vision integrated machine dynamic verification automatic calibration method according to claim 1, characterized in that, if the first data exceeds the preset displacement threshold value, and / or the second data exceeds the preset attitude threshold value, and the third data exceeds the preset temperature threshold value, it is determined that the calibration type required by the target device is the mechanical deformation automatic calibration and the temperature deformation automatic calibration. The real-time acquisition of the feature points of the target reference point at the fixed position comprises: when the target device is the high-definition camera, pixel information and position information of the target reference point photographed by the high-definition camera are extracted by using scale-invariant feature transformation and ORB algorithm, and the pixel information and the position information of the target reference point photographed by the high-definition camera are taken as the feature points of the target reference point acquired by the high-definition camera in real time; when the target device is the millimeter wave radar, echo amplitude and noise point position of the target reference point in a detection region of the millimeter wave radar are extracted by using Doppler velocity constraint, and the echo amplitude and the noise point position are taken as the feature points of the target reference point acquired by the millimeter wave radar in real time. When the target device is a thermal imaging camera, pixel information, position information and thermal radiation gradient of an image in which the target reference point is located are extracted by fusing the thermal radiation gradient with the spatiotemporal feature, and the pixel information, the position information and the thermal radiation gradient of the image in which the target reference point is located are taken as the feature point of the target reference point acquired by the thermal imaging camera in real time.

3. The multi-modal radar-camera integrated machine dynamic verification automatic calibration method according to claim 1, characterized in that, After the feature point of the target reference point at a fixed position is acquired in real time, the method further comprises: mapping the feature point to a spatiotemporal coordinate system of a multi-modal sensor based on Lie group SE(3) which has been constructed; Correspondingly, the method of determining whether the target device needs to be calibrated according to the feature point comprises: determining whether the target device needs to be calibrated according to the feature point mapped to the spatiotemporal coordinate system of the multi-modal sensor based on Lie group SE(3).

4. The multi-modal radar-vision integrated machine dynamic verification automatic calibration method according to claim 3, characterized in that, The method of determining whether the target device needs to be calibrated according to the feature point comprises: determining whether the position information of the feature point extracted by the target device and mapped to the spatiotemporal coordinate system of the multi-modal sensor based on Lie group SE(3) is the same as preset position information, the preset position information being position information of the target reference point acquired in the spatiotemporal coordinate system of the multi-modal sensor based on Lie group SE(3); if the position information of the feature point extracted by the target device and mapped to the spatiotemporal coordinate system of the multi-modal sensor based on Lie group SE(3) is the same as the preset position information, it is indicated that the target device does not need to be calibrated; if the position information of the feature point extracted by the target device and mapped to the spatiotemporal coordinate system of the multi-modal sensor based on Lie group SE(3) is not the same as the preset position information, it is indicated that the target device needs to be calibrated.

5. The multi-modal radar-camera integrated machine dynamic verification automatic calibration method according to claim 1, characterized in that, When the calibration type is temperature deformation automatic calibration, the method of calibrating the target device according to the calibration type comprises: a temperature drift compensation model is constructed by using a long short-term memory artificial neural network combined with an attention mechanism, the temperature drift compensation model taking temperature data collected by a temperature sensor as input and taking an axial offset as output; the third data is input into the temperature drift compensation model, and corresponding axial offset is output to calibrate the target device.

6. The multi-modal radar-camera integrated machine dynamic verification automatic calibration method according to claim 1, characterized in that, When the calibration type is mechanical deformation automatic calibration, the method of calibrating the target device according to the calibration type comprises: angular velocity and linear acceleration are acquired in real time by a six-axis inertial measurement unit; a vibration compensation model is constructed by using a multi-modal frequency domain analysis and an adaptive filtering algorithm, the vibration compensation model taking angular velocity and linear acceleration as input and taking displacement offset as output; the angular velocity and the linear acceleration are input into the vibration compensation model, and corresponding displacement offset is output to calibrate the target device.

7. The multi-modal radar-camera integrated machine dynamic verification automatic calibration method according to claim 6, characterized in that, The feature point is composed of an abscissa and an ordinate; before the angular velocity and the linear acceleration are acquired in real time by the six-axis inertial measurement unit, the method further comprises: a coordinate difference value between the feature point and an initial feature point is calculated to obtain a coordinate difference value, the coordinate difference value including an abscissa difference value and an ordinate difference value; According to the coordinate difference value, a deformation displacement type of the target device when mechanical deformation occurs is determined.

8. The multi-modal radar-vision all-in-one dynamic verification automatic calibration method according to claim 7, characterized in that, The deformation displacement type includes horizontal displacement and pitch displacement; and the determining of the deformation displacement type of the target device when mechanical deformation occurs according to the coordinate difference value includes: determining whether the absolute value of the horizontal coordinate difference value is greater than 0 and whether the absolute value of the vertical coordinate difference value is greater than 0; if the absolute value of the horizontal coordinate difference value is greater than 0 and the absolute value of the vertical coordinate difference value is equal to 0, it is indicated that the deformation displacement type of the target device when mechanical deformation occurs is horizontal displacement; if the absolute value of the horizontal coordinate difference value is equal to 0 and the absolute value of the vertical coordinate difference value is greater than 0, it is indicated that the deformation displacement type of the target device when mechanical deformation occurs is pitch displacement; if the absolute value of the horizontal coordinate difference value is greater than 0 and the absolute value of the vertical coordinate difference value is greater than 0, it is indicated that the deformation displacement type of the target device when mechanical deformation occurs is horizontal displacement and pitch displacement.

9. A multi-modal radar and visual integrated machine dynamic verification automatic calibration device, characterized in that, including: a feature point acquisition module, configured to acquire feature points of a target reference point at a fixed position in real time, the feature points being feature information, signal strength information and position information of the target reference point extracted by a target device, the target device being any one of a high-definition camera, a millimeter wave radar and a thermal imaging camera in a multi-modal radar and camera integrated machine; a calibration judgment and data acquisition module, configured to judge whether the target device needs to be calibrated according to the feature points, and acquire first data, second data and third data when the target device needs to be calibrated, the first data being displacement data collected by a displacement sensor, the second data being attitude data collected by an attitude sensor, and the third data being temperature data collected by a temperature sensor; a calibration module, configured to judge a calibration type of the target device according to the first data, the second data and the third data, and to calibrate the target device according to the calibration type; wherein the calibration type includes temperature deformation automatic calibration and mechanical deformation automatic calibration; and the calibration module is configured to: judge whether the first data exceeds a preset displacement threshold, whether the second data exceeds a preset attitude threshold, and whether the third data exceeds a preset temperature threshold; if the first data exceeds the preset displacement threshold, and / or the second data exceeds the preset attitude threshold, and the third data does not exceed the preset temperature threshold, it is determined that the calibration type required by the target device is the mechanical deformation automatic calibration; if the first data does not exceed the preset displacement threshold, and the second data does not exceed the preset attitude threshold, and the third data exceeds the preset temperature threshold, it is determined that the calibration type required by the target device is the temperature deformation automatic calibration; If the first data exceeds the preset displacement threshold value, and / or the second data exceeds the preset posture threshold value, and the third data exceeds the preset temperature threshold value, it is determined that the calibration type required by the target device is the mechanical deformation automatic calibration and the temperature deformation automatic calibration.

Images