A joint calibration method, device and equipment

Abstract

The application discloses a kind of joint calibration method, device and equipment, it is related to computer processing technical field, to solve the problem that existing sensor group is susceptible to environmental factors and lead to original calibration parameter failure, influence the accuracy of sensor data fusion.The method comprises: obtaining the first calibration parameter of sensor group current, wherein, sensor group includes main sensor and auxiliary sensor;According to the preset detection time interval, the feature points under the main sensor coordinate system and the feature points under the auxiliary sensor coordinate system are used to adaptively detect the first calibration parameter;In the case where the detection result indicates abnormality, the sensor group is recalibrated, and the first calibration parameter is updated based on the recalibrated calibration parameter.The application can update calibration parameter in time in the case where its state changes, guarantee the accuracy of sensor data fusion.

Description

Technical Field

[0001] This invention relates to the field of computer processing technology, and in particular to a joint calibration method, apparatus, and device. Background Technology

[0002] Sensor fusion is a trend in the development of intelligent driving safety. For example, the combined use of cameras and radar can integrate the characteristics of visual and ranging sensors, improving the overall perception capability and safety of intelligent driving. Joint calibration is a crucial part of sensor fusion, enabling the association of sensors of different types, locations, and orientations. This achieves the unification of data from different sources in time and space, allowing multiple sensor groups to collaborate and provide more reliable data. More stable calibration parameters ensure the stable and continuous operation of the system.

[0003] However, in real-world scenarios, sensor arrays are susceptible to environmental factors such as thunderstorms, accidental impacts, and aging support frames, which can cause changes in the sensor's own state, such as position and orientation, leading to the failure of the original calibration parameters and affecting the accuracy of sensor data fusion. Summary of the Invention

[0004] This invention provides a joint calibration method, apparatus, and device to address the problem that existing sensor arrays are susceptible to environmental factors that cause changes in their own state, resulting in the failure of the original calibration parameters and thus affecting the accuracy of sensor data fusion.

[0005] In a first aspect, embodiments of the present invention provide a joint calibration method, comprising:

[0006] Obtain the current first calibration parameters of the sensor group, wherein the sensor group includes a main sensor and a sub-sensor;

[0007] According to the preset detection time interval, the first calibration parameter is adaptively detected using feature points in the main sensor coordinate system and feature points in the sub-sensor coordinate system.

[0008] If the detection result indicates an abnormality, the sensor group is recalibrated, and the first calibration parameter is updated based on the recalibrated calibration parameter.

[0009] Optionally, the step of adaptively detecting the state of the first calibration parameter using feature points in the main sensor coordinate system and feature points in the secondary sensor coordinate system according to a preset detection time interval includes:

[0010] Using feature points in the main sensor coordinate system and feature points in the sub-sensor coordinate system, the first adaptive state detection is performed on the first calibration parameter after the first calibration parameter is initially calibrated at a first preset time interval;

[0011] If the Kth adaptive state detection is normal, the first calibration parameter is subjected to the (K+1)th adaptive state detection at a second preset time interval, wherein the second preset time is longer than the first preset time, and the second preset time is associated with K, where K is a positive integer.

[0012] Optionally, when K is less than or equal to the first threshold, the second preset duration increases as K increases; when K is greater than the first threshold, the second preset duration remains unchanged from the previous interval duration. The first threshold is an integer greater than 2.

[0013] Optionally, obtaining the current first calibration parameters of the sensor group includes:

[0014] The sensor group is calibrated to obtain the first calibration parameters;

[0015] The validity of the first calibration parameter is verified;

[0016] If the first calibration parameter is verified to be valid, the first calibration parameter is retained.

[0017] Optionally, calibrating the sensor group to obtain the first calibration parameters includes:

[0018] Obtain the first boundary coordinate information of the background marker of the main sensor in the main sensor coordinate system, and select boundary feature points from the first boundary coordinate information to obtain the first feature point set;

[0019] Obtain the second boundary coordinate information of the marker in the sub-sensor coordinate system, and select boundary feature points from the second boundary coordinate information to obtain the second feature point set;

[0020] Select n second feature points from the second feature point set, and select n first feature points from the first feature point set that correspond to the n second feature points respectively, where n is an integer greater than 1;

[0021] Based on the n first feature points and the n second feature points, calculate the rotation matrix and translation vector for transforming from the sub-sensor coordinate system to the main sensor coordinate system, wherein the first calibration parameters include the rotation matrix and translation vector.

[0022] Optionally, the validity verification of the first calibration parameter includes:

[0023] Select m second feature points from the second feature point set, wherein the m second feature points are different from the n second feature points, and m is a positive integer;

[0024] The coordinates of the m second feature points are transformed using the first calibration parameters to obtain the coordinates of the m second feature points in the main sensor coordinate system;

[0025] From the first feature point set, determine m first feature points that correspond to the m second feature points respectively;

[0026] Calculate the coordinate deviations between the m second feature points and the m first feature points in the main sensor coordinate system;

[0027] If the coordinate deviation between each second feature point and the corresponding first feature point in the main sensor coordinate system is less than the first preset deviation threshold, the first calibration parameter is determined to be valid.

[0028] Optionally, the adaptive state detection of the first calibration parameters using feature points in the main sensor coordinate system and feature points in the secondary sensor coordinate system includes:

[0029] Select g second feature points from the second feature point set, where g is an integer greater than 1;

[0030] The coordinates of the g second feature points are transformed using the first calibration parameters to obtain the coordinates of the g second feature points in the main sensor coordinate system;

[0031] From the first feature point set, determine g first feature points that correspond to the g second feature points respectively;

[0032] Calculate the coordinate deviations between the g second feature points and the g first feature points in the main sensor coordinate system, and record the number of coordinate deviations greater than or equal to the second preset deviation threshold.

[0033] If the number exceeds a preset value, the detection result is determined to be abnormal.

[0034] Optionally, the step of recalibrating the sensor group when the detection result indicates an abnormality includes:

[0035] If the detection result indicates an abnormality, the third feature point set in the main sensor coordinate system and the fourth feature point set in the sub-sensor coordinate system are reacquired.

[0036] The third feature point set is matched with the first feature point set, and the feature points matched between the third feature point set and the first feature point set are recorded as the fifth feature point set.

[0037] The fourth feature point set is matched with the second feature point set, and the feature point set matched by the fourth feature point set and the second feature point set is recorded as the sixth feature point set;

[0038] Based on the corresponding feature points in the fifth feature point set and the sixth feature point set, the second calibration parameters of the sensor group are calculated.

[0039] Secondly, embodiments of the present invention also provide a joint calibration apparatus, comprising:

[0040] The acquisition module is used to acquire the current first calibration parameters of the sensor group, wherein the sensor group includes a main sensor and a sub-sensor;

[0041] The detection module is used to adaptively detect the state of the first calibration parameter by using feature points in the main sensor coordinate system and feature points in the sub-sensor coordinate system according to a preset detection time interval.

[0042] The processing module is used to recalibrate the sensor group when the detection result indicates an abnormality, and update the first calibration parameter based on the recalibrated calibration parameter.

[0043] Optionally, the detection module includes:

[0044] The first detection unit is used to perform the first adaptive state detection of the first calibration parameter after the first calibration parameter is initially calibrated, by using feature points in the main sensor coordinate system and feature points in the sub-sensor coordinate system at a first preset time interval.

[0045] The second detection unit is used to perform the (K+1)th adaptive state detection on the first calibration parameter at a second preset time interval when the Kth adaptive state detection is normal, wherein the second preset time is longer than the first preset time, and the second preset time is associated with K, where K is a positive integer.

[0046] Optionally, when K is less than or equal to the first threshold, the second preset duration increases as K increases; when K is greater than the first threshold, the second preset duration remains unchanged from the previous interval duration. The first threshold is an integer greater than 2.

[0047] Optionally, the acquisition module includes:

[0048] A calibration submodule is used to calibrate the sensor group and obtain the first calibration parameters;

[0049] The verification submodule is used to verify the validity of the first calibration parameter;

[0050] The processing submodule is used to retain the first calibration parameter if the first calibration parameter is verified to be valid.

[0051] Optionally, the calibration submodule includes:

[0052] The first acquisition unit is used to acquire the first boundary coordinate information of the background marker of the main sensor in the main sensor coordinate system, and select boundary feature points from the first boundary coordinate information to obtain the first feature point set;

[0053] The second acquisition unit is used to acquire the second boundary coordinate information of the marker in the sub-sensor coordinate system, and select boundary feature points from the second boundary coordinate information to obtain the second feature point set;

[0054] The first selection unit is used to select n second feature points from the second feature point set, and to select n first feature points from the first feature point set that correspond to the n second feature points respectively, where n is an integer greater than 1;

[0055] The first calculation unit is used to calculate the rotation matrix and translation vector from the sub-sensor coordinate system to the main sensor coordinate system based on the n first feature points and the n second feature points, wherein the first calibration parameters include the rotation matrix and translation vector.

[0056] Optionally, the verification submodule includes:

[0057] The second selection unit is used to select m second feature points from the second feature point set, wherein the m second feature points are different from the n second feature points, and m is a positive integer;

[0058] The first processing unit is used to perform coordinate transformation on the m second feature points using the first calibration parameters to obtain the coordinates of the m second feature points in the main sensor coordinate system;

[0059] The first determining unit is configured to determine m first feature points from the first feature point set that correspond to the m second feature points respectively;

[0060] The second calculation unit is used to calculate the coordinate deviations between the m second feature points and the m first feature points in the main sensor coordinate system.

[0061] The second determining unit is used to determine that the first calibration parameter is valid when the coordinate deviation between each second feature point and the corresponding first feature point in the main sensor coordinate system is less than a first preset deviation threshold.

[0062] Optionally, the detection module includes:

[0063] The third selection unit is used to select g second feature points from the second feature point set, where g is an integer greater than 1;

[0064] The second processing unit is used to perform coordinate transformation on the g second feature points using the first calibration parameters to obtain the coordinates of the g second feature points in the main sensor coordinate system;

[0065] The third determining unit is used to determine from the first feature point set g first feature points that correspond to the g second feature points respectively;

[0066] The third calculation unit is used to calculate the coordinate deviation between the g second feature points and the g first feature points in the main sensor coordinate system, and record the number of coordinate deviations that are greater than or equal to the second preset deviation threshold.

[0067] The fourth determining unit is used to determine that the detection result is abnormal when the number is greater than a preset value.

[0068] Optionally, the processing module includes:

[0069] The third acquisition unit is used to reacquire the third feature point set in the main sensor coordinate system and the fourth feature point set in the sub-sensor coordinate system when the detection result indicates an abnormality.

[0070] The first matching unit is used to perform feature point matching between the third feature point set and the first feature point set, and to record the feature points matched between the third feature point set and the first feature point set as the fifth feature point set.

[0071] The second matching unit is used to perform feature point matching between the fourth feature point set and the second feature point set, and to record the feature point set matched between the fourth feature point set and the second feature point set as the sixth feature point set.

[0072] The fourth calculation unit is used to calculate the second calibration parameters of the sensor group based on the corresponding feature points in the fifth feature point set and the sixth feature point set.

[0073] Thirdly, embodiments of the present invention also provide a joint calibration device, including: a transceiver, a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps in the joint calibration method described above.

[0074] Fourthly, embodiments of the present invention also provide a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps in the joint calibration method described above.

[0075] In this embodiment of the invention, the current first calibration parameters of the sensor group are obtained, wherein the sensor group includes a main sensor and a sub-sensor; according to a preset detection time interval, the first calibration parameters are adaptively state-detected using feature points in the coordinate system of the main sensor and the coordinate system of the sub-sensor; if the detection result indicates an anomaly, the sensor group is recalibrated, and the first calibration parameters are updated based on the recalibrated calibration parameters. Thus, by adaptively detecting the calibration parameters of the sensor group and recalibrating the sensor group when an anomaly is detected, the calibration parameters can be updated promptly when the state changes, ensuring the accuracy of sensor data fusion. Attached Figure Description

[0076] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0077] Figure 1 This is a flowchart of the joint calibration method provided in the embodiments of the present invention;

[0078] Figure 2 This is an example flowchart of the joint calibration method provided in the embodiments of the present invention;

[0079] Figure 3 This is a structural diagram of the joint calibration device provided in the embodiments of the present invention;

[0080] Figure 4 This is a structural diagram of the joint calibration device provided in an embodiment of the present invention. Detailed Implementation

[0081] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0082] See Figure 1 , Figure 1 This is a flowchart of the joint calibration method provided in the embodiments of the present invention, such as... Figure 1 As shown, it includes the following steps:

[0083] Step 101: Obtain the current first calibration parameters of the sensor group, wherein the sensor group includes a main sensor and a sub-sensor.

[0084] The aforementioned sensor group can consist of multiple sensors that require sensor data fusion. Typically, the deployment environment of the sensor group can be determined based on actual usage requirements, and the sensors in the sensor group can be deployed in basically the same location, with similar orientations or angles. For example, for scenarios involving the detection of roadside data, a sensor group consisting of radar and cameras can be deployed at an appropriate location beside the road, or a sensor group for detecting driving data can be deployed on a vehicle.

[0085] For a given sensor group, one sensor can be selected as the main sensor, and the rest as secondary sensors. For example, if the sensor group includes a camera, radar, and weather sensors, then the camera can be the main sensor, and the radar and weather sensors can be secondary sensors.

[0086] To achieve accurate fusion of detection data from each sensor in the sensor array, the sensor array can be calibrated after deployment, and the calibration parameters obtained can be stored. The calibration method can employ algorithms such as Perspective-n-Point (PnP) or Zhang Zhengyou's calibration method. If the calibration parameters are updated, the updated parameters will replace the currently stored ones. These calibration parameters can refer to parameters such as rotation matrices and translation vectors used for coordinate transformation between the main sensor coordinate system and the sub-sensor coordinate system.

[0087] The aforementioned acquisition of the current first calibration parameter of the sensor group can be the acquisition of currently stored calibration parameters. Specifically, if the sensor group has been initially calibrated and the calibration parameters have not been updated, the acquired first calibration parameter is the initial calibration parameter; if the calibration parameters have been updated after the initial calibration of the sensor group, the acquired parameter is the latest calibration parameter.

[0088] Step 102: According to the preset detection time interval, adaptive state detection is performed on the first calibration parameter using feature points in the main sensor coordinate system and feature points in the sub-sensor coordinate system.

[0089] In this embodiment of the invention, after each acquisition or update of the calibration parameters of the sensor group, adaptive state detection can be performed on the current calibration parameters according to a preset detection time interval to detect whether the current calibration parameters are valid. Specifically, the current calibration parameters can be verified using several feature points in the main sensor coordinate system and several feature points in the sub-sensor coordinate system. This verifies whether the coordinates of the feature points in the sub-sensor coordinate system, after coordinate transformation using the current calibration parameters, are substantially consistent with the coordinates of the corresponding feature points in the main sensor coordinate system. If most are substantially consistent, it can be determined that the current calibration parameters are still valid; otherwise, it can be determined that the current calibration parameters have become invalid, indicating a change in the sensor state within the sensor group, requiring an update of the calibration parameters. Here, "substantially consistent" can mean that the coordinate positions are the same or the coordinate deviations are within a certain range.

[0090] The primary sensor coordinate system can be a three-dimensional coordinate system established with the center of the primary sensor as the center, and the secondary sensor coordinate system can be a three-dimensional coordinate system established with the center of the secondary sensor as the center. When there are multiple secondary sensors, each secondary sensor can have its own three-dimensional coordinate system centered at its own center. Feature points in the primary sensor coordinate system can be boundary feature points of background markers in the raw data detected by the primary sensor, and feature points in the secondary sensor coordinate system can be corresponding boundary feature points of the same background marker in the raw data detected by the secondary sensors.

[0091] The preset detection time interval can be a fixed time period, such as once a day, once every three days, once a week, etc., or a variable time period, such as once every day in the first few times, and once every five days in the later times, etc.

[0092] Optionally, step 102 includes:

[0093] Using feature points in the main sensor coordinate system and feature points in the sub-sensor coordinate system, the first adaptive state detection is performed on the first calibration parameter after the first calibration parameter is initially calibrated at a first preset time interval;

[0094] If the Kth adaptive state detection is normal, the first calibration parameter is subjected to the (K+1)th adaptive state detection at a second preset time interval, wherein the second preset time is longer than the first preset time, and the second preset time is associated with K, where K is a positive integer.

[0095] In one implementation, adaptive state detection of the current calibration parameter can be performed at variable time intervals. Specifically, after the first calibration parameter is obtained for the first time, an adaptive state detection of the first calibration parameter can be performed at a first preset time interval. If the first calibration parameter is the calibration parameter obtained in the initial calibration, an adaptive state detection of the initial calibration parameter can be performed at a first preset time interval after the initial calibration. To ensure the effectiveness of the calibration parameter, the interval for the first detection can be set to a relatively short duration; for example, the first preset time interval can be set to a duration not exceeding three days, such as one day or two days.

[0096] If the first adaptive state detection fails, the process can proceed to step 103 to recalibrate the sensor and periodically detect the updated calibration parameters according to this implementation method. If the first adaptive state detection is normal, the first calibration parameters can be detected a second time at a second preset time interval. The second preset time interval can be slightly longer than the first preset time interval, such as the first preset time interval being one day and the second preset time interval being two days.

[0097] If the second adaptive state detection is normal, the first calibration parameter can be subjected to a third adaptive state detection at a third preset time interval. The third preset time interval can be slightly longer than the first preset time interval and also slightly longer than the second preset time interval, such as the first preset time interval being one day, the second preset time interval being two days, and the third preset time interval being three days.

[0098] For the subsequent Kth adaptive state detection, the first calibration parameter can be adaptively detected at longer intervals as K increases (indicating that the deployment environment of the sensor group is relatively stable). Alternatively, when K increases to a certain value, the current interval can be kept unchanged, and the first calibration parameter can be detected once every such interval until the detection result indicates an abnormality.

[0099] In other words, when K is less than or equal to the first threshold, the second preset duration increases as K increases; when K is greater than the first threshold, the second preset duration remains unchanged from the previous interval duration. The first threshold is an integer greater than 2.

[0100] The first threshold can be reasonably set according to actual needs or multiple experiments. For example, the first threshold can be 4 or 5. Thus, when the number of detections of the first calibration parameter is less than or equal to this threshold, the time interval between subsequent detections is greater than the time interval between previous detections. Conversely, when the number of detections of the first calibration parameter is greater than this threshold, the time interval between subsequent detections can be the same as the time interval between previous detections. For example, the first adaptive state detection can be performed on the first calibration parameter one day after its acquisition. If the detection result is normal, the next adaptive state detection is performed two days later. If the detection result is still normal, the next adaptive state detection is performed three days later, and so on, until a detection is performed five days later. After the detection interval is five days, subsequent detections are performed every five days. If a detection result is confirmed to be abnormal and re-calibration is required, the cycle returns to one day later for detection, and the next detection cycle iteration begins.

[0101] In addition, if the secondary sensor includes a meteorological sensor, then when the meteorological sensor detects severe weather, such as winds of level 7 or above, or heavy rain or above, additional adaptive state detection can be performed. If the detection result is abnormal, then re-joint calibration is required, and the cycle period is updated after calibration.

[0102] In this way, by performing adaptive state detection on the current calibration parameters at variable time intervals, not only can abnormal states be detected in a timely manner, but detection costs can also be reduced to a certain extent.

[0103] Step 103: If the detection result indicates an abnormality, the sensor group is recalibrated, and the first calibration parameter is updated based on the recalibrated calibration parameter.

[0104] If the adaptive detection result indicates that the current calibration parameters are valid and do not need to be updated, then the current calibration parameters are invalid. If the adaptive detection result indicates that the current calibration parameters are invalid, and the sensor data of the sensor group cannot be accurately fused based on the current calibration parameters, the sensor group needs to be recalibrated to update the current calibration parameters.

[0105] Specifically, the above recalibration can be performed in the same way as the initial calibration, using the PnP algorithm or Zhang Zhengyou calibration method to recalibrate the sensor group. Alternatively, it can involve re-acquiring the feature points in the main sensor coordinate system and the feature points in the sub-sensor coordinate system, and combining them with the feature point parameters used in the previous calibration to recalculate the calibration parameters, and updating the stored first calibration parameters with the recalibrated calibration parameters.

[0106] Optionally, step 101 includes:

[0107] The sensor group is calibrated to obtain the first calibration parameters;

[0108] The validity of the first calibration parameter is verified;

[0109] If the first calibration parameter is verified to be valid, the first calibration parameter is retained.

[0110] In one embodiment, obtaining the first calibration parameter of the sensor group can be achieved by calibrating the sensor group. The calibration method can employ the PnP algorithm, Zhang Zhengyou calibration method, or other calibration methods. After obtaining the first calibration parameter through calibration, its validity can be verified. Specifically, this can be done using several feature points in the main sensor coordinate system and the sub-sensor coordinate system to check the validity of the first calibration parameter. The system determines whether the coordinates of a feature point in one coordinate system, after transformation using the first calibration parameter, are substantially consistent with the coordinates of the corresponding feature point in the other coordinate system. If they are substantially consistent, the first calibration parameter is considered valid; otherwise, it is considered invalid and recalibration is required until the validity verification is passed. If the first calibration parameter is verified to be valid, it can be retained, meaning it can be stored as the currently valid calibration parameter of the sensor group.

[0111] In this way, by calibrating the sensor array to obtain calibration parameters and verifying the validity of the calibration parameters, the accuracy and reliability of the calibration parameters can be guaranteed, thereby ensuring the accuracy of sensor data fusion based on the calibration parameters.

[0112] Optionally, calibrating the sensor group to obtain the first calibration parameters includes:

[0113] Obtain the first boundary coordinate information of the background marker of the main sensor in the main sensor coordinate system, and select boundary feature points from the first boundary coordinate information to obtain the first feature point set;

[0114] Obtain the second boundary coordinate information of the marker in the sub-sensor coordinate system, and select boundary feature points from the second boundary coordinate information to obtain the second feature point set;

[0115] Select n second feature points from the second feature point set, and select n first feature points from the first feature point set that correspond to the n second feature points respectively, where n is an integer greater than 1;

[0116] Based on the n first feature points and the n second feature points, calculate the rotation matrix and translation vector for transforming from the sub-sensor coordinate system to the main sensor coordinate system, wherein the first calibration parameters include the rotation matrix and translation vector.

[0117] In one implementation, the calibration parameters can be obtained by acquiring feature points in the main sensor coordinate system and the secondary sensor coordinate system respectively, and by using the PnP algorithm to calculate the transformation parameters between the two coordinate systems, namely the rotation matrix and the translation vector, based on several corresponding feature points in the two coordinate systems.

[0118] Specifically, before the initial calibration of the sensor group, the deployment environment of the sensor group can be determined first, and background markers with clear outlines and obvious features can be selected according to the deployment environment. For example, for roadside sensors, roadside flower beds, road signs, etc., can be selected as background markers. Then, the boundary coordinate information of the background markers in the main sensor coordinate system and the sub-sensor coordinate system can be extracted respectively, and feature points in the boundary coordinates, such as inflection points and endpoints, can be selected. To ensure the accuracy of the calibration parameter calculation, as many feature points as possible can be selected. Thus, the point set composed of the boundary feature points extracted in the main sensor coordinate system can be denoted as the first feature point set, and the point set composed of the boundary feature points extracted in the sub-sensor coordinate system can be denoted as the second feature point set.

[0119] Then, n first feature points can be selected from the first feature point set, and n second feature points corresponding to the n first feature points can be selected from the second feature point set. The correspondence between the first and second feature points can refer to feature points at the same position in the background marker. The value of n can be determined based on the actual number of points required to calculate the calibration parameters. For example, if at least 6 points are needed to calculate the calibration parameters, then n can be 6 or greater. It should be noted that, based on the detection range of the sub-sensor, to avoid the disappearance of boundary feature points due to sensor position shift, a selection fence can be set, removing 10% of the range from the top, bottom, left, and right boundaries. Within the retained central detection range, n first feature points are randomly selected from the first feature point set, and the corresponding n second feature points are obtained from the second feature point set.

[0120] Finally, based on the n first feature points and the n second feature points, the PnP algorithm can be used to calculate the rotation matrix and translation vector from the sub-sensor coordinate system to the main sensor coordinate system. The rotation matrix and translation vector are the calibration parameters that need to be calculated.

[0121] The PnP algorithm refers to an algorithm that uses multiple pairs of 3D and 2D matching points to minimize the camera's extrinsic parameters by minimizing the reprojection error, given both known and unknown camera intrinsic parameters. Simply put, it's about estimating the camera's pose (i.e., the camera's orientation in coordinate system A) given the coordinates of n points in 3D space (relative to a specified coordinate system A) and their 2D projection positions. For example, in an image, knowing the relative coordinates of at least four specific points in 3D space allows us to estimate the camera's pose relative to these points, or in other words, the pose of these 3D points in the camera's coordinate system. Here, pose includes both position and orientation, representing a 6-DOF state.

[0122] Specifically, the relationship between 3D coordinate points (X,Y,Z) in the point cloud and 2D coordinate points (u,v) in the image is as shown in formula (1): in, Let R be the camera intrinsic parameter matrix, λ be a 3×3 rotation matrix from 3D coordinates to 2D coordinates, λ be the scale factor from the point cloud coordinate system to the image coordinate system, (u,v) be the ideal distortion-free pixel coordinates, and T be a 3×1 translation vector from 3D coordinates to 2D coordinates. R and T are the extrinsic parameters required for joint calibration. The above formula (1) can be written as formula (2):

[0123] The relationship between a point (x, y, z) in the image coordinate system and a point (X, Y, Z) in the point cloud coordinate system is shown in formula (3):

[0124] In this way, the calibration parameters of the sensor group can be obtained more accurately through this implementation method, without the need for a lot of hardware equipment, and the operation is simple.

[0125] Optionally, the validity verification of the first calibration parameter includes:

[0126] Select m second feature points from the second feature point set, wherein the m second feature points are different from the n second feature points, and m is a positive integer;

[0127] The coordinates of the m second feature points are transformed using the first calibration parameters to obtain the coordinates of the m second feature points in the main sensor coordinate system;

[0128] From the first feature point set, determine m first feature points that correspond to the m second feature points respectively;

[0129] Calculate the coordinate deviations between the m second feature points and the m first feature points in the main sensor coordinate system;

[0130] If the coordinate deviation between each second feature point and the corresponding first feature point in the main sensor coordinate system is less than the first preset deviation threshold, the first calibration parameter is determined to be valid.

[0131] In one implementation, the validity of the obtained first calibration parameters can be verified by additionally selecting several feature points from the feature point sets in the main sensor coordinate system and the sub-sensor coordinate system.

[0132] Specifically, m additional second feature points can be selected from the second feature point set under the secondary sensor coordinate system. These m second feature points are different from the n second feature points. The number m can be determined according to actual needs. When high accuracy of the calibration parameters is required, a larger number of feature points can be selected to verify the current calibration parameters. Furthermore, m first feature points corresponding to the m second feature points can be determined from the first feature point set under the primary sensor coordinate system, meaning there is a one-to-one correspondence between the m first feature points and the m second feature points.

[0133] Then, the first calibration parameters can be used to transform the coordinates of the m second feature points to obtain their coordinates in the main sensor coordinate system. For example, the coordinates of the m second feature points can be multiplied by a rotation matrix and then a translation vector added to convert them into coordinates in the main sensor coordinate system. The transformed coordinates of the m second feature points are then compared one-to-one with the coordinates of the m first feature points. The deviation between the transformed coordinates of each second feature point and the corresponding first feature point coordinates is calculated, and it is determined whether each deviation is less than a first preset deviation threshold. The first preset deviation threshold can be determined based on the accuracy requirements of the calibration parameters in actual applications; the higher the accuracy requirement, the smaller the first preset deviation threshold can be set.

[0134] In this embodiment, relatively strict requirements can be set for the calibration parameters obtained during the initial calibration to ensure their reliability. The first calibration parameter is determined to be valid only if the coordinate deviation between each second feature point and the corresponding first feature point in the main sensor coordinate system is less than the first preset deviation threshold; otherwise, the first calibration parameter is determined to be invalid.

[0135] In this way, by utilizing the remaining feature points in the main sensor coordinate system and the sub-sensor coordinate system, the validity of the obtained first calibration parameters can be verified, ensuring the validity of the verification and guaranteeing that the calibration parameters have high reliability.

[0136] Optionally, step 102 includes:

[0137] Select g second feature points from the second feature point set, where g is an integer greater than 1;

[0138] The coordinates of the g second feature points are transformed using the first calibration parameters to obtain the coordinates of the g second feature points in the main sensor coordinate system;

[0139] From the first feature point set, determine g first feature points that correspond to the g second feature points respectively;

[0140] Calculate the coordinate deviations between the g second feature points and the g first feature points in the main sensor coordinate system, and record the number of coordinate deviations greater than or equal to the second preset deviation threshold.

[0141] If the number exceeds a preset value, the detection result is determined to be abnormal.

[0142] In one embodiment, the adaptive state detection may utilize a large number of feature points in the main sensor coordinate system and the sub-sensor coordinate system to verify whether the current calibration parameters are still valid.

[0143] Specifically, g second feature points can be selected from the set of second feature points in the secondary sensor coordinate system. To ensure detection effectiveness, g can be greater than m; in one approach, g can be equal to m+n. That is, the n second feature points used during calibration and the m second feature points used to verify the validity of the calibration parameters can be used to adaptively detect the state of the first calibration parameters. Furthermore, g first feature points corresponding to the g second feature points can be determined from the set of first feature points in the primary sensor coordinate system, meaning there is a one-to-one correspondence between the g first feature points and the g second feature points.

[0144] Then, the coordinates of the g second feature points can be transformed using the first calibration parameters to obtain their coordinates in the main sensor coordinate system. For example, the coordinates of the g second feature points can be multiplied by a rotation matrix and then a translation vector added to convert them into coordinates in the main sensor coordinate system. The converted coordinates of the g second feature points are then compared one by one with the coordinates of the g first feature points. The deviation between the converted coordinates of each second feature point and the corresponding first feature point coordinates is calculated, and it is determined whether each deviation is less than a second preset deviation threshold. The second preset deviation threshold can be determined based on the accuracy requirements of the calibration parameters in actual applications. The higher the accuracy requirement, the smaller the second preset deviation threshold can be set. The second preset deviation threshold can be the same as or different from the aforementioned first preset deviation threshold.

[0145] When calculating the coordinate deviation between the transformed coordinates of each second feature point and the corresponding first feature point, the number of deviations exceeding a second preset deviation threshold can be recorded. If this number exceeds a certain value, such as a preset value, the detection result can be determined to be abnormal. The preset value can be determined according to actual needs; the higher the required detection accuracy, the smaller this value can be set. In one approach, when g equals m+n, if the number of coordinate deviations exceeding the second preset deviation threshold exceeds m / 2, the current sensor group's state is considered abnormal; otherwise, the sensor group's state is considered normal.

[0146] In this way, the abnormal state of the sensor group can be detected more accurately and effectively, and the abnormal situation can be discovered in time, so as to update the calibration parameters in a timely manner.

[0147] Optionally, the step of recalibrating the sensor group when the detection result indicates an abnormality includes:

[0148] If the detection result indicates an abnormality, the third feature point set in the main sensor coordinate system and the fourth feature point set in the sub-sensor coordinate system are reacquired.

[0149] The third feature point set is matched with the first feature point set, and the feature points matched between the third feature point set and the first feature point set are recorded as the fifth feature point set.

[0150] The fourth feature point set is matched with the second feature point set, and the feature point set matched by the fourth feature point set and the second feature point set is recorded as the sixth feature point set;

[0151] Based on the corresponding feature points in the fifth feature point set and the sixth feature point set, the second calibration parameters of the sensor group are calculated.

[0152] In one implementation, if the adaptive detection result indicates an anomaly, the joint calibration parameters can be recalculated. Specifically, the third feature point set in the main sensor coordinate system and the fourth feature point set in the sub-sensor coordinate system can be re-acquired. For example, the edge detection method can be used again to extract the boundary coordinate information of the background marker in the main sensor coordinate system and the sub-sensor coordinate system, respectively, and select feature points in the boundary coordinates, such as inflection points and endpoints, thereby obtaining the point set composed of these feature points in the main sensor coordinate system, which can be denoted as the third feature point set, and obtaining the point set composed of these feature points in the sub-sensor coordinate system, which can be denoted as the fourth feature point set.

[0153] Then, the third feature point set can be matched with the first feature point set extracted in the previous calibration. Specifically, the Simultaneous Localization and Mapping (SLAM) algorithm can be used to match the feature points to obtain the set of feature points in the main sensor coordinate system that can match the original state in the current state, which is the fifth feature point set. Similarly, the fourth feature point set can be matched with the second feature point set extracted in the previous calibration to obtain the set of feature points in the secondary sensor coordinate system that can match the original state in the current state, which is the sixth feature point set.

[0154] Finally, based on the mapping relationship between the first and second feature point sets (i.e., the feature points with corresponding relationships), the mapping relationship between the fifth and sixth feature point sets can be determined. This means determining the feature points in the fifth and sixth feature point sets that have corresponding relationships. Some feature points in the fifth and sixth feature point sets may not have corresponding relationships; these can be summarized and used as the feature points for recalibration parameters. In other words, based on the feature points in the fifth and sixth feature point sets that have corresponding relationships, the second calibration parameters of the sensor group can be calculated. Specifically, using the corresponding feature points in the fifth and sixth feature point sets, the PnP algorithm can be used to calculate the new rotation matrix and translation vector for transforming from the sub-sensor coordinate system to the main sensor coordinate system, which are the updated calibration parameters.

[0155] Of course, redundant feature points in the fifth and sixth feature point sets (i.e., feature points not used to calculate the updated calibration parameters) can also be used to verify the validity of the second calibration parameter. The verification method is the same as described above: additional h feature points are selected from the sixth feature point set in the sub-sensor coordinate system. These points are different from the feature points used to calculate the updated joint calibration parameters. If the coordinates of these h feature points after the updated joint calibration parameter transformation deviate from the coordinates of the corresponding feature points in the main sensor coordinate system by less than a preset deviation threshold, then this joint calibration parameter is considered a valid value, and it overwrites the previous calibration parameter.

[0156] In this way, by re-acquiring feature points and combining them with previous feature points, the calibration parameters of the sensor group can be recalculated, ensuring that the calibration parameters can adapt to changes in the state of the sensor group and ensuring the effectiveness of the calibration parameters.

[0157] The following is combined with, for example Figure 2 The implementation process of this invention will be illustrated by example:

[0158] Step 21: Perform initial calibration on the sensor group and record the calibration parameters.

[0159] Step 22: Perform adaptive state detection on the current calibration parameters.

[0160] Step 23: Determine whether there is any abnormality in the current sensor group based on the detection results.

[0161] Step 24: If there are no abnormalities, retain the current calibration parameters.

[0162] Step 25: If an anomaly exists, extract the coordinates of the new feature points.

[0163] Step 26: Update the calibration parameters based on the new feature point coordinates.

[0164] The joint calibration method of this invention obtains the current first calibration parameters of a sensor group, wherein the sensor group includes a main sensor and a sub-sensor; according to a preset detection time interval, adaptive state detection is performed on the first calibration parameters using feature points in the coordinate systems of the main sensor and the sub-sensor; if the detection result indicates an anomaly, the sensor group is recalibrated, and the first calibration parameters are updated based on the recalibrated calibration parameters. Thus, by adaptively detecting the calibration parameters of the sensor group and recalibrating the sensor group when an anomaly is detected, the calibration parameters can be updated promptly when the state changes, ensuring the accuracy of sensor data fusion.

[0165] This invention proposes a method for adaptively adjusting the joint calibration parameters of a sensor group in response to abnormal conditions. It promptly detects changes in the state of the sensor group, automatically corrects and updates the calibration parameters to ensure effective sensor group collaboration. Compared to existing technologies, this invention completes joint calibration by automatically acquiring scene features and can respond to changes in joint calibration under the influence of abnormal conditions. It adaptively detects the accuracy of the current calibration state. When the deviation of the current calibration parameters is large, it automatically acquires the associated feature points of the current state based on the feature matching relationship between the current state and the initial state, and calculates the joint calibration parameters between the sensor groups based on these associated feature points.

[0166] This invention also provides a joint calibration device. See [link to related document]. Figure 3 , Figure 3 This is a structural diagram of the joint calibration device provided in an embodiment of the present invention. Since the principle of the joint calibration device in solving the problem is similar to that of the joint calibration method in the embodiment of the present invention, the implementation of the joint calibration device can refer to the implementation of the method, and the repeated parts will not be described again.

[0167] like Figure 3 As shown, the joint calibration device 300 includes:

[0168] The acquisition module 301 is used to acquire the current first calibration parameters of the sensor group, wherein the sensor group includes a main sensor and a sub-sensor;

[0169] The detection module 302 is used to adaptively detect the first calibration parameter by using feature points in the main sensor coordinate system and feature points in the sub-sensor coordinate system according to a preset detection time interval.

[0170] The processing module 303 is used to recalibrate the sensor group when the detection result indicates an abnormality, and update the first calibration parameter based on the recalibrated calibration parameter.

[0171] Optionally, the detection module 302 includes:

[0172] The first detection unit is used to perform the first adaptive state detection of the first calibration parameter after the first calibration parameter is initially calibrated, by using feature points in the main sensor coordinate system and feature points in the sub-sensor coordinate system at a first preset time interval.

[0173] The second detection unit is used to perform the (K+1)th adaptive state detection on the first calibration parameter at a second preset time interval when the Kth adaptive state detection is normal, wherein the second preset time is longer than the first preset time, and the second preset time is associated with K, where K is a positive integer.

[0174] Optionally, when K is less than or equal to the first threshold, the second preset duration increases as K increases; when K is greater than the first threshold, the second preset duration remains unchanged from the previous interval duration. The first threshold is an integer greater than 2.

[0175] Optionally, the acquisition module 301 includes:

[0176] A calibration submodule is used to calibrate the sensor group and obtain the first calibration parameters;

[0177] The verification submodule is used to verify the validity of the first calibration parameter;

[0178] The processing submodule is used to retain the first calibration parameter if the first calibration parameter is verified to be valid.

[0179] Optionally, the calibration submodule includes:

[0180] The first acquisition unit is used to acquire the first boundary coordinate information of the background marker of the main sensor in the main sensor coordinate system, and select boundary feature points from the first boundary coordinate information to obtain the first feature point set;

[0181] The second acquisition unit is used to acquire the second boundary coordinate information of the marker in the sub-sensor coordinate system, and select boundary feature points from the second boundary coordinate information to obtain the second feature point set;

[0182] The first selection unit is used to select n second feature points from the second feature point set, and to select n first feature points from the first feature point set that correspond to the n second feature points respectively, where n is an integer greater than 1;

[0183] The first calculation unit is used to calculate the rotation matrix and translation vector from the sub-sensor coordinate system to the main sensor coordinate system based on the n first feature points and the n second feature points, wherein the first calibration parameters include the rotation matrix and translation vector.

[0184] Optionally, the verification submodule includes:

[0185] The second selection unit is used to select m second feature points from the second feature point set, wherein the m second feature points are different from the n second feature points, and m is a positive integer;

[0186] The first processing unit is used to perform coordinate transformation on the m second feature points using the first calibration parameters to obtain the coordinates of the m second feature points in the main sensor coordinate system;

[0187] The first determining unit is configured to determine m first feature points from the first feature point set that correspond to the m second feature points respectively;

[0188] The second calculation unit is used to calculate the coordinate deviations between the m second feature points and the m first feature points in the main sensor coordinate system.

[0189] The second determining unit is used to determine that the first calibration parameter is valid when the coordinate deviation between each second feature point and the corresponding first feature point in the main sensor coordinate system is less than a first preset deviation threshold.

[0190] Optionally, the detection module 302 includes:

[0191] The third selection unit is used to select g second feature points from the second feature point set, where g is an integer greater than 1;

[0192] The second processing unit is used to perform coordinate transformation on the g second feature points using the first calibration parameters to obtain the coordinates of the g second feature points in the main sensor coordinate system;

[0193] The third determining unit is used to determine from the first feature point set g first feature points that correspond to the g second feature points respectively;

[0194] The third calculation unit is used to calculate the coordinate deviation between the g second feature points and the g first feature points in the main sensor coordinate system, and record the number of coordinate deviations that are greater than or equal to the second preset deviation threshold.

[0195] The fourth determining unit is used to determine that the detection result is abnormal when the number is greater than a preset value.

[0196] Optionally, the processing module 303 includes:

[0197] The third acquisition unit is used to reacquire the third feature point set in the main sensor coordinate system and the fourth feature point set in the sub-sensor coordinate system when the detection result indicates an abnormality.

[0198] The first matching unit is used to perform feature point matching between the third feature point set and the first feature point set, and to record the feature points matched between the third feature point set and the first feature point set as the fifth feature point set.

[0199] The second matching unit is used to perform feature point matching between the fourth feature point set and the second feature point set, and to record the feature point set matched between the fourth feature point set and the second feature point set as the sixth feature point set.

[0200] The fourth calculation unit is used to calculate the second calibration parameters of the sensor group based on the corresponding feature points in the fifth feature point set and the sixth feature point set.

[0201] The joint calibration device provided in this embodiment of the invention can execute the above method embodiment, and its implementation principle and technical effect are similar, so it will not be described again here.

[0202] The joint calibration device 300 of this embodiment acquires the current first calibration parameters of a sensor group, wherein the sensor group includes a main sensor and a sub-sensor. According to a preset detection time interval, it adaptively detects the first calibration parameters using feature points in the coordinate systems of the main sensor and the sub-sensor. If the detection result indicates an anomaly, it recalibrates the sensor group and updates the first calibration parameters based on the recalibrated calibration parameters. Thus, by adaptively detecting the calibration parameters of the sensor group and recalibrating the sensor group when an anomaly is detected, the calibration parameters can be updated promptly when the state changes, ensuring the accuracy of sensor data fusion.

[0203] This invention also provides a joint calibration device. Since the principle behind the problem solved by the joint calibration device is similar to the joint calibration method in this invention, the implementation of the joint calibration device can be found in the implementation of the method, and repeated details will not be elaborated further. Figure 4 As shown, the joint calibration device of this invention includes:

[0204] Processor 400 is used to read the program from memory 420 and execute the following procedures:

[0205] Obtain the current first calibration parameters of the sensor group, wherein the sensor group includes a main sensor and a sub-sensor;

[0206] According to the preset detection time interval, the first calibration parameter is adaptively detected using feature points in the main sensor coordinate system and feature points in the sub-sensor coordinate system.

[0207] If the detection result indicates an abnormality, the sensor group is recalibrated, and the first calibration parameter is updated based on the recalibrated calibration parameter.

[0208] Transceiver 410 is used to receive and send data under the control of processor 400.

[0209] Among them, Figure 4In this context, the bus architecture may include any number of interconnected buses and bridges, specifically linking various circuits together, represented by one or more processors (processor 400) and memory (memory 420). The bus architecture may also link together various other circuits such as peripheral devices, voltage regulators, and power management circuits, which are well known in the art and therefore will not be described further herein. The bus interface provides an interface. The transceiver 410 may be multiple elements, including transmitters and transceivers, providing a unit for communicating with various other devices over a transmission medium. The processor 400 is responsible for managing the bus architecture and general processing, and the memory 420 may store data used by the processor 400 during operation.

[0210] Optionally, the processor 400 is also used to read the program from the memory 420 and perform the following steps:

[0211] Using feature points in the main sensor coordinate system and feature points in the sub-sensor coordinate system, the first adaptive state detection is performed on the first calibration parameter after the first calibration parameter is initially calibrated at a first preset time interval;

[0212] If the Kth adaptive state detection is normal, the first calibration parameter is subjected to the (K+1)th adaptive state detection at a second preset time interval, wherein the second preset time is longer than the first preset time, and the second preset time is associated with K, where K is a positive integer.

[0213] Optionally, when K is less than or equal to the first threshold, the second preset duration increases as K increases; when K is greater than the first threshold, the second preset duration remains unchanged from the previous interval duration. The first threshold is an integer greater than 2.

[0214] Optionally, the processor 400 is also used to read the program from the memory 420 and perform the following steps:

[0215] The sensor group is calibrated to obtain the first calibration parameters;

[0216] The validity of the first calibration parameter is verified;

[0217] If the first calibration parameter is verified to be valid, the first calibration parameter is retained.

[0218] Optionally, the processor 400 is also used to read the program from the memory 420 and perform the following steps:

[0219] Obtain the first boundary coordinate information of the background marker of the main sensor in the main sensor coordinate system, and select boundary feature points from the first boundary coordinate information to obtain the first feature point set;

[0220] Obtain the second boundary coordinate information of the marker in the sub-sensor coordinate system, and select boundary feature points from the second boundary coordinate information to obtain the second feature point set;

[0221] Select n second feature points from the second feature point set, and select n first feature points from the first feature point set that correspond to the n second feature points respectively, where n is an integer greater than 1;

[0222] Based on the n first feature points and the n second feature points, calculate the rotation matrix and translation vector for transforming from the sub-sensor coordinate system to the main sensor coordinate system, wherein the first calibration parameters include the rotation matrix and translation vector.

[0223] Optionally, the processor 400 is also used to read the program from the memory 420 and perform the following steps:

[0224] Select m second feature points from the second feature point set, wherein the m second feature points are different from the n second feature points, and m is a positive integer;

[0225] The coordinates of the m second feature points are transformed using the first calibration parameters to obtain the coordinates of the m second feature points in the main sensor coordinate system;

[0226] From the first feature point set, determine m first feature points that correspond to the m second feature points respectively;

[0227] Calculate the coordinate deviations between the m second feature points and the m first feature points in the main sensor coordinate system;

[0228] If the coordinate deviation between each second feature point and the corresponding first feature point in the main sensor coordinate system is less than the first preset deviation threshold, the first calibration parameter is determined to be valid.

[0229] Optionally, the processor 400 is also used to read the program from the memory 420 and perform the following steps:

[0230] Select g second feature points from the second feature point set, where g is an integer greater than 1;

[0231] The coordinates of the g second feature points are transformed using the first calibration parameters to obtain the coordinates of the g second feature points in the main sensor coordinate system;

[0232] From the first feature point set, determine g first feature points that correspond to the g second feature points respectively;

[0233] Calculate the coordinate deviations between the g second feature points and the g first feature points in the main sensor coordinate system, and record the number of coordinate deviations greater than or equal to the second preset deviation threshold.

[0234] If the number exceeds a preset value, the detection result is determined to be abnormal.

[0235] Optionally, the processor 400 is also used to read the program from the memory 420 and perform the following steps:

[0236] If the detection result indicates an abnormality, the third feature point set in the main sensor coordinate system and the fourth feature point set in the sub-sensor coordinate system are reacquired.

[0237] The third feature point set is matched with the first feature point set, and the feature points matched between the third feature point set and the first feature point set are recorded as the fifth feature point set.

[0238] The fourth feature point set is matched with the second feature point set, and the feature point set matched by the fourth feature point set and the second feature point set is recorded as the sixth feature point set;

[0239] Based on the corresponding feature points in the fifth feature point set and the sixth feature point set, the second calibration parameters of the sensor group are calculated.

[0240] The joint calibration device provided in this embodiment of the invention can execute the above method embodiment, and its implementation principle and technical effect are similar, so it will not be described again here.

[0241] Furthermore, the computer-readable storage medium of this embodiment of the invention is used to store a computer program, which can be executed by a processor to implement the following steps:

[0242] Obtain the current first calibration parameters of the sensor group, wherein the sensor group includes a main sensor and a sub-sensor;

[0243] According to the preset detection time interval, the first calibration parameter is adaptively detected using feature points in the main sensor coordinate system and feature points in the sub-sensor coordinate system.

[0244] If the detection result indicates an abnormality, the sensor group is recalibrated, and the first calibration parameter is updated based on the recalibrated calibration parameter.

[0245] Optionally, the computer program may be executed by a processor to perform the following steps:

[0246] Using feature points in the main sensor coordinate system and feature points in the sub-sensor coordinate system, the first adaptive state detection is performed on the first calibration parameter after the first calibration parameter is initially calibrated at a first preset time interval;

[0247] If the Kth adaptive state detection is normal, the first calibration parameter is subjected to the (K+1)th adaptive state detection at a second preset time interval, wherein the second preset time is longer than the first preset time, and the second preset time is associated with K, where K is a positive integer.

[0248] Optionally, when K is less than or equal to the first threshold, the second preset duration increases as K increases; when K is greater than the first threshold, the second preset duration remains unchanged from the previous interval duration. The first threshold is an integer greater than 2.

[0249] Optionally, the computer program may be executed by a processor to perform the following steps:

[0250] The sensor group is calibrated to obtain the first calibration parameters;

[0251] The validity of the first calibration parameter is verified;

[0252] If the first calibration parameter is verified to be valid, the first calibration parameter is retained.

[0253] Optionally, the computer program may be executed by a processor to perform the following steps:

[0254] Obtain the first boundary coordinate information of the background marker of the main sensor in the main sensor coordinate system, and select boundary feature points from the first boundary coordinate information to obtain the first feature point set;

[0255] Obtain the second boundary coordinate information of the marker in the sub-sensor coordinate system, and select boundary feature points from the second boundary coordinate information to obtain the second feature point set;

[0256] Select n second feature points from the second feature point set, and select n first feature points from the first feature point set that correspond to the n second feature points respectively, where n is an integer greater than 1;

[0257] Based on the n first feature points and the n second feature points, calculate the rotation matrix and translation vector for transforming from the sub-sensor coordinate system to the main sensor coordinate system, wherein the first calibration parameters include the rotation matrix and translation vector.

[0258] Optionally, the computer program may be executed by a processor to perform the following steps:

[0259] Select m second feature points from the second feature point set, wherein the m second feature points are different from the n second feature points, and m is a positive integer;

[0260] The coordinates of the m second feature points are transformed using the first calibration parameters to obtain the coordinates of the m second feature points in the main sensor coordinate system;

[0261] From the first feature point set, determine m first feature points that correspond to the m second feature points respectively;

[0262] Calculate the coordinate deviations between the m second feature points and the m first feature points in the main sensor coordinate system;

[0263] If the coordinate deviation between each second feature point and the corresponding first feature point in the main sensor coordinate system is less than the first preset deviation threshold, the first calibration parameter is determined to be valid.

[0264] Optionally, the computer program may be executed by a processor to perform the following steps:

[0265] Select g second feature points from the second feature point set, where g is an integer greater than 1;

[0266] The coordinates of the g second feature points are transformed using the first calibration parameters to obtain the coordinates of the g second feature points in the main sensor coordinate system;

[0267] From the first feature point set, determine g first feature points that correspond to the g second feature points respectively;

[0268] Calculate the coordinate deviations between the g second feature points and the g first feature points in the main sensor coordinate system, and record the number of coordinate deviations greater than or equal to the second preset deviation threshold.

[0269] If the number exceeds a preset value, the detection result is determined to be abnormal.

[0270] Optionally, the computer program may be executed by a processor to perform the following steps:

[0271] If the detection result indicates an abnormality, the third feature point set in the main sensor coordinate system and the fourth feature point set in the sub-sensor coordinate system are reacquired.

[0272] The third feature point set is matched with the first feature point set, and the feature points matched between the third feature point set and the first feature point set are recorded as the fifth feature point set.

[0273] The fourth feature point set is matched with the second feature point set, and the feature point set matched by the fourth feature point set and the second feature point set is recorded as the sixth feature point set;

[0274] Based on the corresponding feature points in the fifth feature point set and the sixth feature point set, the second calibration parameters of the sensor group are calculated.

[0275] In the several embodiments provided in this application, it should be understood that the disclosed methods and apparatus can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0276] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can be physically comprised separately, or two or more units can be integrated into one unit. The integrated unit described above can be implemented in hardware or in the form of hardware plus software functional units.

[0277] The integrated units implemented as software functional units described above can be stored in a computer-readable storage medium. These software functional units, stored in a storage medium, include several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute some steps of the transmission and reception methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0278] The above description represents the preferred embodiments of the present invention. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A joint calibration method, characterized in that, include: Obtain the current first calibration parameters of the sensor group, wherein the sensor group includes a main sensor and a sub-sensor; According to the preset detection time interval, the first calibration parameter is adaptively detected using feature points in the main sensor coordinate system and feature points in the sub-sensor coordinate system. If the detection result indicates an abnormality, the sensor group is recalibrated, and the first calibration parameter is updated based on the recalibrated calibration parameter. The step of adaptively detecting the state of the first calibration parameter using feature points in the main sensor coordinate system and feature points in the secondary sensor coordinate system according to a preset detection time interval includes: Using feature points in the main sensor coordinate system and feature points in the sub-sensor coordinate system, the first adaptive state detection is performed on the first calibration parameter after the first calibration parameter is initially calibrated at a first preset time interval; If the Kth adaptive state detection is normal, the first calibration parameter is subjected to the (K+1)th adaptive state detection at a second preset time interval, wherein the second preset time is longer than the first preset time, and the second preset time is associated with K, where K is a positive integer; When K is less than or equal to the first threshold, the second preset duration increases as K increases; when K is greater than the first threshold, the second preset duration remains unchanged from the previous interval duration. The first threshold is an integer greater than 2. The acquisition of the current first calibration parameters of the sensor group includes: The sensor group is calibrated to obtain the first calibration parameters; The validity of the first calibration parameter is verified; If the first calibration parameter is verified to be valid, the first calibration parameter is retained.

2. The method according to claim 1, characterized in that, The calibration of the sensor group to obtain the first calibration parameters includes: Obtain the first boundary coordinate information of the background marker of the main sensor in the main sensor coordinate system, and select boundary feature points from the first boundary coordinate information to obtain the first feature point set; Obtain the second boundary coordinate information of the marker in the sub-sensor coordinate system, and select boundary feature points from the second boundary coordinate information to obtain the second feature point set; Select n second feature points from the second feature point set, and select n first feature points from the first feature point set that correspond to the n second feature points respectively, where n is an integer greater than 1; Based on the n first feature points and the n second feature points, calculate the rotation matrix and translation vector for transforming from the sub-sensor coordinate system to the main sensor coordinate system, wherein the first calibration parameters include the rotation matrix and translation vector.

3. The method according to claim 2, characterized in that, The validity verification of the first calibration parameter includes: Select m second feature points from the second feature point set, wherein the m second feature points are different from the n second feature points, and m is a positive integer; The coordinates of the m second feature points are transformed using the first calibration parameters to obtain the coordinates of the m second feature points in the main sensor coordinate system; From the first feature point set, determine m first feature points that correspond to the m second feature points respectively; Calculate the coordinate deviations between the m second feature points and the m first feature points in the main sensor coordinate system; If the coordinate deviation between each second feature point and the corresponding first feature point in the main sensor coordinate system is less than the first preset deviation threshold, the first calibration parameter is determined to be valid.

4. The method according to claim 2, characterized in that, The adaptive state detection of the first calibration parameter using feature points in the main sensor coordinate system and feature points in the secondary sensor coordinate system includes: Select g second feature points from the second feature point set, where g is an integer greater than 1; The coordinates of the g second feature points are transformed using the first calibration parameters to obtain the coordinates of the g second feature points in the main sensor coordinate system; From the first feature point set, determine g first feature points that correspond to the g second feature points respectively; Calculate the coordinate deviations between the g second feature points and the g first feature points in the main sensor coordinate system, and record the number of coordinate deviations greater than or equal to the second preset deviation threshold. If the number exceeds a preset value, the detection result is determined to be abnormal.

5. The method according to claim 2, characterized in that, The recalibration of the sensor group in the event of an abnormal detection result includes: If the detection result indicates an abnormality, the third feature point set in the main sensor coordinate system and the fourth feature point set in the sub-sensor coordinate system are reacquired. The third feature point set is matched with the first feature point set, and the feature points matched between the third feature point set and the first feature point set are recorded as the fifth feature point set. The fourth feature point set is matched with the second feature point set, and the feature point set matched by the fourth feature point set and the second feature point set is recorded as the sixth feature point set; Based on the corresponding feature points in the fifth feature point set and the sixth feature point set, the second calibration parameters of the sensor group are calculated.

6. A joint calibration device, characterized in that, include: The acquisition module is used to acquire the current first calibration parameters of the sensor group, wherein the sensor group includes a main sensor and a sub-sensor; The detection module is used to adaptively detect the state of the first calibration parameter by using feature points in the main sensor coordinate system and feature points in the sub-sensor coordinate system according to a preset detection time interval. The processing module is used to recalibrate the sensor group when the detection result indicates an abnormality, and update the first calibration parameter based on the recalibrated calibration parameter; The detection module includes: The first detection unit is used to perform the first adaptive state detection of the first calibration parameter after the first calibration parameter is initially calibrated, by using feature points in the main sensor coordinate system and feature points in the sub-sensor coordinate system at a first preset time interval. The second detection unit is used to perform the (K+1)th adaptive state detection on the first calibration parameter at a second preset time interval when the Kth adaptive state detection is normal, wherein the second preset time is longer than the first preset time, and the second preset time is associated with K, where K is a positive integer. When K is less than or equal to the first threshold, the second preset duration increases as K increases; when K is greater than the first threshold, the second preset duration remains unchanged from the previous interval duration. The first threshold is an integer greater than 2. The acquisition module includes: A calibration submodule is used to calibrate the sensor group and obtain the first calibration parameters; The verification submodule is used to verify the validity of the first calibration parameter; The processing submodule is used to retain the first calibration parameter if the first calibration parameter is verified to be valid.

7. A combined calibration device, comprising: A transceiver, a memory, a processor, and a computer program stored in the memory and executable on the processor; characterized in that the processor is configured to read the program in the memory to implement the steps in the joint calibration method as described in any one of claims 1 to 5.

8. A computer-readable storage medium for storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the steps in the joint calibration method as described in any one of claims 1 to 5.

Images