An external parameter monitoring and calibration method and device of a sensor, and a storage medium

Abstract

The application discloses a kind of external parameter monitoring and calibration method and device of sensor, storage medium, the method comprises: monitoring the calculation process of odometer of laser radar-vision SLAM (Simultaneous Localization and Mapping, real-time positioning and map construction) system, after each odometer calculation ends, the current external parameter of current target sensor is acquired;Inquire the historical external parameter obtained after the last odometer calculation ends, determine the external parameter variation range based on the current external parameter and the historical external parameter;If the external parameter variation range does not exceed the preset range threshold, then start calibration system, the external parameter of laser radar-vision SLAM system sensor is coarsely calibrated, and the coarse calibration result is used as the initial laser radar-vision SLAM system sensor calibration external parameter, and fine calibration is carried out.

Description

Technical Field

[0001] This application relates to the field of sensor fusion calibration technology, and in particular to a method and apparatus for monitoring and calibrating the external parameters of a sensor, as well as a storage medium. Background Technology

[0002] Simultaneous localization and mapping (SLAM) estimates pose and maps the surrounding environment while a robot or autonomous vehicle is moving. Currently, the main sensors for perceiving environmental information are cameras and LiDAR. Visual SLAM uses cameras to extract features from the surrounding environment for pose estimation and map building; however, it is prone to failure due to the influence of lighting and weakly textured areas. LiDAR SLAM uses point cloud information collected by 3D LiDAR for feature extraction to estimate pose, but it also fails in open scenes or "long corridor scenes." Therefore, fusing the two sensors for simultaneous localization and mapping is superior to using a single sensor. However, LiDAR-visual SLAM systems cannot identify and correct sensor extrinsic parameters in real time. This leads to the system using incorrect parameters when sensor extrinsic parameters change. Therefore, real-time monitoring and timely correction of the sensor extrinsic parameters in a LiDAR-visual SLAM system are crucial. Summary of the Invention

[0003] In view of this, this application provides a method, apparatus, and storage medium for monitoring and calibrating the external parameters of a sensor. The main purpose is to solve the problem that the LiDAR-Vision SLAM system cannot identify and correct the external parameters of the sensor in real time during operation, which leads to the LiDAR-Vision SLAM system using the original incorrect parameters once the external parameters of the sensor change.

[0004] According to one aspect of this application, a method for monitoring and calibrating the external parameters of a sensor is provided, the method comprising:

[0005] Monitor the odometry calculation process of the LiDAR-Visual SLAM (Simultaneous Localization and Mapping) system, and obtain the current external parameters of the target sensor at the current moment after each odometry calculation.

[0006] Query the historical external parameters obtained after the last odometer calculation, and determine the range of external parameter changes based on the current external parameters and the historical external parameters;

[0007] If the range of change of the external parameters does not exceed the preset range threshold, the calibration system is activated to perform coarse calibration of the external parameters of the LiDAR-Vision SLAM system sensor, and the coarse calibration result is used as the initial calibration external parameters of the LiDAR-Vision SLAM system sensor for fine calibration.

[0008] Optionally, before the odometry calculation process of the monitoring lidar-visual SLAM (Simultaneous Localization and Mapping) system, the method further includes:

[0009] The lidar-visual SLAM system is initialized, and the initial external parameters of the target sensor are obtained. It is then determined whether the initial external parameters are consistent with the default parameters.

[0010] If the initial external parameters are consistent with the default parameters, then monitor the odometry calculation process of the LiDAR-Visual SLAM system.

[0011] Optionally, obtaining the current external parameters of the target sensor at the current moment includes:

[0012] The attitude information output during the operation of the LiDAR-Vision SLAM system is acquired, and the current external parameters are estimated based on the attitude information.

[0013] Optionally, after determining the range of change of the external parameters based on the current external parameters and the historical external parameters, the method further includes:

[0014] Determine whether the range of change of the external parameter exceeds a preset threshold.

[0015] If the range of change of the external parameters exceeds the preset range threshold, a stop operation command is sent to the lidar-visual SLAM system.

[0016] If the range of change of the external parameter does not exceed the preset range threshold, then the similarity value between the current external parameter and the initial external parameter is calculated.

[0017] Determine whether the similarity value is less than a preset threshold;

[0018] If the similarity value is less than the preset threshold, the calibration system is activated to coarsely calibrate the external parameters of the LiDAR-Visual SLAM system sensor.

[0019] Otherwise, the external parameters of the LiDAR-visual SLAM system are assumed to be in normal condition.

[0020] Optionally, activating the calibration system to coarsely calibrate the external parameters of the LiDAR-visual SLAM system sensors includes:

[0021] When the odometer of the LiDAR-Vision SLAM system can output information normally, the hand-eye calibration algorithm is used to coarsely calibrate the external parameters of the LiDAR-Vision SLAM system sensor. When any part of the information in the odometer of the LiDAR-Vision SLAM system fails, the mutual information algorithm is used to coarsely calibrate and obtain the coarse calibration parameters.

[0022] The coarse calibration parameters are output to the LiDAR-Vision SLAM system, and the LiDAR-Vision SLAM system is maintained in operation according to the coarse calibration parameters.

[0023] Optionally, the step of using the coarse calibration result as the initial calibration extrinsic parameters of the LiDAR-visual SLAM system sensor for fine calibration includes:

[0024] The grayscale image and laser point cloud intensity map of the camera image at the target time are acquired. Based on the calibration external parameters of the initial lidar-visual SLAM system sensor, the laser point cloud intensity map is projected onto the pixel plane to obtain the point cloud intensity map projection image.

[0025] Histogram equalization is performed on the grayscale images of the point cloud intensity map projection and the camera image.

[0026] The camera image grayscale image is extracted using a pre-constructed edge detector to obtain the camera image edges, and the camera image edges with an edge length less than a preset threshold are removed.

[0027] The pre-constructed edge detector is used to extract edges from the projection image of the point cloud intensity map to obtain the edge of the point cloud image, and the edge of the point cloud image with an edge length less than a preset threshold is removed.

[0028] A loss function is established based on the camera image edges and the point cloud image edges;

[0029] The external parameter values ​​in the loss function are iteratively optimized by setting the number of iterations and the optimal threshold of the loss function to obtain the finely calibrated parameters.

[0030] The fine calibration parameters are output to the LiDAR-Vision SLAM system, and the LiDAR-Vision SLAM system is maintained in operation according to the fine calibration parameters.

[0031] Optionally, before querying the historical external parameters obtained after the last odometer calculation, the method further includes:

[0032] After each mileage calculation, the current external parameters are stored as historical external parameters.

[0033] According to another aspect of this application, a sensor external parameter monitoring and calibration device is provided, the device comprising:

[0034] The first monitoring module is used to monitor the odometry calculation process of the LiDAR-Vision SLAM (Simultaneous Localization and Mapping) system. After each odometry calculation, it obtains the current external parameters of the target sensor at the current moment.

[0035] The query module is used to query the historical external parameters obtained after the last odometer calculation, and to determine the range of external parameter changes based on the current external parameters and the historical external parameters.

[0036] The startup module is used to start the calibration system if the range of change of the external parameters does not exceed the preset range threshold, to perform coarse calibration of the external parameters of the LiDAR-Vision SLAM system sensor, and to use the coarse calibration result as the initial calibration external parameters of the LiDAR-Vision SLAM system sensor for fine calibration.

[0037] Optionally, the device further includes:

[0038] The acquisition module is used to initialize the lidar-visual SLAM system, acquire the initial external parameters of the target sensor, and identify whether the initial external parameters are consistent with the default parameters.

[0039] The second monitoring module is used to monitor the odometry calculation process of the lidar-visual SLAM system if the initial external parameters are consistent with the default parameters.

[0040] Optionally, the first monitoring module is further configured to:

[0041] The attitude information output during the operation of the LiDAR-Vision SLAM system is acquired, and the current external parameters are estimated based on the attitude information.

[0042] Optionally, the device further includes:

[0043] The first judgment module is used to determine whether the range of change of the external parameter exceeds a preset range threshold.

[0044] The sending module is used to send a stop working command to the lidar-visual SLAM system if the range of change of the external parameters exceeds the preset range threshold.

[0045] The calculation module is used to calculate the similarity value between the current external parameter and the initial external parameter if the range of change of the external parameter does not exceed the preset range threshold.

[0046] The second judgment module is used to determine whether the similarity value is less than a preset threshold.

[0047] A coarse calibration module is used to activate the calibration system to coarsely calibrate the external parameters of the LiDAR-Visual SLAM system sensor if the similarity value is less than the preset threshold.

[0048] The default module is used otherwise, by default the external parameters of the LiDAR-Visual SLAM system sensors are in a normal state.

[0049] Optionally, the coarse calibration module is further used for:

[0050] When the odometer of the LiDAR-Vision SLAM system can output information normally, the hand-eye calibration algorithm is used to coarsely calibrate the external parameters of the LiDAR-Vision SLAM system sensor. When any part of the information in the odometer of the LiDAR-Vision SLAM system fails, the mutual information algorithm is used to coarsely calibrate and obtain the coarse calibration parameters.

[0051] The coarse calibration parameters are output to the LiDAR-Vision SLAM system, and the LiDAR-Vision SLAM system is maintained in operation according to the coarse calibration parameters.

[0052] Optionally, the startup module is further configured to:

[0053] The grayscale image and laser point cloud intensity map of the camera image at the target time are acquired. Based on the calibration external parameters of the initial lidar-visual SLAM system sensor, the laser point cloud intensity map is projected onto the pixel plane to obtain the point cloud intensity map projection image.

[0054] Histogram equalization is performed on the grayscale images of the point cloud intensity map projection and the camera image.

[0055] The camera image grayscale image is extracted using a pre-constructed edge detector to obtain the camera image edges, and the camera image edges with an edge length less than a preset threshold are removed.

[0056] The pre-constructed edge detector is used to extract edges from the projection image of the point cloud intensity map to obtain the edge of the point cloud image, and the edge of the point cloud image with an edge length less than a preset threshold is removed.

[0057] A loss function is established based on the camera image edges and the point cloud image edges;

[0058] The external parameter values ​​in the loss function are iteratively optimized by setting the number of iterations and the optimal threshold of the loss function to obtain the finely calibrated parameters.

[0059] The fine calibration parameters are output to the LiDAR-Vision SLAM system, and the LiDAR-Vision SLAM system is maintained in operation according to the fine calibration parameters.

[0060] According to another aspect of this application, a storage medium is provided that stores a computer program thereon, which, when executed by a processor, implements the above-described method for monitoring and calibrating the external parameters of the sensor.

[0061] According to another aspect of this application, a computer device is provided, including a storage medium, a processor, and a computer program stored on the storage medium and executable on the processor, wherein the processor executes the program to implement the method for monitoring and calibrating the external parameters of the aforementioned sensor.

[0062] By employing the above technical solution, this application provides a method, apparatus, and storage medium for monitoring and calibrating the external parameters of a sensor. First, it monitors the odometry calculation process of a LiDAR-Vision SLAM (Simultaneous Localization and Mapping) system. After each odometry calculation, it acquires the current external parameters of the target sensor. Second, it queries the historical external parameters acquired after the last odometry calculation. Based on the current and historical external parameters, it determines the range of external parameter variation. Finally, if the range of external parameter variation does not exceed a preset threshold, it initiates a calibration system to coarsely calibrate the external parameters of the LiDAR-Vision SLAM system sensor. The coarse calibration result is then used as the initial calibration external parameters for the LiDAR-Vision SLAM system sensor for fine calibration. This avoids the problem that the LiDAR-Vision SLAM system cannot perform real-time identification and correction of the sensor's external parameters during operation, which would cause the LiDAR-Vision SLAM system to operate with previously incorrect parameters once the sensor's external parameters change.

[0063] The above description is only an overview of the technical solution of this application. In order to better understand the technical means of this application and to implement it in accordance with the contents of the specification, and to make the above and other objects, features and advantages of this application more obvious and understandable, the following are specific embodiments of this application. Attached Figure Description

[0064] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0065] Figure 1 A flowchart illustrating a method for monitoring and calibrating the external parameters of a sensor according to an embodiment of this application is shown.

[0066] Figure 2 A flowchart illustrating another method for monitoring and calibrating the external parameters of a sensor provided in an embodiment of this application is shown.

[0067] Figure 3 This paper shows a schematic diagram of the structure of a sensor external parameter monitoring and calibration device provided in an embodiment of this application;

[0068] Figure 4 This paper illustrates a schematic diagram of the coarse calibration process for another sensor external parameter monitoring and calibration method provided in an embodiment of this application.

[0069] Figure 5 This paper illustrates a schematic diagram of the feature extraction and matching process after radar point cloud projection for another sensor external parameter monitoring and calibration method provided in an embodiment of this application.

[0070] Figure 6 This paper illustrates a fine calibration process diagram of another sensor external parameter monitoring and calibration method provided in an embodiment of this application;

[0071] Figure 7 A schematic diagram of the device structure of a computer device provided in an embodiment of this application is shown. Detailed Implementation

[0072] The present application will be described in detail below with reference to the accompanying drawings and embodiments. It should be noted that, unless otherwise specified, the embodiments and features described in the embodiments of the present application can be combined with each other.

[0073] This embodiment provides a method for monitoring and calibrating the external parameters of a sensor, such as... Figure 1 As shown, the method includes:

[0074] Step 101: Monitor the odometry calculation process of the LiDAR-Vision SLAM (Simultaneous Localization and Mapping) system. After each odometry calculation, obtain the current external parameters of the target sensor at the current moment.

[0075] Step 102: Query the historical external parameters obtained after the last odometer calculation, and determine the range of external parameter changes based on the current external parameters and the historical external parameters.

[0076] Step 103: If the range of change of the external parameters does not exceed the preset range threshold, the calibration system is started to perform coarse calibration of the external parameters of the LiDAR-Vision SLAM system sensor, and the coarse calibration result is used as the initial calibration external parameters of the LiDAR-Vision SLAM system sensor for fine calibration.

[0077] In this embodiment, firstly, the odometry calculation process of the LiDAR-Visual SLAM system is monitored. After each odometry calculation, the monitoring system runs once to monitor changes in the external parameters of the target sensor. By acquiring the attitude information output by the LiDAR-Visual SLAM system during operation, the current external parameters are estimated using the hand-eye calibration method. Based on the current external parameters and the historical external parameters estimated after the last odometry calculation, and according to the attitude information output, the range of external parameter variation is determined.

[0078] Secondly, it is determined whether the range of external parameter changes exceeds the preset range threshold. If the range of external parameter changes does not exceed the preset range threshold, the calibration system is started. The calibration system is used to quickly perform a coarse calibration of the external parameters of the LiDAR-Vision SLAM system sensor to obtain coarse calibration parameters. The coarse calibration parameters are used to maintain the operation of the LiDAR-Vision SLAM system. Finally, the coarse calibration parameters are used as the initial external parameters for fine calibration of the external parameters of the LiDAR-Vision SLAM system sensor to obtain fine calibration parameters.

[0079] By applying the technical solution of this embodiment, after each odometer calculation, the monitoring system runs once to obtain the current external parameters of the target sensor at the current moment. Based on the current external parameters obtained each time and the historical external parameters obtained after the last odometer calculation, the range of external parameter variation is determined, and it is judged whether the range of external parameter variation exceeds a preset threshold. If the range of external parameter variation does not exceed the preset threshold, the calibration system quickly performs a coarse calibration of the external parameters of the LiDAR-Vision SLAM system sensor. The obtained coarse calibration parameters are used to maintain the operation of the LiDAR-Vision SLAM system, and the coarse calibration parameters are used as the initial calibration external parameters of the LiDAR-Vision SLAM system sensor for fine calibration. This avoids the problem that the LiDAR-Vision SLAM system cannot perform real-time identification and correction of the sensor's external parameters during operation, which would cause the LiDAR-Vision SLAM system to use the previously incorrect parameters once the sensor's external parameters change.

[0080] Furthermore, as a refinement and extension of the specific implementation of the above embodiments, and to fully illustrate the specific implementation process of this embodiment, another method for monitoring and calibrating the external parameters of a sensor is provided, such as... Figure 2 As shown, the method includes:

[0081] Step 201: Initialize the LiDAR-Vision SLAM system and obtain the initial external parameters of the target sensor, and identify whether the initial external parameters are consistent with the default parameters.

[0082] Step 202: If the initial external parameters are consistent with the default parameters, monitor the odometry calculation process of the LiDAR-Vision SLAM system.

[0083] Step 203: Monitor the odometry calculation process of the LiDAR-Vision SLAM (Simultaneous Localization and Mapping) system. After each odometry calculation, obtain the current external parameters of the target sensor at the current moment.

[0084] First, the LiDAR-Vision SLAM system is initialized, and the external parameters of the LiDAR-Vision SLAM system sensor are monitored once using the external parameter real-time monitoring module to obtain the initial external parameters of the target sensor. It is then determined whether the initial external parameters are consistent with the default parameters. If the initial external parameters are consistent with the default parameters, the odometry calculation process of the LiDAR-Vision SLAM system is monitored. After each odometry calculation, the monitoring system runs once to monitor the changes in the external parameters of the target sensor. By acquiring the attitude information output by the LiDAR-Vision SLAM system during operation, the current external parameters are estimated using the hand-eye calibration method based on the attitude information.

[0085] Step 204: Store the current external parameters obtained after each mileage calculation as the historical external parameters.

[0086] Step 205: Query the historical external parameters obtained after the last odometer calculation, and determine the range of external parameter changes based on the current external parameters and the historical external parameters.

[0087] Next, after each odometer calculation, the current external parameters estimated by acquiring the attitude information output during the operation of the LiDAR-Vision SLAM system are stored as historical external parameters. Based on the current external parameters and the historical external parameters estimated after the last odometer calculation, the range of external parameter variation is determined.

[0088] Step 206: Determine whether the range of change of the external parameter exceeds a preset range threshold.

[0089] Step 207: If the range of change of the external parameters exceeds the preset range threshold, a stop operation command is sent to the lidar-visual SLAM system.

[0090] Next, by comparing the range of external parameter changes with a preset range threshold, it is determined whether the range of external parameter changes exceeds the preset range threshold. If the range of external parameter changes exceeds the preset range threshold, causing the LiDAR-Vision SLAM system to malfunction, a stop operation command is sent to the LiDAR-Vision SLAM system.

[0091] Step 208: If the range of change of the external parameters does not exceed the preset range threshold, then start the calibration system to perform coarse calibration on the external parameters of the LiDAR-Vision SLAM system sensor.

[0092] Next, by comparing the range of external parameter changes with a preset threshold, if the range of external parameter changes does not exceed the preset threshold, the similarity value between the current external parameter and the initial external parameter is calculated using a similarity calculation formula. This similarity calculation formula can simultaneously consider the similarity of rotation and translation, as shown in Formula 1:

[0093]

[0094] Where rc and ri represent the rotation vectors of the current external parameters and the initial external parameters, respectively, and t c and ti represent the translation vectors of the current external parameters and the initial external parameters, respectively; α and β are weight coefficients; and s is the similarity value between the current external parameters and the initial external parameters.

[0095] If the similarity value is less than a preset threshold, the calibration system is activated to coarsely calibrate the external parameters of the LiDAR-visual SLAM system, such as... Figure 4 As shown, at the same time, when the odometer of the LiDAR-Vision SLAM system can output information normally, the hand-eye calibration algorithm is used to coarsely calibrate the external parameters of the LiDAR-Vision SLAM system sensor. When any part of the information in the odometer of the LiDAR-Vision SLAM system fails, the mutual information algorithm is used to coarsely calibrate, and the obtained coarse calibration parameters are used to maintain the operation of the LiDAR-Vision SLAM system.

[0096] If the similarity value corresponding to the current external parameter is greater than or equal to the preset threshold, the external parameters of the LiDAR-Visual SLAM system sensor are in normal condition by default.

[0097] Step 209: Use the coarse calibration results as the initial calibration external parameters of the LiDAR-Vision SLAM system sensor for fine calibration.

[0098] It should be noted that using the obtained coarse calibration parameters to maintain the operation of the LiDAR-Vision SLAM system has low accuracy and therefore cannot be used for extended periods. Therefore, after coarse calibration, the calibration results need to be further refined, i.e., fine calibration. Since coarse calibration provides a short-term usable result, fine calibration does not need to meet real-time requirements and can run independently in a thread parallel to the LiDAR-Vision SLAM system. Fine calibration methods typically require extracting correlated features, such as lines, planes, and edges, from both the image and the LiDAR point cloud to optimize the calculation of the sensor's extrinsic parameters. However, such methods rely on good initial values ​​for the optimization algorithm to converge to the global optimum; therefore, the coarse calibration results can be used to provide these initial values.

[0099] Next, as Figure 5 As shown, this method projects the LiDAR point cloud onto a 2D image, applies histogram equalization to all images to enhance contrast, and uses the Canny edge detector to extract edges. Edges extracted from the depth and reflectance images are merged, resulting in a set of characteristic edges present in both the camera and LiDAR images, which are then matched based on the nearest edge distance.

[0100] After pairing radar edge line features and camera edge line features, shortest distance matching optimization is performed based on the points on the radar edge line features and the points on the camera image features, as shown in Equation 2. Here, n is the number of camera edge points within the distance threshold of the LiDAR edge point, m is the number of nearest neighbor edge line feature endpoints, N is the total number of camera points, and b is the penalty factor. When the edge line feature matching is incorrect, the loss function CF value will be very large. Experiments show that a value of b of 10 is a good candidate value and can be used as the default value. Where P... ik lidar This represents the radar projection points obtained from the point cloud after extrinsic parameter transformation. The minimum value of the loss function is found through iterative optimization using the gradient descent method. The extrinsic parameters obtained when the loss function is minimized are... This is the extrinsic parameter matrix of the camera radar.

[0101]

[0102] Edge alignment matching is performed according to the fine-matching design process. The loss function is established below, as shown in formulas 3 and 4:

[0103]

[0104]

[0105] Where n is the number of camera edge points within the distance threshold of the LiDAR edge points, m is the number of nearest neighbors matched, N is the total number of camera points, and δ is the penalty factor. Due to the penalty term, the cost of this loss function will be very large when the edge line feature matching is incorrect. Where P... j lidar The point cloud is transformed by extrinsic parameters to obtain radar projection points. The minimum cost is found by iterative optimization using gradient descent. The extrinsic parameter T obtained when the loss function is minimized is the extrinsic parameter matrix of the camera radar.

[0106] like Figure 6 As shown, a fine calibration system independent of the LiDAR-visual SLAM system was designed. This system extracts relevant features from images and point clouds and optimizes extrinsic parameters to maximize the matching degree of relevant features, thereby obtaining the optimal extrinsic parameters.

[0107] The algorithm flow for fine calibration is as follows:

[0108] The grayscale image and laser point cloud intensity map of the camera image at the target time are acquired. Based on the initial calibration external parameters of the LiDAR-Vision SLAM system sensor, the laser point cloud intensity map is projected onto the pixel plane to obtain the point cloud intensity map projection image.

[0109] Histogram equalization is performed on the grayscale images of the point cloud intensity map projection and the camera image.

[0110] The edge of the camera image is extracted by using a pre-built edge detector, and the edge of the camera image is removed if the edge length is less than a preset threshold.

[0111] Edges are extracted from the point cloud intensity map projection image using a pre-built edge detector to obtain the edge of the point cloud image, and edge removal is performed on the point cloud image edge whose edge length is less than a preset threshold.

[0112] A loss function is established based on the camera image edges and the point cloud image edges;

[0113] The external parameter values ​​in the loss function are iteratively optimized by setting the number of iterations and the optimal threshold of the loss function.

[0114] The above loss function is iteratively solved to find the optimal solution. First-order gradient descent is used to perform a first-order Taylor expansion on the objective function. The iteration continues in the direction of gradient descent of the objective function by solving for the first derivative until the calibration convergence threshold or the number of convergences is reached, at which point the iteration stops. The external parameters at this point are the results of the joint online fine calibration of the external parameters of the camera and LiDAR sensor. The fine calibration parameters are output to the LiDAR-Vision SLAM system, and the LiDAR-Vision SLAM system is maintained according to the fine calibration parameters.

[0115] By applying the technical solution of this embodiment, the LiDAR-Visual SLAM system is initialized, and the external parameters of the LiDAR-Visual SLAM system sensor are monitored once using the external parameter real-time monitoring module to obtain the initial external parameters of the target sensor. It is then determined whether the initial external parameters are consistent with the default parameters. If the initial external parameters are consistent with the default parameters, the odometry calculation process of the LiDAR-Visual SLAM system is monitored. After each odometry calculation, the monitoring system runs once to obtain the current external parameters of the target sensor at the current moment. Based on the current external parameters obtained each time and the parameters obtained after the previous odometry calculation... Historical external parameters are used to determine the range of external parameter changes. It is then determined whether the range exceeds a preset threshold. If the range does not exceed the threshold, a rapid coarse calibration of the LiDAR-Vision SLAM system's external parameters is performed. These coarse calibration parameters are used to maintain the LiDAR-Vision SLAM system's operation. These coarse calibration parameters are then used as the initial calibration external parameters for the LiDAR-Vision SLAM system's sensors for fine calibration. The fine calibration parameters are then output to the LiDAR-Vision SLAM system, and its operation is maintained based on these parameters. This avoids the problem of the LiDAR-Vision SLAM system failing to identify and correct sensor external parameters in real time during operation, which could lead to the system using incorrect parameters if sensor external parameters change.

[0116] Furthermore, as Figure 1 In terms of specific implementation, this application provides a device for monitoring and calibrating the external parameters of a sensor, such as... Figure 3 As shown, the device includes:

[0117] The first monitoring module is used to monitor the odometry calculation process of the LiDAR-Vision SLAM (Simultaneous Localization and Mapping) system. After each odometry calculation, it obtains the current external parameters of the target sensor at the current moment.

[0118] The query module is used to query the historical external parameters obtained after the last odometer calculation, and to determine the range of external parameter changes based on the current external parameters and the historical external parameters.

[0119] The startup module is used to start the calibration system if the range of change of the external parameters does not exceed the preset range threshold, to perform coarse calibration of the external parameters of the LiDAR-Vision SLAM system sensor, and to use the coarse calibration result as the initial calibration external parameters of the LiDAR-Vision SLAM system sensor for fine calibration.

[0120] Optionally, the device further includes:

[0121] The acquisition module is used to initialize the lidar-visual SLAM system, acquire the initial external parameters of the target sensor, and identify whether the initial external parameters are consistent with the default parameters.

[0122] The second monitoring module is used to monitor the odometry calculation process of the lidar-visual SLAM system if the initial external parameters are consistent with the default parameters.

[0123] Optionally, the first monitoring module is further configured to:

[0124] The attitude information output during the operation of the LiDAR-Vision SLAM system is acquired, and the current external parameters are estimated based on the attitude information.

[0125] Optionally, the device further includes:

[0126] The first judgment module is used to determine whether the range of change of the external parameter exceeds a preset range threshold.

[0127] The sending module is used to send a stop working command to the lidar-visual SLAM system if the range of change of the external parameters exceeds the preset range threshold.

[0128] The calculation module is used to calculate the similarity value between the current external parameter and the initial external parameter if the range of change of the external parameter does not exceed the preset range threshold.

[0129] The second judgment module is used to determine whether the similarity value is less than a preset threshold.

[0130] A coarse calibration module is used to activate the calibration system to coarsely calibrate the external parameters of the LiDAR-Visual SLAM system sensor if the similarity value is less than the preset threshold.

[0131] The default module is used otherwise, by default the external parameters of the LiDAR-visual SLAM system are in a normal state.

[0132] Optionally, the coarse calibration module is further used for:

[0133] When the odometer of the LiDAR-Vision SLAM system can output information normally, the hand-eye calibration algorithm is used to coarsely calibrate the external parameters of the LiDAR-Vision SLAM system sensor. When any part of the information in the odometer of the LiDAR-Vision SLAM system fails, the mutual information algorithm is used to coarsely calibrate and obtain the coarse calibration parameters.

[0134] The coarse calibration parameters are output to the LiDAR-Vision SLAM system, and the LiDAR-Vision SLAM system is maintained in operation according to the coarse calibration parameters.

[0135] Optionally, the startup module is further configured to:

[0136] The grayscale image and laser point cloud intensity map of the camera image at the target time are acquired. Based on the calibration external parameters of the initial lidar-visual SLAM system sensor, the laser point cloud intensity map is projected onto the pixel plane to obtain the point cloud intensity map projection image.

[0137] Histogram equalization is performed on the grayscale images of the point cloud intensity map projection and the camera image.

[0138] The camera image grayscale image is extracted using a pre-constructed edge detector to obtain the camera image edges, and the camera image edges with an edge length less than a preset threshold are removed.

[0139] The pre-constructed edge detector is used to extract edges from the projection image of the point cloud intensity map to obtain the edge of the point cloud image, and the edge of the point cloud image with an edge length less than a preset threshold is removed.

[0140] A loss function is established based on the camera image edges and the point cloud image edges;

[0141] The external parameter values ​​in the loss function are iteratively optimized by setting the number of iterations and the optimal threshold of the loss function to obtain the finely calibrated parameters.

[0142] The fine calibration parameters are output to the LiDAR-Vision SLAM system, and the LiDAR-Vision SLAM system is maintained in operation according to the fine calibration parameters.

[0143] It should be noted that other corresponding descriptions of the functional units involved in the sensor external parameter monitoring and calibration device provided in this application embodiment can be found in the following references. Figures 1 to 2 The corresponding descriptions in the method will not be repeated here.

[0144] This application also provides a computer device, which may specifically be a personal computer, a server, a network device, etc. Figure 7 As shown, the computer device includes a bus, a processor, memory, and a communication interface, and may also include an input / output interface and a display device. The processor provides computing and control capabilities. The memory includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores an operating system, computer programs, and a database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. The database stores location information. The network interface allows communication with external terminals via a network connection. When the computer program is executed by the processor, it implements the steps in the various method embodiments.

[0145] Those skilled in the art will understand that Figure 7 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0146] In one embodiment, a computer-readable storage medium is provided, which may be non-volatile or volatile, having stored thereon a computer program that, when executed by a processor, implements the steps in the above method embodiments.

[0147] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above method embodiments.

[0148] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties.

[0149] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0150] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0151] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A method for monitoring and calibrating external parameters of a sensor, characterized in that, The method comprises the following steps: monitoring the calculation process of the odometer of the laser radar-vision SLAM system, obtaining the current external parameter of the target sensor at the current time after each odometer calculation is completed; querying the historical external parameter obtained after the last odometer calculation is completed, determining the external parameter change range based on the current external parameter and the historical external parameter; judging whether the external parameter change range exceeds the preset range threshold; if the external parameter change range exceeds the preset range threshold, sending a stop working instruction to the laser radar-vision SLAM system; if the external parameter change range does not exceed the preset range threshold, calculating the similarity value of the current external parameter and the initial external parameter; judging whether the similarity value is less than the preset threshold; if the similarity value is less than the preset threshold, starting the calibration system, performing coarse calibration on the external parameter of the laser radar-vision SLAM system sensor, and taking the coarse calibration result as the initial calibration external parameter of the laser radar-vision SLAM system sensor for fine calibration; otherwise, defaulting the external parameter of the laser radar-vision SLAM system sensor as normal.

2. The method of claim 1, wherein, Before the step of monitoring the calculation process of the odometer of the laser radar-vision SLAM system, the method further comprises the following steps: initializing the laser radar-vision SLAM system, obtaining the initial external parameter of the target sensor, and identifying whether the initial external parameter is consistent with the default parameter; if the initial external parameter is consistent with the default parameter, monitoring the calculation process of the odometer of the laser radar-vision SLAM system.

3. The method of claim 1, wherein, The step of obtaining the current external parameter of the target sensor at the current time comprises the following steps: obtaining the attitude information output during the running process of the laser radar-vision SLAM system, and estimating the current external parameter according to the attitude information.

4. The method of claim 1, wherein, The step of starting the calibration system to perform coarse calibration on the external parameter of the laser radar-vision SLAM system sensor comprises the following steps: when the odometer of the laser radar-vision SLAM system can normally output information, using a hand-eye calibration algorithm to perform coarse calibration on the external parameter of the laser radar-vision SLAM system sensor, and when any part of the odometer of the laser radar-vision SLAM system is invalid, using a mutual information algorithm to perform coarse calibration to obtain the coarse calibration parameter; outputting the coarse calibration parameter to the laser radar-vision SLAM system, and maintaining the running of the laser radar-vision SLAM system according to the coarse calibration parameter.

5. The method of claim 1, wherein, The step of taking the coarse calibration result as the initial calibration external parameter of the laser radar-vision SLAM system sensor for fine calibration comprises the following steps: obtaining the gray image of the camera image and the laser point cloud intensity image at the target time, projecting the laser point cloud intensity image to the pixel plane according to the initial calibration external parameter of the laser radar-vision SLAM system sensor to obtain a point cloud intensity image projection image; performing histogram equalization processing on the point cloud intensity image projection image and the gray image of the camera image; edge extraction is performed on the gray image of the camera image by using a pre-constructed edge detector to obtain camera image edges, and edge elimination is performed on the camera image edges with edge lengths less than a preset threshold value; edge extraction is performed on the point cloud intensity image by using the pre-constructed edge detector to obtain point cloud image edges, and edge elimination is performed on the point cloud image edges with edge lengths less than a preset threshold value; a loss function is established according to the camera image edges and the point cloud image edges; iterative optimization is performed on the external parameter value in the loss function, and optimization is completed by setting the number of iterations and a loss function optimization threshold value to obtain fine calibration parameters; the fine calibration parameters are output to the laser radar-vision SLAM system, and the laser radar-vision SLAM system is maintained to operate according to the fine calibration parameters.

6. The method of claim 1, wherein, Before the historical external parameters obtained after the last odometer calculation, the method further comprises: The current external parameters obtained after each odometer calculation are stored as the historical external parameters.

7. An apparatus for monitoring and calibrating external parameters of a sensor, characterized in that The device comprises: A first monitoring module for monitoring the calculation process of the odometer of the laser radar-vision SLAM system, and obtaining the current external parameters of the target sensor at the current time after each odometer calculation; A query module for querying the historical external parameters obtained after the last odometer calculation, and determining the external parameter change range based on the current external parameters and the historical external parameters; A first judgment module for judging whether the external parameter change range exceeds a preset range threshold value; A sending module for sending a stop working instruction to the laser radar-vision SLAM system if the external parameter change range exceeds the preset range threshold value; A calculation module for calculating the similarity value of the current external parameters and the initial external parameters if the external parameter change range does not exceed the preset range threshold value; A second judgment module for judging whether the similarity value is less than a preset threshold value; A coarse calibration module for starting a calibration system to perform coarse calibration on the external parameters of the laser radar-vision SLAM system sensor and taking the coarse calibration result as the initial calibration external parameters of the laser radar-vision SLAM system sensor for fine calibration if the similarity value is less than the preset threshold value; A default module for otherwise defaulting the external parameters of the laser radar-vision SLAM system to be in a normal state.

8. A storage medium having stored thereon a computer program, characterized in that The computer program is executed by the processor to implement the method of any one of claims 1 to 6.

9. A computer device comprising a storage medium, a processor, and a computer program stored on the storage medium and executable on the processor, characterized in that, The processor executes the computer program to implement the method of any one of claims 1 to 6.

Images