Radar point cloud fusion method, radar sensor calibration method and related equipment

Abstract

The present application relates to a radar point cloud fusion method, a radar sensor calibration method, and related equipment. The method includes: collecting image point clouds based on multiple calibrated image sensors, mapping the image point clouds to a preset coordinate system; generating a grid map within the preset coordinate system based on the image point clouds, and determining the grid state of each grid in the grid map based on the state of each point in the image point cloud occupying each grid; collecting radar point clouds based on an uncalibrated radar sensor, mapping the radar point cloud to the preset coordinate system according to the initial design position of the radar sensor in the preset coordinate system, and obtaining the grid state value of the grid occupied by each point in the radar point cloud; adjusting the position of the radar sensor within the preset coordinate system based on the grid state value of the grid occupied by each point in the radar point cloud, and obtaining the actual installation position of the radar sensor within the preset coordinate system. The present invention improves the calibration efficiency of vehicle-mounted radar sensors.

Description

Technical Field

[0001] The present application relates to the field of machine vision technology, and in particular to a radar point cloud fusion method, a radar sensor calibration method, and related equipment. Background Art

[0002] The development of intelligent driving and intelligent transportation technologies requires the integration and analysis of various sensor data to accurately perceive environmental information. Single sensor data often has limitations. For example, camera sensors can directly capture visual information but cannot perceive depth. Laser sensors, while capable of precise distance measurement, lack rich environmental color data. Fusion of various sensor data can overcome the shortcomings of single sensors.

[0003] The data from various sensors can be integrated into a unified coordinate system to achieve real-time fusion of the data collected by the sensors.

[0004] Since different vehicles cannot guarantee that the radar installation posture is completely consistent, every vehicle-mounted radar leaving the factory requires manual calibration, which is inefficient. Summary of the Invention

[0005] The radar point cloud fusion method, radar sensor calibration method, and related equipment provided in this embodiment at least solve the problem of low calibration efficiency of vehicle-mounted radar sensors in related technologies.

[0006] A radar sensor calibration method, comprising:

[0007] Acquire image point clouds based on the calibrated multiple image sensors, and map the image point clouds to a preset coordinate system;

[0008] generating a grid map in the preset coordinate system based on the image point cloud, and determining a grid state of each grid in the grid map according to a state in which each point in the image point cloud occupies each grid;

[0009] collecting a radar point cloud based on an uncalibrated radar sensor, mapping the radar point cloud to the preset coordinate system according to an initial design pose of the radar sensor in the preset coordinate system, and obtaining a value of a grid state of a grid occupied by each point in the radar point cloud;

[0010] According to the grid state value of the grid occupied by each point in the radar point cloud, the posture of the radar sensor is adjusted in the preset coordinate system to obtain the actual installation posture of the radar sensor in the preset coordinate system.

[0011] In some embodiments, the image point cloud and the radar point cloud are both three-dimensional point clouds, the preset coordinate system is a three-dimensional spatial coordinate system, and the grid map is a three-dimensional grid map; obtaining the grid state value of the grid occupied by each point in the radar point cloud includes:

[0012] Determining a value of a grid state at any point in the grid map by performing trilinear interpolation on the grid map;

[0013] According to the position of each point in the radar point cloud in the grid map, a value of the grid state of each point in the radar point cloud is determined.

[0014] In some embodiments, adjusting the position and posture of the radar sensor in the preset coordinate system according to a grid state value of a grid occupied by each point in the radar point cloud includes:

[0015] determining a total value of the grid state of the radar point cloud according to the grid state value of each point in the radar point cloud;

[0016] determining a gradient of the radar point cloud according to a value of a grid state of each point in the radar point cloud;

[0017] In the preset coordinate system, the pose of the radar point cloud is adjusted along the gradient direction of the radar point cloud so that the total value of the grid state of the radar point cloud converges.

[0018] In some embodiments, adjusting the pose of the radar point cloud along the gradient direction of the radar point cloud in the preset coordinate system so that the total value of the grid state of the radar point cloud converges includes:

[0019] Determining whether a change in a total value of the radar point cloud before and after adjusting the posture of the radar point cloud is no greater than a preset threshold;

[0020] When a change in the total value of the radar point cloud before and after the posture of the radar point cloud is adjusted is not greater than the preset threshold, it is determined that the total value of the grid state of the radar point cloud has converged.

[0021] In some embodiments, the method further comprises:

[0022] The size of the grid in the grid map is gradually reduced to calibrate the radar sensor with gradually increasing accuracy until the size of the grid in the grid map reaches a set size, and then the actual installation posture of the radar sensor in the preset coordinate system is obtained.

[0023] In some embodiments, there are multiple radar sensors; and the method further includes:

[0024] The distribution differences of the posture distribution forms of the plurality of radar sensors in the initial design posture and the actual installation posture are compared, and the optimized postures of the plurality of radar sensors are determined with the goal of minimizing the distribution differences.

[0025] In some embodiments, comparing the distribution differences of the pose distribution forms of the plurality of radar sensors in the initial design pose and the actual installation pose, and determining the optimized poses of the plurality of radar sensors with the goal of minimizing the distribution differences includes:

[0026] The initial design poses and the actual installation poses of the plurality of radar sensors are obtained as vertices, and the pose errors between the initial design poses and the actual installation poses of the plurality of radar sensors are obtained as edges to construct a graph optimization model; and the optimized poses of the plurality of radar sensors are determined by minimizing the distribution error based on the graph optimization model.

[0027] A radar point cloud fusion method, comprising:

[0028] Acquire image point clouds based on the calibrated multiple image sensors, and map the image point clouds to a preset coordinate system;

[0029] generating a grid map in the preset coordinate system based on the image point cloud, and determining a grid state of each grid in the grid map according to a state in which each point in the image point cloud occupies each grid;

[0030] collecting a radar point cloud based on an uncalibrated radar sensor, mapping the radar point cloud to the preset coordinate system according to an initial design pose of the radar sensor in the preset coordinate system, and obtaining a value of a grid state of a grid occupied by each point in the radar point cloud;

[0031] According to the grid state value of the grid occupied by each point in the radar point cloud, the position and posture of the radar point cloud are adjusted in the preset coordinate system to fuse the radar point cloud with the image point cloud.

[0032] In some embodiments, the plurality of image sensors form a surround view image sensor array, and the image point cloud collected by the plurality of image sensors after calibration is a bowl-shaped dense point cloud.

[0033] In some embodiments, the image point cloud and the radar point cloud are both three-dimensional point clouds, the preset coordinate system is a three-dimensional spatial coordinate system, and the grid map is a three-dimensional grid map; obtaining the grid state value of the grid occupied by each point in the radar point cloud includes:

[0034] Determining a value of a grid state at any point in the grid map by performing trilinear interpolation on the grid map;

[0035] According to the position of each point in the radar point cloud in the grid map, a value of the grid state of each point in the radar point cloud is determined.

[0036] In some embodiments, adjusting the position and posture of the radar sensor in the preset coordinate system according to a grid state value of a grid occupied by each point in the radar point cloud includes:

[0037] determining a total value of the grid state of the radar point cloud according to the grid state value of each point in the radar point cloud;

[0038] determining a gradient of the radar point cloud according to a value of a grid state of each point in the radar point cloud;

[0039] In the preset coordinate system, the pose of the radar point cloud is adjusted along the gradient direction of the radar point cloud so that the total value of the grid state of the radar point cloud converges.

[0040] In some embodiments, adjusting the pose of the radar point cloud along the gradient direction of the radar point cloud in the preset coordinate system so that the total value of the grid state of the radar point cloud converges includes:

[0041] Determining whether a change in a total value of the radar point cloud before and after adjusting the posture of the radar point cloud is no greater than a preset threshold;

[0042] When a change in the total value of the radar point cloud before and after the posture of the radar point cloud is adjusted is not greater than the preset threshold, it is determined that the total value of the grid state of the radar point cloud has converged.

[0043] In some embodiments, the method further comprises:

[0044] The size of the grid in the grid map is gradually reduced to fuse the radar sensor with gradually improved accuracy, until the size of the grid in the grid map reaches a set size, and then it is determined that the radar point cloud and the image point cloud are completely fused.

[0045] A non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to cause the computer to execute the above method.

[0046] The radar point cloud fusion method, radar sensor calibration method, and related equipment provided in this embodiment collect image point clouds based on multiple calibrated image sensors and map the image point clouds to a preset coordinate system; generate a grid map within the preset coordinate system based on the image point clouds, and determine the grid state of each grid in the grid map based on the state of each point in the image point cloud; collect radar point clouds based on an uncalibrated radar sensor, map the radar point cloud to the preset coordinate system according to the initial design position of the radar sensor in the preset coordinate system, and obtain the grid state value of the grid occupied by each point in the radar point cloud; adjust the radar sensor's position within the preset coordinate system based on the grid state value of the grid occupied by each point in the radar point cloud, and obtain the actual installation position of the radar sensor in the preset coordinate system. This embodiment solves the problem of low calibration efficiency of vehicle-mounted radar sensors and improves the calibration efficiency of vehicle-mounted radar sensors.

[0047] The details of one or more embodiments of the present application are set forth in the following drawings and description to make other features, objects, and advantages of the present application more readily apparent. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0049] Figure 1 is a flowchart of the radar sensor calibration method of this embodiment.

[0050] Figure 2 is a flow chart of a calibration method for multiple image sensors according to this embodiment.

[0051] Figure 3 4 is a flowchart of the method for processing image point clouds of this embodiment.

[0052] Figure 4 4 is a flowchart of the multi-resolution grid matching method based on the grid map of this embodiment.

[0053] Figure 5 This is a preferred flow chart of the radar sensor calibration method of this embodiment.

[0054] Figure 6 Schematic diagram of constructing a graph optimization model in this embodiment.

[0055] Figure 7 4 is a flowchart of the radar point cloud fusion method of this embodiment.

[0056] Figure 8 It is a structural block diagram of the electronic device of this embodiment. DETAILED DESCRIPTION

[0057] The following describes embodiments of the present invention in more detail with reference to the accompanying drawings. Although certain embodiments of the present invention are shown in the accompanying drawings, it should be understood that the present invention can be implemented in various forms and should not be construed as being limited to the embodiments described herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of the present invention. It should be understood that the accompanying drawings and embodiments of the present invention are for illustrative purposes only and are not intended to limit the scope of protection of the present invention.

[0058] It should be understood that the various steps described in the method implementation of this embodiment can be performed in different orders and / or in parallel. In addition, the method implementation may include additional steps and / or omit the steps shown. The scope of this embodiment is not limited in this respect.

[0059] As used herein, the term "including" and its variations are open-ended, i.e., "including but not limited to." The term "based on" means "based, at least in part, on." The term "one embodiment" means "at least one embodiment," the term "another embodiment" means "at least one additional embodiment," and the term "some embodiments" means "at least some embodiments." Other terms are defined in the following description.

[0060] It should be noted that the modifications of "one" and "multiple" mentioned in this embodiment are illustrative rather than restrictive. Those skilled in the art should understand that unless otherwise clearly indicated in the context, they should be understood as "one or more".

[0061] Intelligent driving, 3D scene modeling, intelligent monitoring, and other fields all involve data fusion from multiple types (or multimodal) of sensors. For example, the fusion of image sensor data with radar sensor data is a key technology for intelligent driving. Image sensors include but are not limited to visible light and infrared sensors. Their imaging principle is to use photoelectric conversion devices, such as charge-coupled devices (CCDs), to convert images captured by an optical lens module into electrical signals to produce an image. The intrinsic parameters of an image sensor represent the transformation relationship between the image coordinate system and the camera coordinate system. In addition, due to the presence of fisheye distortion, the camera intrinsic parameters may also include distortion coefficients used to correct for fisheye distortion. Typically, the intrinsic parameters of an image sensor are fixed and provided at the time of shipment. The extrinsic parameters of an image sensor represent the transformation relationship between the camera coordinate system and the world coordinate system. These parameters are related to the image sensor's position in the world coordinate system. Extrinsic parameters can only be calibrated after the image sensor is installed in a fixed position and set to a fixed position.

[0062] This embodiment provides a radar sensor calibration method. The purpose of calibrating a radar sensor is to obtain the actual installation position of the radar sensor. Figure 1 FIG. 1 is a flow chart of the radar sensor calibration method of this embodiment. Figure 1 As shown, the process includes the following steps:

[0063] Step S101 : collecting image point clouds based on multiple calibrated image sensors, and mapping the image point clouds to a preset coordinate system.

[0064] Step S102 : generating a grid map in a preset coordinate system based on the image point cloud, and determining the grid state of each grid in the grid map according to the state of each point in the image point cloud occupying each grid.

[0065] In step S103 , a radar point cloud is collected based on the uncalibrated radar sensor, the radar point cloud is mapped to a preset coordinate system according to an initial design pose of the radar sensor in the preset coordinate system, and a grid state value of a grid occupied by each point in the radar point cloud is obtained.

[0066] Step S104 : adjusting the position and posture of the radar sensor in the preset coordinate system according to the grid state value of the grid occupied by each point in the radar point cloud, and obtaining the actual installation position and posture of the radar sensor in the preset coordinate system.

[0067] Compared to existing technologies, the above steps, on the one hand, eliminate the need for simultaneous calibration of the image sensor and radar sensor. Instead, the image sensor is calibrated first, followed by the radar sensor, to avoid the poor lidar calibration caused by simultaneous calibration of both sensors. Furthermore, in the above steps, the state of the image point cloud in the grid map is used to assign a value to each grid in the grid map to obtain a grid state. Then, by mapping the radar point cloud to the grid map, a map-based matching method is used to align the radar point cloud with the image point cloud, obtaining the actual installation pose after optimizing the radar point cloud.

[0068] Through the above steps, the calibration of the radar sensor fully utilizes the calibration results of the image sensor, and the above process can be implemented based on a computer program, eliminating the need for manual calibration of the lidar using a calibration plate, thereby saving labor costs and improving calibration efficiency.

[0069] It should be noted that the postures in this embodiment all refer to postures in the same coordinate system, that is, the preset coordinate system referred to in this embodiment, and the preset coordinate system may be, for example, a world coordinate system.

[0070] Image point clouds collected by image sensors and radar sensors are called dense point clouds. These are the opposite of sparse point clouds. Sparse point clouds originate from feature points, which are distinct, easy-to-detect and match points in an image, such as building corners and edges. Dense point clouds, on the other hand, are point clouds collected by sensors.

[0071] To perceive objects in the space surrounding an object, multiple image sensors form a surround-view image sensor array, referred to in this embodiment as a surround-view camera. The image point cloud captured by the calibrated multiple image sensors is a dense, bowl-shaped point cloud. Step S101 can be implemented using methods such as a calibration plate to calibrate the multiple image sensors. Figure 2 is a flow chart of the calibration method of multiple image sensors of this embodiment, as shown in FIG. Figure 2 As shown, the surround-view camera is first calibrated to extract dense point cloud data, generate a bowl-shaped model, and define the model's coordinate origin as the world coordinate system. The calibration of the surround-view camera is divided into intrinsic and extrinsic parameters. The camera's intrinsic parameters reflect the projection relationship between the camera plane coordinates and the image coordinate system. The Zhang Zhengyou calibration method is used to obtain the camera's intrinsic parameters and distortion coefficients. The camera's extrinsic parameters reflect the rotation and translation relationship between the camera coordinate system and the world coordinate system. Using a calibration plate, two adjacent cameras are ensured to simultaneously capture images of the calibration plate. The corner features of the calibration plate are then matched to calculate the rotation and translation relationship between the two cameras. Finally, the different cameras are matched pairwise to complete the calibration of all image sensors.

[0072] Due to the existence of fisheye distortion, after the surround view camera is calibrated, the point cloud data collected by each image sensor cannot be directly spliced ​​together and needs to be corrected for distortion. Figure 3 is a flow chart of the method for processing image point cloud of this embodiment, as shown in FIG. Figure 3 As shown in the figure, extracting an image point cloud involves two main steps: first, performing distortion correction and stereo correction on the image to obtain more accurate image data. After completing camera intrinsic calibration, image distortion is corrected using intrinsic parameters. After completing extrinsic calibration, epipolar lines are corrected using the rotation and translation relationship, which is stereo correction. Second, a disparity map is obtained using the semi-global block matching (SGBM) algorithm. This disparity map is converted into a depth map using a conversion formula, and feature point extraction is used to obtain 3D point cloud data based on the depth image. Afterwards, the world coordinate origin is determined, and the point clouds collected by all image sensors are spliced ​​together to create a bowl-shaped point cloud model.

[0073] Unlike the strategy of directly matching single-frame camera images with radar data, this embodiment first completes the calibration between cameras to obtain the point cloud data of all images. Then, the SGBM algorithm is used to obtain the disparity map and depth map. Finally, the 3D point cloud data is extracted from the depth map. This avoids the problem of frame-to-frame matching easily falling into the local minimum and failing to converge.

[0074] In the above embodiment, the grid map is called an occupancy grid map. The grid map divides the space into grids and uses the grid status to represent the probability of each grid being occupied or idle.

[0075] For example, for a grid, it is either occupied by a point cloud (Occupancy state, represented by 1) or not occupied by a point cloud (Free state, represented by 0). To express the probability of the Free state, use To represent the probability of the Occupancy state, the sum of the two is 1. Since there are too many possibilities for two values, the ratio of the two is introduced as the state of the grid:

[0076] ;

[0077] When multiple point clouds hit the same grid, the probability of the Occupancy state increases, so the state of the grid will be updated. If the state of the grid at the current moment is , when a point cloud hits this grid, the grid status will be updated as follows:

[0078] ;

[0079] The above expression is similar to the conditional probability, which means that under the condition that z (a new point cloud hits the grid) occurs probability.

[0080] According to Bayes' formula, we know that:

[0081] ;

[0082] ;

[0083] Bringing this into the grid state update formula yields:

[0084] ;

[0085] Taking the logarithm of both sides of the equation yields:

[0086] ;

[0087] Thus, the only items containing measurement values ​​are , we call this ratio the measurement model (Measurement Model), marked as lomeas. There are two types of measurement models, namely:

[0088] ,and ;

[0089] And the above two models are both fixed values, so the state update can be further simplified as:

[0090] ,or .

[0091] Therefore, by utilizing the state update of the grid map, in the above embodiment, the image point cloud can be mapped onto the three-dimensional grid map to obtain the grid map.

[0092] Among them, when there is no measurement value, the initial state of the grid is 0.

[0093] After obtaining a grid map generated from the image point cloud, theoretically, by applying rigid transformations such as translation and rotation to the radar point cloud, if each point in the radar point cloud can precisely hit each grid cell occupied by the image point cloud, then the translation and rotation values, combined with the initial design pose of the radar point cloud, can be used to determine the actual installation pose of the radar sensor. The radar sensor calibration problem then becomes how to match the radar point cloud with the grid map. This matching problem can be addressed using map-based matching (SCM) methods.

[0094] In a three-dimensional grid map, each grid cell occupies a specific three-dimensional space and has a state, or grid state value. When the grid cells are small enough, any point in the three-dimensional space can be considered to have a grid state value. However, the smaller the grid cells, the greater the number of grid cells in the grid map, requiring more computer processing resources. Therefore, in practical applications, given the available computer processing resources, the grid size is typically not too small.

[0095] In this embodiment, two methods are used to determine the grid state value at any point in space. One method is to assign the value of the entire space occupied by the grid as the grid state value of the grid. However, this method often results in large errors, especially when the grid size is large. Another method is to obtain the grid state value at any point in space through interpolation.

[0096] In the case where both the image point cloud and the radar point cloud are three-dimensional point clouds, the preset coordinate system is a three-dimensional spatial coordinate system, and the grid map is a three-dimensional grid map, in some embodiments, the grid state value of the grid is used as the value of the center point of the grid, and the grid state value of any point in space is obtained by trilinear interpolation. For example, in step S103 above, obtaining the grid state value of the grid occupied by each point in the radar point cloud includes the following steps: determining the grid state value of any point in the grid map by performing trilinear interpolation on the grid map; and determining the grid state value of each point in the radar point cloud based on the position of each point in the radar point cloud in the three grid maps.

[0097] In this embodiment, three-dimensional linear interpolation is used to determine the grid state value at any point in space. For each unit-length cubic grid, the eight nearest grid center points surrounding the point to be determined are selected. These eight grid center points encompass a cubic volume in space. Based on the grid state values ​​of these eight grid center points, interpolation is performed along the x-, y-, and z-axes to determine the grid state value for any point within the space enclosed by these eight grid points.

[0098] In step S104, adjusting the position and posture of the radar sensor in a preset coordinate system according to the value of the grid state of the grid occupied by each point in the radar point cloud includes: determining a total value of the grid state of the radar point cloud according to the value of the grid state of each point in the radar point cloud; determining a gradient of the radar point cloud according to the value of the grid state of each point in the radar point cloud; and adjusting the position and posture of the radar point cloud along the gradient direction of the radar point cloud in the preset coordinate system so that the total value of the grid state of the radar point cloud converges.

[0099] Among them, after obtaining the grid state value corresponding to each point in the radar point cloud through trilinear interpolation, when calculating the total grid state value of the radar point cloud, the grid state value of each point in the radar point cloud can be directly added to obtain the total value of the radar point cloud; the total value of the radar point cloud can also be calculated by a kernel function, for example, the kernel function can be: linear kernel function, polynomial kernel function or Gaussian kernel function, etc.

[0100] For example, the Gaussian kernel function is used to calculate the total value of the radar point cloud:

[0101] ;

[0102] in is the Euclidean distance from the current point to the center point, In this embodiment, it can be set to 0.5. Indicates the point The value in the current grid.

[0103] The above embodiment exhibits excellent robustness and stability. For example, when calculating the total value of a radar point cloud, different kernel functions can be used based on different environmental characteristics, thereby improving the robustness of the matching algorithm. Furthermore, the above embodiment utilizes a dense point cloud as the processing target, eliminating the need to extract linear features such as lines and arcs in the image. This allows for application in environments with relatively limited geometric features, resulting in excellent stability and robustness.

[0104] Ideally, if, after the translation and rotation transformations of the radar point cloud, each point in the radar point cloud precisely hits each grid cell occupied by the image point cloud, the total value of the radar point cloud will reach its maximum value. Therefore, in the above embodiment, the maximum total value of the radar point cloud can be used as the convergence condition. After the radar point cloud is translated and rotated, the radar point cloud can be accurately matched to the grid map. If the translation and rotation amounts are randomly adjusted, the matching convergence efficiency is low. Therefore, in this embodiment, to improve convergence efficiency, the gradient of the radar point cloud is determined based on the grid state value of each point in the radar point cloud. Within a preset coordinate system, the position of the radar point cloud is adjusted along the gradient direction of the radar point cloud to increase the total value of the radar point cloud and ultimately achieve convergence.

[0105] To determine whether the radar point cloud's pose has reached convergence conditions, the rate of change of the radar point cloud's total values ​​can be used to determine this. For example, the change in the total values ​​of the radar point cloud before and after the pose of the radar point cloud is adjusted is determined to be no greater than a preset threshold. If the change in the total values ​​of the radar point cloud before and after the pose of the radar point cloud is adjusted is no greater than the preset threshold, the pose of the radar point cloud is considered to have reached convergence conditions. At this point, the current pose of the radar sensor in the preset coordinate system is obtained as the actual installation pose of the radar sensor in the preset coordinate system. Using the rate of change of the total values ​​of the radar point cloud to determine whether the radar point cloud has been accurately matched to the grid map can further improve convergence efficiency, that is, enhance the matching efficiency of the radar point cloud.

[0106] To speed up optimization, some embodiments employ a multi-resolution grid method. Initially, a low-resolution (i.e., large-size) grid is used to determine the radar sensor's initial pose. The grid map resolution is then gradually increased, with the grid size gradually reduced. Once the grid size reaches the set size, the radar sensor's actual installed pose within the preset coordinate system is determined. This process results in increasingly accurate radar sensor poses.

[0107] Figure 4 : is a flowchart of the multi-resolution grid matching method based on the grid map of this embodiment. Figure 4 As shown, the process includes the following steps:

[0108] Step S401: Generate a grid map with an initial size grid using the image point cloud.

[0109] Step S402: performing cubic linear interpolation on the grid map so that the values ​​of any point in the grid map are continuous.

[0110] Step S403 : Mapping the radar point cloud to a grid map with continuous values, calculating the total value of the radar point cloud, and calculating the gradient of the radar point cloud.

[0111] Step S404 , iterating along the gradient direction in the direction of increasing total value of the radar point cloud to optimize the pose of the radar point cloud until the value change is less than a preset threshold, thereby obtaining the actual installation pose of the radar sensor in the grid map with the current grid size.

[0112] Step S405: Reduce the size of the grid, regenerate the grid map, and then execute step S402.

[0113] For example, in each round of calculating the radar sensor's actual installation pose, the grid size in the grid map is half the grid size used in the previous round of calculating the radar sensor's actual installation pose. For example, the grid sizes used in each round are 20 cm, 10 cm, 5 cm, 2.5 cm, and so on. The preset threshold in step S404 also decreases with each round of calculation as the grid size decreases.

[0114] In the first round of radar point cloud pose iteration, the initial design pose of the radar sensor is used as the starting pose for optimization. In each subsequent round of radar point cloud pose iteration, the actual installation pose of the radar sensor calculated from the low-resolution grid map in the previous round is used as the starting pose for the current round of radar point cloud pose iteration.

[0115] In the above embodiment, the radar point cloud is mapped onto a grid map, and first matched with a low-resolution grid map (SCM, map-based matching) to obtain an approximate optimal solution for the radar. Then, the grid map resolution is gradually increased to make the radar pose gradually approach the optimal solution. Figure 3 Linear interpolation makes any pose of the raster map continuously differentiable, and gradient descent is used to optimize the radar pose. This method not only speeds up matching but also prevents the matching results from falling into local minima, effectively improving matching accuracy. This method avoids local minima and achieves high matching accuracy. It also gradually increases map resolution during the matching process, enabling the optimal pose to be fitted on the global map.

[0116] Through matching, the optimized radar sensor pose is obtained. Since the radar sensor pose is obtained by matching with the image point cloud data, it may be affected by image sensor errors. To further improve the matching accuracy of the radar pose, this embodiment also combines the pose distribution of the radar sensor to further optimize the radar sensor pose.

[0117] In some embodiments, there are multiple radar sensors. After obtaining the actual installation posture of the radar sensor in the above step S104, the following optimization step may be further included: comparing the distribution differences of the posture distribution forms of the multiple radar sensors under the initial design posture and the actual installation posture; and determining the optimized postures of the multiple radar sensors with the goal of minimizing the distribution differences.

[0118] It is known that the installation postures of multiple radar sensors do not change between their initial design postures and their actual installation postures. Therefore, the posture distributions formed by their initial design postures and their actual installation postures should theoretically be identical. However, due to computational errors, errors caused by image sensors, and errors between the actual initial design postures of the radar sensors and their design values, errors may occur in the calibration of the multiple radar sensors. This error manifests itself as the posture distributions of the multiple radar sensors not remaining consistent before and after calibration. In this embodiment, the optimized postures of the multiple radar sensors can be determined by comparing the distribution differences of the posture distributions before and after calibration and minimizing these distribution differences.

[0119] In some embodiments, the distribution errors of multiple radar sensors are optimized by a graph optimization method. Figure 5 This is a preferred flow chart of the radar sensor calibration method of this embodiment. Figure 5 As shown, the process includes the following steps:

[0120] Step S501 : collecting image point clouds based on multiple calibrated image sensors, and mapping the image point clouds to a preset coordinate system.

[0121] Step S502 : generating a grid map in a preset coordinate system based on the image point cloud, and determining the grid state of each grid in the grid map according to the state of each point in the image point cloud occupying each grid.

[0122] In step S503 , a radar point cloud is collected based on the uncalibrated radar sensor, the radar point cloud is mapped to a preset coordinate system according to an initial design pose of the radar sensor in the preset coordinate system, and a grid state value of a grid occupied by each point in the radar point cloud is obtained.

[0123] Step S504 : Adjust the position and posture of the radar sensor in the preset coordinate system according to the grid state value of the grid occupied by each point in the radar point cloud, and obtain the actual installation position and posture of the radar sensor in the preset coordinate system.

[0124] Step S505: Acquire the initial design poses and actual installation poses of the plurality of radar sensors as vertices, acquire the pose errors between the initial design poses and actual installation poses of the plurality of radar sensors as edges, and construct a graph optimization model;

[0125] Step S506 : Minimizing the distribution error based on the graph optimization model to determine the optimized poses of the multiple radar sensors.

[0126] Figure 6 is a schematic diagram of constructing a graph optimization model in this embodiment, Figure 6 In the figure, the solid circle represents the initial design pose of the radar sensor, and the shaded circle represents the actual installation pose of the radar sensor. The dashed circle represents the pose of the radar sensor after the actual installation pose is transformed. The dashed arrows represent the transformation of the actual installation pose, and the solid arrows represent the residual term. In this embodiment, based on the principles of graph optimization, the actual installation pose and the initial design pose of the radar sensor are used as vertices to construct a pose graph optimization (PGO) model. The entire process only optimizes the pose of the radar sensor.

[0127] Assume that the initial design poses of multiple radar sensors are After the above calibration process, the actual installation postures of multiple radar sensors are obtained as follows: ,calculate , The relative posture relationship of , is the same radar sensor selected from multiple radar sensors), denoted as , making

[0128] ;

[0129] The following posture distribution can be obtained: , T .make , we can get the following pose points: ,…, , .

[0130] After obtaining the two pose distribution forms, the relationship between adjacent poses is calculated, with the pose of the radar sensor as the vertex and the pose error as the edge, and the least squares term is obtained:

[0131]

[0132] in, Indicates that The observation value of the point, in this scheme Indicates Observed The pose of the point.

[0133] After derivation and optimization, the optimized pose of the radar sensor can be obtained:

[0134] ,…, , ;

[0135] in, ,and Represents different poses of the same radar sensor.

[0136] In addition to the least square method, other optimization methods may be used to minimize the distribution difference, which is not limited in this embodiment.

[0137] In some embodiments, after obtaining the optimized posture, it is possible to further perform weighted averaging based on the initial design posture and the optimized posture according to a manually set weighting coefficient to manually optimize the calibration results and obtain the final posture of the radar sensor in the preset coordinate system.

[0138] In the above embodiment, when constructing the graph optimization model, the initial design pose of the radar sensor and the optimized actual installation pose are added to the graph, with only the radar pose as the vertex of the graph, and the deviation between the observed value and the design value of the radar sensor as the residual term to optimize the radar pose. During the optimization process, it is not affected by other sensors (such as image sensors). In the graph optimization model constructed by the above embodiment, each radar sensor has an initial pose (in the above embodiment, the first radar sensor is selected, and the corresponding poses are respectively , ), each radar sensor has two vertices in the graph. In the post-optimization processing, the same or different weighting coefficients can be given to each vertex, and the final radar pose can be obtained by taking the weighted average.

[0139] The graph optimization model constructed using the above method is constrained by the initial design pose of the radar sensor and the joint calibration pose of the radar sensor and image sensor. When the initial design pose of the radar sensor is inaccurate or the joint calibration result of the radar sensor and image sensor is inaccurate, resulting in the radar point cloud not conforming to the real environment characteristics, the radar pose can be effectively calibrated by adjusting the weight of each vertex and the weighted average coefficient to obtain more accurate results.

[0140] In summary, the above-described embodiments of the present application convert the image captured by the image sensor into a depth map, extract the image point cloud from the depth map, construct a three-dimensional grid map from the image point cloud, and then map the radar point cloud data onto the three-dimensional grid map. Trilinear interpolation is used to make the grid map continuous and differentiable, and gradient descent is used to optimize the radar sensor's position. This method is based on map matching, and the impact of single-frame image data on the matching results is minimal. Furthermore, using grid maps of varying resolutions prevents the matching from falling into local minima.

[0141] The grid map generated by the image sensor can be saved. In the event of unexpected changes in the radar sensor posture, image sensor posture, or image sensor damage, the radar sensor can be recalibrated with the saved grid map to obtain the optimized radar sensor posture.

[0142] The entire solution is robust. The above embodiment eliminates the need to search for geometrically feature-rich areas in the environment to extract geometric features such as straight lines. In the event of an image sensor failure, previously calibrated, saved point cloud data and raster maps can be used. Whether the initial (installation) pose of the radar sensor is inaccurate or the combined calibration of the radar sensor and image sensor is poor, the calibration effect can be improved by adjusting vertex weights or the weight coefficients used in weighted averaging.

[0143] This embodiment also provides a radar point cloud fusion method. Figure 7 FIG. 1 is a flow chart of the radar point cloud fusion method of this embodiment. Figure 7 As shown, the process includes the following steps:

[0144] Step S701 : collecting image point clouds based on the calibrated multiple image sensors, and mapping the image point clouds to a preset coordinate system.

[0145] Step S702 : generating a grid map in a preset coordinate system based on the image point cloud, and determining the grid state of each grid in the grid map according to the state of each point in the image point cloud occupying each grid.

[0146] In step S703 , a radar point cloud is collected based on the uncalibrated radar sensor, the radar point cloud is mapped to the preset coordinate system according to the initial design pose of the radar sensor in the preset coordinate system, and the grid state value of the grid occupied by each point in the radar point cloud is obtained.

[0147] In step S704 , the position and orientation of the radar point cloud are adjusted in a preset coordinate system according to the grid state values ​​of the grids occupied by each point in the radar point cloud, so as to fuse the radar point cloud with the image point cloud.

[0148] In some embodiments, multiple image sensors form a surround view image sensor array. The image point cloud captured by these calibrated image sensors is a dense, bowl-shaped point cloud. A dense point cloud is the opposite of a sparse point cloud. A sparse point cloud originates from feature points, which are distinct, easy-to-detect and match points in an image, such as building corners and edges. A dense point cloud is the point cloud captured by the sensors.

[0149] To perceive objects in the space around an object, multiple image sensors form a surround view image sensor array, which is also referred to as a surround view camera in this embodiment. The image point cloud collected by the calibrated multiple image sensors is a dense bowl-shaped point cloud.

[0150] In some embodiments, collecting image point clouds based on multiple calibrated image sensors includes: collecting multiple images based on the multiple calibrated image sensors; performing distortion correction on each image, and performing stereo correction on two images collected by two adjacent image sensors in the multiple images; obtaining disparity images of the two images based on the two stereo-corrected images, and obtaining a depth image based on the disparity images of the two images; and extracting the image point cloud based on the depth image.

[0151] Continue to refer Figure 2 . First, calibrate the surround-view camera, extract dense point cloud data, generate a bowl-shaped model, and define the model coordinate origin as the world coordinate system. The calibration of the surround-view camera is divided into internal parameters and external parameters. The camera internal parameters reflect the projection relationship between the camera plane coordinates and the image coordinate system. The Zhang Zhengyou calibration method is used to obtain the camera's internal parameters and distortion coefficients. The camera's external parameters reflect the rotation and translation relationship between the camera coordinate system and the world coordinate system. Through the calibration plate, it is ensured that two adjacent cameras simultaneously capture the image of a calibration plate, match the corner point features of the calibration plate, and solve the rotation and translation relationship between the two cameras. Finally, different cameras are matched pairwise to complete the calibration of all image sensors.

[0152] Due to the existence of fisheye distortion, after calibrating the surround view camera, the point cloud data collected by each image sensor cannot be directly spliced ​​together and needs to be corrected for distortion. Figure 3Extracting an image point cloud involves two main steps: First, image distortion and stereo correction are performed to obtain more accurate image data. After completing camera intrinsic calibration, image distortion is corrected using intrinsic parameters. After completing extrinsic calibration, epipolar lines are corrected using rotation and translation relationships, a process known as stereo correction. Second, a disparity map is obtained using the semi-global block matching (SGBM) algorithm. This is then converted into a depth map using a conversion formula. Feature point extraction is then used to obtain three-dimensional point cloud data based on the depth image. Afterwards, the world coordinate origin is determined, and the point clouds collected by all image sensors are spliced ​​together to create a bowl-shaped point cloud model.

[0153] Unlike the strategy of directly matching single-frame camera images with radar data, this embodiment first completes the calibration between cameras to obtain the point cloud data of all images. Then, the SGBM algorithm is used to obtain the disparity map and depth map. Finally, the 3D point cloud data is extracted from the depth map. This avoids the problem of frame-to-frame matching easily falling into the local minimum and failing to converge.

[0154] In some embodiments, both the image point cloud and the radar point cloud are three-dimensional point clouds, the preset coordinate system is a three-dimensional spatial coordinate system, and the grid map is a three-dimensional grid map; obtaining the value of the grid state of the grid occupied by each point in the radar point cloud includes: determining the value of the grid state of any point in the grid map by performing trilinear interpolation on the grid map; and determining the value of the grid state of each point in the radar point cloud according to the position of each point in the radar point cloud in the grid map.

[0155] In this embodiment, two methods are used to determine the grid state value at any point in space. One method is to assign the value of the entire space occupied by the grid as the grid state value of the grid. However, this method often results in large errors, especially when the grid size is large. Another method is to obtain the grid state value at any point in space through interpolation.

[0156] In some embodiments, the grid state value of the grid is used as the value of the center point of the grid, and the grid state value of any point in space is obtained by trilinear interpolation. For example, in step S703, obtaining the grid state value of the grid occupied by each point in the radar point cloud includes the following steps: determining the grid state value of any point in the grid map by performing trilinear interpolation on the grid map; and determining the grid state value of each point in the radar point cloud based on the position of each point in the radar point cloud in the three grid maps.

[0157] In this embodiment, three-dimensional linear interpolation is used to determine the grid state value at any point in space. For each unit-length cubic grid, the center points of the eight closest grids are selected at the position of the point to be determined. These eight grid center points enclose a cubic volume in space. Based on the grid state values ​​of these eight grid center points, interpolation is performed along the x-, y-, and z-axes to determine the grid state value for any point within the space enclosed by these eight grid points.

[0158] In step S704, adjusting the position and posture of the radar sensor in a preset coordinate system according to the value of the grid state of the grid occupied by each point in the radar point cloud includes: determining a total value of the grid state of the radar point cloud according to the value of the grid state of each point in the radar point cloud; determining a gradient of the radar point cloud according to the value of the grid state of each point in the radar point cloud; and adjusting the position and posture of the radar point cloud along the gradient direction of the radar point cloud in the preset coordinate system so that the total value of the grid state of the radar point cloud converges.

[0159] The grid state value corresponding to each point in the radar point cloud is obtained through trilinear interpolation. When calculating the total grid state value of the radar point cloud, the grid state values ​​of each point in the radar point cloud can be directly added together to obtain the total value of the radar point cloud. Alternatively, the total value of the radar point cloud can be calculated using a kernel function, such as a linear kernel function, a polynomial kernel function, or a Gaussian kernel function.

[0160] For example, the Gaussian kernel function is used to calculate the total value of the radar point cloud:

[0161] ;

[0162] in is the Euclidean distance from the current point to the center point, In this embodiment, it can be set to 0.5. Indicates the point The value in the current grid.

[0163] The above embodiment exhibits excellent robustness and stability. For example, when calculating the total value of a radar point cloud, different kernel functions can be used based on different environmental characteristics, thereby improving the robustness of the matching algorithm. Furthermore, the above embodiment utilizes a dense point cloud as the processing target, eliminating the need to extract linear features such as lines and arcs in the image. This allows for application in environments with relatively limited geometric features, resulting in excellent stability and robustness.

[0164] Ideally, if, after the translation and rotation transformation of the radar point cloud, each point in the radar point cloud precisely hits each grid occupied by the image point cloud, the total value of the radar point cloud will be maximized. Therefore, in the above embodiment, with maximizing the total value of the radar point cloud as the optimization goal, the radar point cloud can be accurately matched to the grid map after translation and rotation transformation. If the translation and rotation amounts are randomly adjusted, the matching convergence efficiency is low. To this end, in this embodiment, the gradient of the radar point cloud can also be determined based on the grid state value of each point in the radar point cloud; within a preset coordinate system, the position of the radar point cloud is adjusted along the gradient direction of the radar point cloud to increase the total value of the radar point cloud.

[0165] To determine whether the radar point cloud's pose has reached convergence conditions, the rate of change of the radar point cloud's total values ​​can be used to determine this. For example, the change in the total value of the radar point cloud before and after the pose of the radar point cloud is adjusted is determined to be no greater than a preset threshold. If the change in the total value of the radar point cloud before and after the pose of the radar point cloud is adjusted is no greater than the preset threshold, the pose of the radar point cloud is considered to have reached convergence conditions, and the radar point cloud and image point cloud are determined to be fully fused. Using the rate of change of the total value of the radar point cloud to determine whether the radar point cloud has been accurately matched to the grid map can further improve convergence efficiency, that is, improve the matching efficiency of the radar point cloud.

[0166] To speed up optimization, some embodiments employ a multi-resolution grid method. Initially, a low-resolution (i.e., large-size) grid is used to determine the initial radar sensor pose. The grid map resolution is then gradually increased, with the grid size gradually reduced. Once the grid size reaches a predetermined size, the radar point cloud and image point cloud are considered fused. This process results in increasingly accurate fusion of the radar and image point clouds.

[0167] This embodiment further provides an electronic device, comprising: at least one processor; and a memory communicatively connected to the at least one processor. The memory stores a computer program executable by the at least one processor, wherein the computer program, when executed by the at least one processor, causes the electronic device to perform a method according to an embodiment of the present disclosure.

[0168] This embodiment further provides a non-transitory computer-readable storage medium storing a computer program, wherein the computer program, when executed by a processor of a computer, is used to cause the computer to perform the method according to the embodiment of the present disclosure.

[0169] refer to Figure 8, a block diagram of an electronic device 800 that can serve as a server or client of the present disclosure will now be described, which is an example of a hardware device that can be applied to various aspects of the present disclosure. The electronic device is intended to represent various forms of digital electronic computer devices, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0170] like Figure 8 As shown, electronic device 800 includes a computing unit 801, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 802 or a computer program loaded from a storage unit 808 into a random access memory (RAM) 803. RAM 803 may also store various programs and data required for the operation of device 800. Computing unit 801, ROM 802, and RAM 803 are interconnected via a bus 804. An input / output (I / O) interface 805 is also connected to bus 804.

[0171] Multiple components within electronic device 800 are connected to I / O interface 805, including an input unit 806, an output unit 807, a storage unit 808, and a communication unit 809. Input unit 806 can be any type of device capable of inputting information into electronic device 800. Input unit 806 can receive input numeric or character information and generate key signal inputs related to user settings and / or function control of the electronic device. Output unit 807 can be any type of device capable of presenting information and may include, but is not limited to, a display, a speaker, a video / audio output terminal, a vibrator, and / or a printer. Storage unit 808 may include, but is not limited to, a magnetic disk or an optical disk. Communication unit 809 allows electronic device 800 to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks and may include, but is not limited to, a modem, a network card, an infrared communication device, a wireless communication transceiver such as a Bluetooth device, a WiFi device, a WiMax device, a cellular communication device, and / or the like.

[0172] The computing unit 801 may be any general-purpose and / or specialized processing component with processing and computing capabilities. Some examples of the computing unit 801 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various specialized artificial intelligence (AI) computing units, various computing units that run machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The computing unit 801 performs the various methods and processes described above. For example, in some embodiments, the above methods (e.g., Figure 1 or Figure 5 The methods described above may be implemented as computer software programs that are tangibly contained in a machine-readable medium, such as the storage unit 808. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 800 via the ROM 802 and / or the communication unit 809. In some embodiments, the computing unit 801 may be configured to perform the methods described above in any other appropriate manner (e.g., by means of firmware).

[0173] The program code for implementing the method of the present disclosure can be written in any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device so that when the program code is executed by the processor or controller, the functions / operations specified in the flow chart and / or block diagram are implemented. The program code can be executed entirely on the machine, partially on the machine, as a stand-alone software package, partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0174] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared systems, apparatuses, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media may include an electrical connection based on one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), optical fibers, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0175] As used in this disclosure, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, apparatus, and / or device (e.g., a magnetic disk, an optical disk, a memory, a programmable logic device (PLD)) for providing machine instructions and / or data to a programmable processor, including machine-readable media that receive machine instructions as machine-readable signals. The term "machine-readable signal" refers to any signal used to provide machine instructions and / or data to a programmable processor.

[0176] To provide for interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) display) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or joystick) through which the user can provide input to the computer. Other types of devices can also be used to provide for interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0177] The systems and techniques described herein can be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), and the Internet.

[0178] Computer systems may include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The client and server relationship arises through computer programs running on the respective computers and having a client-server relationship to each other.

[0179] It should be understood that the specific embodiments described herein are only used to explain this application and are not used to limit it. Based on the embodiments provided in this application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0180] Obviously, the accompanying drawings are merely examples or embodiments of the present application. A person skilled in the art can also apply the present application to other similar situations based on these drawings without inventive effort. Furthermore, it is understandable that, although the work involved in this development process may be complex and lengthy, certain design, manufacturing, or production changes based on the technical content disclosed in this application are merely routine technical means for a person skilled in the art and should not be considered to constitute a deficiency in the disclosure of the present application.

[0181] The term "embodiment" in this application refers to specific features, structures, or characteristics described in conjunction with the embodiment that can be included in at least one embodiment of the present application. The various positions where this phrase appears in the specification do not necessarily mean the same embodiment, nor do they mean that it is mutually exclusive with other embodiments and is independent or optional. It is understood by those of ordinary skill in the art, either explicitly or implicitly, that the embodiments described in this application can be combined with other embodiments in the absence of conflict.

[0182] The above-described embodiments merely represent several embodiments of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of protection of the present invention. It should be noted that a person of ordinary skill in the art may make various modifications and improvements without departing from the spirit of the present application, and such modifications and improvements are all within the scope of protection of the present application. Therefore, the scope of protection of the present application shall be determined by the appended claims.

Claims

1. A radar sensor calibration method, comprising: Acquire image point clouds based on the calibrated multiple image sensors, and map the image point clouds to a preset coordinate system; generating a grid map in the preset coordinate system based on the image point cloud, and determining a grid state of each grid in the grid map according to a state in which each point in the image point cloud occupies each grid; collecting a radar point cloud based on an uncalibrated radar sensor, mapping the radar point cloud to the preset coordinate system according to an initial design pose of the radar sensor in the preset coordinate system, and obtaining a value of a grid state of a grid occupied by each point in the radar point cloud; According to the grid state value of the grid occupied by each point in the radar point cloud, the posture of the radar sensor is adjusted in the preset coordinate system to obtain the actual installation posture of the radar sensor in the preset coordinate system.

2. The method according to claim 1, wherein The image point cloud and the radar point cloud are both three-dimensional point clouds, the preset coordinate system is a three-dimensional space coordinate system, and the grid map is a three-dimensional grid map; The values ​​of the grid state of the grid occupied by each point in the radar point cloud are obtained by: Determining a value of a grid state at any point in the grid map by performing trilinear interpolation on the grid map; According to the position of each point in the radar point cloud in the grid map, a value of the grid state of each point in the radar point cloud is determined.

3. The method according to claim 1, wherein Adjusting the position and posture of the radar sensor in the preset coordinate system according to the grid state value of the grid occupied by each point in the radar point cloud includes: determining a total value of the grid state of the radar point cloud according to the grid state value of each point in the radar point cloud; determining a gradient of the radar point cloud according to a value of a grid state of each point in the radar point cloud; In the preset coordinate system, the pose of the radar point cloud is adjusted along the gradient direction of the radar point cloud so that the total value of the grid state of the radar point cloud converges.

4. The method according to claim 3, wherein: Adjusting the pose of the radar point cloud along the gradient direction of the radar point cloud in the preset coordinate system so that the total value of the grid state of the radar point cloud converges includes: Determining whether a change in a total value of the radar point cloud before and after adjusting the posture of the radar point cloud is no greater than a preset threshold; When a change in the total value of the radar point cloud before and after the posture of the radar point cloud is adjusted is not greater than the preset threshold, it is determined that the total value of the grid state of the radar point cloud has converged.

5. The method according to any one of claims 1 to 4, wherein The method further comprises: The size of the grid in the grid map is gradually reduced to calibrate the radar sensor with gradually increasing accuracy until the size of the grid in the grid map reaches a set size, and then the actual installation posture of the radar sensor in the preset coordinate system is obtained.

6. The method according to any one of claims 1 to 4, wherein There are multiple radar sensors; and the method further includes: The distribution differences of the posture distribution forms of the plurality of radar sensors in the initial design posture and the actual installation posture are compared, and the optimized postures of the plurality of radar sensors are determined with the goal of minimizing the distribution differences.

7. The method according to claim 6, wherein: Comparing the distribution differences of the pose distribution forms of the plurality of radar sensors in the initial design pose and in the actual installation pose, and determining the optimized poses of the plurality of radar sensors with the goal of minimizing the distribution differences, includes: The initial design poses and the actual installation poses of the plurality of radar sensors are obtained as vertices, and the pose errors between the initial design poses and the actual installation poses of the plurality of radar sensors are obtained as edges to construct a graph optimization model; and the optimized poses of the plurality of radar sensors are determined by minimizing the distribution error based on the graph optimization model.

8. A radar point cloud fusion method, comprising: Acquire image point clouds based on the calibrated multiple image sensors, and map the image point clouds to a preset coordinate system; generating a grid map in the preset coordinate system based on the image point cloud, and determining a grid state of each grid in the grid map according to a state in which each point in the image point cloud occupies each grid; collecting a radar point cloud based on an uncalibrated radar sensor, mapping the radar point cloud to the preset coordinate system according to an initial design pose of the radar sensor in the preset coordinate system, and obtaining a value of a grid state of a grid occupied by each point in the radar point cloud; According to the grid state value of the grid occupied by each point in the radar point cloud, the position and posture of the radar point cloud are adjusted in the preset coordinate system to fuse the radar point cloud with the image point cloud.

9. The method according to claim 8, wherein The multiple image sensors form a surround view image sensor array, and the image point cloud collected by the calibrated multiple image sensors is a bowl-shaped dense point cloud.

10. The method according to claim 8, wherein The image point cloud and the radar point cloud are both three-dimensional point clouds, the preset coordinate system is a three-dimensional space coordinate system, and the grid map is a three-dimensional grid map; The values ​​of the grid state of the grid occupied by each point in the radar point cloud are obtained by: Determining a value of a grid state at any point in the grid map by performing trilinear interpolation on the grid map; According to the position of each point in the radar point cloud in the grid map, a value of the grid state of each point in the radar point cloud is determined.

11. The method according to claim 8, wherein Adjusting the position and posture of the radar sensor in the preset coordinate system according to the grid state value of the grid occupied by each point in the radar point cloud includes: determining a total value of the grid state of the radar point cloud according to the grid state value of each point in the radar point cloud; determining a gradient of the radar point cloud according to a value of a grid state of each point in the radar point cloud; In the preset coordinate system, the pose of the radar point cloud is adjusted along the gradient direction of the radar point cloud so that the total value of the grid state of the radar point cloud converges.

12. The method according to claim 11, wherein Adjusting the pose of the radar point cloud along the gradient direction of the radar point cloud in the preset coordinate system so that the total value of the grid state of the radar point cloud converges includes: Determining whether a change in a total value of the radar point cloud before and after adjusting the posture of the radar point cloud is no greater than a preset threshold; When a change in the total value of the radar point cloud before and after the posture of the radar point cloud is adjusted is not greater than the preset threshold, it is determined that the total value of the grid state of the radar point cloud has converged.

13. The method according to any one of claims 8 to 12, wherein The method further comprises: The size of the grid in the grid map is gradually reduced to fuse the radar sensor with gradually improved accuracy, until the size of the grid in the grid map reaches a set size, and then it is determined that the radar point cloud and the image point cloud are completely fused.

14. A non-transitory computer-readable storage medium storing computer instructions for causing the computer to execute the method according to any one of claims 1 to 13.

Images