Sensor calibration method and device

Abstract

The embodiment of the invention discloses a sensor calibration method and device, and the method comprises the steps: sensing a same calibration object through a first sensor and a second sensor, and obtaining a second conversion coefficient of a second sensing coordinate system corresponding to the second sensor relative to a first sensing coordinate system corresponding to the first sensor; the relative positions of the first sensor and the second sensor are fixed. The calibration method comprises the following steps: sensing a calibration object through a first sensor to obtain a coordinate of the calibration object in a first sensing coordinate system, and determining a first conversion parameter of the first sensing coordinate system relative to a world coordinate system according to the coordinate of the calibration object in the world coordinate system and the coordinate of the calibration object in the first sensing coordinate system, and multi-sensor fusion sensing can be realized according to the first conversion parameter and the second conversion parameter. The relative positions of the first sensor and the second sensor are fixed, so that the second conversion parameter is fixed, the plurality of sensors do not need to be calibrated one by one subsequently, and the overall calibration process of the sensor system is simplified.

Description

Technical Field

[0001] The present invention relates to the field of intelligent transportation, and particularly to a method and device for sensor calibration. Background Art

[0002] With the continuous development of intelligent transportation systems, especially the continuous development of embedded technology and vehicle networking technology, intelligent driving vehicles have been more and more widely used in daily production and life. To ensure the stable operation of intelligent driving vehicles, the intelligent transportation system has put forward higher requirements for the accuracy, environmental adaptability and stability of vehicle-road collaborative roadside intelligent perception technology. As one of the key research points of the intelligent transportation system, roadside intelligent perception technology mainly uses sensors such as cameras, millimeter-wave radars and lidar to perform environmental perception, and sends the perceived result information to the vehicle end and the cloud through Ethernet, V2X, etc., for assisting the decision-making and scheduling of intelligent driving vehicles.

[0003] The current roadside perception technology is mainly divided into two schemes. The first is the environmental perception scheme based on a single sensor. Most of this scheme uses a single type of sensor to perceive the surrounding environment. For example, only a camera is used to collect image information, or only lidar is used to collect road point cloud information. The second is the environmental perception scheme based on multi-sensor fusion. This scheme uses two or more types of sensor groups to perform fusion perception on the surrounding environment, so as to improve the perception ability and expand the perception area. Compared with the single-sensor scheme, it has a wider perception range, richer perception information and more accurate perception results, and is the most used roadside perception scheme at present.

[0004] However, in the current environmental perception scheme based on multi-sensor fusion, the calibration method of multi-sensors, as a necessary link in fusion perception, has a cumbersome process and poor stability. Summary of the Invention

[0005] To solve the above technical problems, the embodiments of the present application provide a method and device for sensor calibration, which simplifies the calibration process and improves the stability of the environmental perception scheme.

[0006] The embodiments of the present application provide a method for sensor calibration, the method comprising:

[0007] Perceiving the same calibration object through a first sensor and a second sensor to obtain a second conversion parameter of a second sensing coordinate system corresponding to the second sensor relative to a first sensing coordinate system corresponding to the first sensor, wherein the relative positions of the first sensor and the second sensor are fixed;

[0008] Perceiving a calibration object through the first sensor to obtain detection information of the calibration object in the first sensing coordinate system;

[0009] Determine a first conversion parameter of the first sensing coordinate system relative to the world coordinate system according to the actual information of the calibration object in the world coordinate system and the detection information of the calibration object in the first sensing coordinate system;

[0010] The first conversion parameter is used to convert the first original detection information of the target object in the first sensing coordinate system into the first converted detection information of the target object in the world coordinate system, and in combination with the second conversion parameter, convert the second original detection information of the target object in the second sensing coordinate system into the second converted detection information of the target object in the world coordinate system, so as to determine the fused detection information of the target object according to the first converted detection information and the second converted detection information.

[0011] Optionally, the method further includes:

[0012] Sense the same calibration object through the first sensor and the third sensor to obtain a third conversion parameter of the third sensing coordinate system corresponding to the third sensor relative to the first sensing coordinate system, and the relative positions of the first sensor and the third sensor are fixed;

[0013] The third conversion parameter is used to, in combination with the first conversion parameter, convert the third original detection information of the target object in the third sensing coordinate system into the third converted detection information of the target object in the world coordinate system, so as to determine the fused detection information of the target object according to the first converted detection information, the second converted detection information, and the third converted detection information.

[0014] Optionally, the first conversion parameter and the second conversion parameter are used to determine a fourth conversion parameter of the second sensing coordinate system relative to the world coordinate system, and the fourth conversion parameter is used to convert the second original detection information of the target object in the second sensing coordinate system into the second converted detection information of the target object in the world coordinate system;

[0015] The first conversion parameter and the third conversion parameter are used to determine a fifth conversion parameter of the third sensing coordinate system relative to the world coordinate system, and the fifth conversion parameter is used to convert the third original detection information of the target object in the third sensing coordinate system into the third converted detection information of the target object in the world coordinate system.

[0016] Optionally, the first original detection information includes first original position information and first original motion information, the second original detection information includes second original position information and second original motion information, and the third original detection information includes third original position information and third original motion information.

[0017] Optionally, the first sensor includes a lidar, the second sensor is one of a camera and a millimeter-wave radar, and the third sensor is the other of the camera and the millimeter-wave radar.

[0018] Optionally, the method further includes:

[0019] Based on the longitude, latitude, and elevation of the calibration object, the actual information of the calibration object in the world coordinate system is calculated.

[0020] Optionally, the method further includes:

[0021] The first conversion parameter is corrected according to the correction matrix.

[0022] An embodiment of the present application provides a sensor calibration device, and the device includes:

[0023] A second conversion parameter determination unit, configured to sense the same calibration object through a first sensor and a second sensor, and obtain a second conversion parameter of a second sensing coordinate system corresponding to the second sensor relative to a first sensing coordinate system corresponding to the first sensor, where the relative positions of the first sensor and the second sensor are fixed;

[0024] A first sensing unit, configured to sense a calibration object through the first sensor to obtain detection information of the calibration object in the first sensing coordinate system;

[0025] A first conversion parameter determination unit, configured to determine a first conversion parameter of the first sensing coordinate system relative to the world coordinate system according to the actual information of the calibration object in the world coordinate system and the detection information of the calibration object in the first sensing coordinate system;

[0026] The first conversion parameter is used to convert first original detection information of a target object in the first sensing coordinate system into first converted detection information of the target object in the world coordinate system, and in combination with the second conversion parameter, convert second original detection information of the target object in the second sensing coordinate system into second converted detection information of the target object in the world coordinate system, so as to determine fusion detection information of the target object according to the first converted detection information and the second converted detection information.

[0027] Optionally, the device further includes:

[0028] A third conversion parameter determination unit, configured to sense the same calibration object through the first sensor and a third sensor, and obtain a third conversion parameter of a third sensing coordinate system corresponding to the third sensor relative to the first sensing coordinate system, where the relative positions of the first sensor and the third sensor are fixed;

[0029] The third conversion parameter is used to combine with the first conversion parameter to convert the third original detection information of the target object in the third sensing coordinate system into the third converted detection information of the target object in the world coordinate system, so as to determine the fused detection information of the target object according to the first converted detection information, the second converted detection information, and the third converted detection information.

[0030] Optionally, the first conversion parameter and the second conversion parameter are used to determine a fourth conversion parameter of the second sensing coordinate system relative to the world coordinate system, and the fourth conversion parameter is used to convert the second original detection information of the target object in the second sensing coordinate system into the second converted detection information of the target object in the world coordinate system;

[0031] The first conversion parameter and the third conversion parameter are used to determine a fifth conversion parameter of the third sensing coordinate system relative to the world coordinate system, and the fifth conversion parameter is used to convert the third original detection information of the target object in the third sensing coordinate system into the third converted detection information of the target object in the world coordinate system.

[0032] Optionally, the first original detection information includes first original position information and first original motion information, the second original detection information includes second original position information and second original motion information, and the third original detection information includes third original position information and third original motion information.

[0033] Optionally, the first sensor includes a lidar, the second sensor is one of a camera and a millimeter-wave radar, and the third sensor is the other of the camera and the millimeter-wave radar.

[0034] Optionally, the device further includes:

[0035] An actual information determination unit, configured to calculate the actual information of the calibration object in the world coordinate system according to the longitude, latitude, and elevation of the calibration object.

[0036] Optionally, the device further includes:

[0037] A correction unit, configured to correct the first conversion parameter according to a correction matrix.

[0038] Embodiments of the present application provide a sensor calibration method and device. By using a first sensor and a second sensor to sense the same calibration object, a second conversion coefficient of the second sensing coordinate system corresponding to the second sensor relative to the first sensing coordinate system corresponding to the first sensor can be obtained, and the relative positions of the first sensor and the second sensor are fixed. By using the first sensor to sense a calibration object, the coordinates of the calibration object in the first sensing coordinate system can be obtained. According to the coordinates of the calibration object in the world coordinate system and the coordinates of the calibration object in the first sensing coordinate system, the first conversion parameter of the first sensing coordinate system relative to the world coordinate system can be determined. In this way, according to the first conversion parameter, the first original detection information of the target object in the first sensing coordinate system can be converted into the first converted detection information of the target object in the world coordinate system. According to the first conversion parameter and the second conversion parameter, the second original detection information of the target object in the second sensing coordinate system can be converted into the second converted detection information of the target object in the world coordinate system. Furthermore, according to the first converted detection information and the second converted detection information, the fusion detection information of the target object can be determined, realizing multi-sensor fusion perception. During the calibration process of determining the first conversion parameter and the second conversion parameter, since the relative positions of the first sensor and the second sensor are fixed, the second conversion parameter is fixed. After the sensor system is installed, and when the position of the sensor system composed of the first sensor and the second sensor changes, only the first conversion parameter needs to be determined and updated to achieve the calibration of multiple sensors, without calibrating each sensor one by one, simplifying the overall calibration process of the sensor system, with less workload, good robustness, high accuracy, and improving the stability of the environmental perception scheme. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] To more clearly illustrate the technical solutions in the embodiments of the present application, the following will briefly introduce the drawings required for the description of the embodiments. Obviously, the drawings in the following description are only some embodiments recorded in the present application. For those of ordinary skill in the art, other drawings can also be obtained according to these drawings.

[0040] Figure 1 It is a flowchart of a sensor calibration method provided by an embodiment of the present application;

[0041] Figure 2 It is a schematic diagram of a calibration process provided by an embodiment of the present application;

[0042] Figure 3 It is an effect diagram of joint calibration of a laser point cloud and a camera image provided by an embodiment of the present application;

[0043] Figure 4 It is a structural block diagram of a sensor calibration device provided by an embodiment of the present application. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0044] To make the above objects, features, and advantages of the present application more obvious and understandable, the following will describe in detail the specific embodiments of the present application with reference to the accompanying drawings.

[0045] In the following description, many specific details are set forth to facilitate a thorough understanding of the present application. However, the present application may be implemented in other ways different from those described herein. Those skilled in the art can make similar generalizations without departing from the spirit of the present application. Therefore, the present application is not limited by the specific embodiments disclosed below.

[0046] As mentioned above, how to achieve accurate environmental perception through the fusion of multi-sensor information is the primary problem to be solved in intelligent transportation systems. In current environmental perception solutions based on multi-sensor fusion, the calibration method of multi-sensors, as an essential link in fusion perception, greatly affects the accuracy and stability of the final perception results. During the calibration process, the calibration of multi-sensors needs to be carried out separately, and the process is relatively cumbersome. Moreover, multiple sensors are independently deployed, and there is no fixed spatial geometric relationship between them. When the position or perspective of the sensor system changes due to reasons such as weather or vibration, the conversion parameters of each sensor need to be recalibrated, and the system stability is poor.

[0047] Based on this, the embodiments of the present application provide a sensor calibration method and device. By using a first sensor and a second sensor to sense the same calibration object, the second conversion parameter of the second sensing coordinate system corresponding to the second sensor relative to the first sensing coordinate system corresponding to the first sensor can be obtained. The relative positions of the first sensor and the second sensor are fixed. By using the first sensor to sense the calibration object, the coordinates of the calibration object in the first sensing coordinate system can be obtained. According to the coordinates of the calibration object in the world coordinate system and the coordinates of the calibration object in the first sensing coordinate system, the first conversion parameter of the first sensing coordinate system relative to the world coordinate system can be determined. In this way, according to the first conversion parameter, the first original detection information of the target object in the first sensing coordinate system can be converted into the first converted detection information of the target object in the world coordinate system. According to the first conversion parameter and the second conversion parameter, the second original detection information of the target object in the second sensing coordinate system can be converted into the second converted detection information of the target object in the world coordinate system. Furthermore, according to the first converted detection information and the second converted detection information, the fused detection information of the target object can be determined, realizing multi-sensor fusion perception. During the calibration process of determining the first conversion parameter and the second conversion parameter, since the relative positions of the first sensor and the second sensor are fixed, the second conversion parameter is fixed. After the sensor system is installed, and when the position of the sensor system composed of the first sensor and the second sensor changes, only the first conversion parameter needs to be determined and updated to achieve the calibration of multiple sensors, without the need to calibrate each sensor one by one, simplifying the overall calibration process of the sensor system, with less workload, good robustness, and high accuracy, improving the stability of the environmental perception solution.

[0048] The following will, with reference to the accompanying drawings, through embodiments, elaborate in detail on the specific implementation manners of a sensor calibration method and device provided by the embodiments of the present application.

[0049] Refer to Figure 1 As shown, it is a flowchart of a sensor calibration method provided by the embodiments of the present application, which may include the following steps.

[0050] S101, sense the same calibration object through a first sensor and a second sensor, and obtain the second conversion parameter of the second sensing coordinate system corresponding to the second sensor relative to the first sensing coordinate system corresponding to the first sensor, where the relative positions of the first sensor and the second sensor are fixed.

[0051] In the embodiments of this application, multi-sensor fusion perception can be achieved. Compared with the solution of a single sensor, it has a wider perception range, richer perception information, and more accurate perception results, and is currently the most widely used roadside perception solution. The multiple sensors can include, for example, lidar, cameras, millimeter-wave radars, etc. The multiple sensors are respectively denoted as the first sensor and the second sensor. When there are more sensors, a third sensor can also be included, and a fourth sensor can also be included, etc. Different sensors can be of the same type or different types. Sensors of the same type can have different perspectives, and the multiple sensors can include multiple cameras with different orientations.

[0052] In the multi-sensor fusion perception solution, lidar obtains information such as the position, speed, and heading angle of the target through the reflection of laser light, and has the advantages of high detection accuracy and stable position detection. However, there are also deficiencies that the quality of the laser point cloud is greatly affected by the environment, and the high hardware cost results in a limited number of lidars deployed in the system, requiring supplementary perception; cameras are generally used for the recognition and detection of targets, and are accurate in detecting the existence and category of targets, but the perception results are easily affected by ambient light, and the position detection accuracy is relatively low; millimeter-wave radars obtain the position and speed of the target by sending electromagnetic waves to the target and receiving the echo, can work all-weather, and are less affected by the environment. However, the point cloud density of millimeter-wave radars is small, and the detection accuracy is relatively lower than that of lidar. Each of the three types of sensors has its own advantages and disadvantages. In order to give full play to the advantages of each sensor and make up for the deficiencies, the current mainstream roadside perception system will, through calibration, convert the perception of multiple sensors of the target to the same coordinate system (such as the world coordinate system), so as to better apply the fusion algorithm and obtain more accurate perception results.

[0053] As an example, the roadside perception system can be a roadside perception system deployed in a port environment, with lidar as the main sensor. The first sensor includes lidar, the second sensor is one of a camera and a millimeter-wave radar, and the third sensor is the other of a camera and a millimeter-wave radar. For example, the second sensor is a camera and the third sensor is a millimeter-wave radar.

[0054] Among multiple sensors, each sensor can sense an object, and the sensing result is the detection information of the object in the coordinate system corresponding to the sensor. To perform multi-sensor fusion sensing, it is necessary to convert the detection information of the object in the coordinate system corresponding to the sensor (i.e., the detection information of the object in the detection range of the sensor) to the same coordinate system. Therefore, sensor calibration is required. By calibrating the sensor, the conversion relationship between the coordinate system corresponding to the sensor and the aforementioned same coordinate system can be determined, so as to perform the conversion of the sensing result subsequently. Among them, the detection information can include at least one of position information and motion information. The position information is, for example, coordinate information, and the coordinate information includes coordinate values on different coordinate axes (such as the coordinate value x of the x-axis and the coordinate value y of the y-axis). The motion information can include at least one of velocity information v and acceleration information a.

[0055] In the embodiments of the present application, the relative positions between at least two sensors among multiple sensors are determined during the design stage of the device. The relative positions include relative distance, relative angle, etc., that is, the relative positions of the first sensor and the second sensor are fixed, the relative positions of the first sensor and the third sensor are fixed, and so on. For example, the relative positions between sensors are fixed by brackets. In this way, the conversion parameters between each sensor can be determined in advance. During installation and calibration, it is not necessary to calibrate each sensor. Only a few or even one sensor needs to be calibrated, and then the conversion calculation can be performed according to the conversion parameters between each sensor and the calibrated sensor, avoiding the problem of cumbersome processes caused by calibrating each sensor separately, and greatly reducing the workload of on-site deployment and calibration work.

[0056] In the embodiments of the present application, the coordinate system corresponding to the first sensor is denoted as the first sensing coordinate system, the coordinate system corresponding to the second sensor is denoted as the second sensing coordinate system, the coordinate system corresponding to the third sensor is denoted as the third sensing coordinate system, and so on. Through the first sensor, the detection information of the object in the first sensing coordinate system can be detected. Through the second sensor, the detection information of the object in the second sensing coordinate system can be detected. Through the third sensor, the detection information of the object in the third sensing coordinate system can be detected.

[0057] When multiple sensors include a first sensor and a second sensor, the conversion parameters between the first sensor and the second sensor can be determined. In this way, the detection information in the first sensing coordinate system can be converted to the second sensing coordinate system according to the conversion parameters, and the detection information in the second coordinate system can also be converted to the first sensing coordinate system. Specifically, during implementation, the first sensor and the second sensor can sense the same calibration object. In this way, based on the detection information detected by the first sensor and the detection information detected by the second sensor, the second conversion parameter of the second sensing coordinate system relative to the first sensing coordinate system can be determined.

[0058] When multiple sensors include a first sensor, a second sensor, and a third sensor, the conversion parameters between the first sensor and the second sensor, and the conversion parameters between the first sensor and the third sensor can be determined. In this way, the detection information in the first sensor coordinate system, the second sensor coordinate system, and the third sensor coordinate system can be mutually converted according to the conversion parameters. Specifically, during implementation, the first conversion parameter can be determined by having the first sensor and the second sensor sense the same calibration object, and the same calibration object can be sensed by the first sensor and the third sensor. In this way, based on the detection information detected by the first sensor and the detection information detected by the third sensor, the third conversion parameter of the third sensor coordinate system relative to the first sensor coordinate system can be determined.

[0059] The conversion parameters between sensor coordinate systems can be determined in a laboratory environment. The aforementioned same calibration object can be a calibration board, and the calibration board has feature points. The detection information obtained by lidar detection is lidar point cloud, the detection information obtained by camera detection is camera image, and the detection information obtained by millimeter-wave radar detection is feature point cloud.

[0060] The conversion parameters can include a translation parameter T and a rotation parameter R. The translation parameter is, for example, a translation matrix, and the rotation parameter is, for example, a rotation matrix. The second conversion parameter includes a second translation parameter T1 and a second rotation parameter R1, and the third conversion parameter includes a third translation parameter T2 and a third rotation parameter R2. Denote the detection information in the second sensor coordinate system as G co1 (x, y, v, a, …) and G lo2 (x, y, v, a, …), denote the detection information in the third sensor coordinate system as G ro (x, y, v, a, …), and denote the detection information in the first sensor coordinate system as G lo (x, y, v, a, …). Then, the following relationship exists between the detection information and the conversion parameters:

[0061] G lo1 (x, y, v, a, …) = R1G co (x, y, v, a, …) + T1, (1)

[0062] G lo2 (x, y, v, a, …) = R2G ro (x, y, v, a, …) + T2, (2)

[0063] In this way, the second conversion parameter and the third conversion parameter can be determined through the detection information in the aforementioned formulas (1) and (2).

[0064] Reference Figure 2As shown in the figure, it is a schematic diagram of a calibration process provided by an embodiment of the present application. Taking the first sensor as a lidar, the second sensor as a camera, and the third sensor as a millimeter-wave radar as an example, the calibration board is sensed by the lidar and the camera to obtain a laser point cloud and a camera image. According to the detection information of the first feature point on the calibration board in the laser point cloud and the camera image, the second conversion parameter between the coordinate systems corresponding to the lidar and the camera is determined. The method for determining the second conversion parameter is the Zhang Zhengyou calibration method, referring to the relational expression in formula (1). Refer to Figure 3 As shown, it is an effect diagram of joint calibration of a laser point cloud and a camera image provided by an embodiment of the present application.

[0065] After that, the same area is sensed by the lidar and the millimeter-wave radar to obtain a laser point cloud and a feature point cloud. According to the detection information of the second feature point in the feature point cloud and the laser point cloud, the third conversion parameter between the coordinate systems corresponding to the lidar and the millimeter-wave radar is determined, referring to the relational tree in formula (2). During the process of determining the second conversion parameter and the third conversion parameter, multiple measurements and calibrations can be performed to reduce the measurement error and the error caused by equipment inconsistency.

[0066] S102, sense the calibration object through the first sensor to obtain the detection information of the calibration object in the first sensing coordinate system.

[0067] After determining the conversion parameter between the sensing coordinate systems, the sensor system can be deployed and installed on-site so that the sensor system can be put into use. After the deployment and installation, the calibration object can be sensed within the detection range of the first sensor. The calibration object can include a third feature point, and the detection information of the third feature point in the first sensing coordinate system can be determined from the sensing result. Refer to Figure 2 As shown. The third feature point is used as a calibration point, and the number can be multiple, such as 10 - 12. The coordinate information in the detection information of the third feature point in the first sensing coordinate system can be denoted as G lo (x, y).

[0068] S103, determine the first conversion parameter of the first sensing coordinate system relative to the world coordinate system according to the actual information of the calibration object in the world coordinate system and the detection information of the calibration object in the first sensing coordinate system.

[0069] After determining the detection information of the calibration object in the first sensing coordinate system, the first conversion parameter of the first sensing coordinate system relative to the world coordinate system can be determined according to the actual information of the calibration object in the world coordinate system and the detection information of the calibration object in the first sensing coordinate system. Refer to Figure 2 As shown. The first conversion parameter can include the first translation parameter T l and the first rotation parameter Rl 。

[0070] The coordinate information in the detection information of the third feature point in the first sensing coordinate system can be denoted as G lo (x, y), and the coordinate information in its actual information in the world coordinate system is denoted as G lw (x, y). Then, by matching the detection information and the actual information of the calibration object one by one, the first conversion parameter can be determined. Specifically, the following relationship exists among the detection information, the actual information, and the first conversion parameter of the calibration object:

[0071] G lw (x, y) = R l G lo (x, y) + T l , (3)

[0072] In this way, the first conversion parameter can be determined through the detection information and the actual information in the foregoing formula (3).

[0073] Specifically, before determining the first conversion parameter, based on the longitude, latitude, and elevation of the calibration object, the actual information G lw (x, y) of the calibration object in the world coordinate system can be calculated, as shown in the reference Figure 2 . For the third feature point, the GPS coordinates (actual longitude and latitude information) of the third feature point can be measured by a Real Time Kinematic (RTK) device. The longitude and latitude can be represented as L and B, the elevation is represented as H, N is the radius of curvature of the prime vertical of the earth ellipsoid model, e is the first eccentricity of the ellipsoid, a is the semi-major axis of the ellipsoid, and x and y are the coordinate values in the actual information G lw (x, y), then:

[0074] x = (N + H)cosBcosL, (4)

[0075] y = (N + H)cosBsinL, (5)

[0076]

[0077] In this way, through the foregoing formulas (4)-(6), the actual information of the specified object in the world coordinate system can be calculated based on its longitude, latitude, and elevation, and based on this, the first conversion parameter can be determined.

[0078] After determining the first conversion parameter, the second conversion parameter, or determining the first conversion parameter, the second conversion parameter, and the third conversion parameter, the calibration of the sensor system is achieved. If the sensor system moves after calibration, the first conversion parameter can be re-determined, or the first conversion parameter can be corrected through a correction matrix. The correction matrix can be determined through re-calibration, can be determined through on-site measurement, or can be determined through the comparison result of at least two detection results detected by at least one of the first sensor, the second sensor, and the third sensor before and after movement.

[0079] Since the second conversion parameter and the third conversion parameter are determined in the initial design stage, only the first conversion parameter needs to be determined during the calibration process. The overall system deployment and calibration process is simple and the workload is small. In addition, the calibration method has good robustness. Compared with a distributed system where the whole system needs to be re-calibrated after one sensor moves, this method only needs to re-determine the first conversion parameter. If the on-site calibration conditions are temporarily unavailable, a correction matrix can also be added to the whole system to correct the overall movement of the system based on the first conversion parameter, and still ensure the effectiveness of a certain sensing function; (3) High precision, avoiding the appearance of abnormal conversion matrices, and the relative position relationship of the sensors will not move during the deployment process, ensuring the high-precision sensing of the system. Therefore, this solution solves the problems of poor sensor environmental adaptability, functional safety, and service life in the traditional decentralized deployment solution of fusion perception devices, and is a multi-sensor integration calibration method for a multi-sensor fusion perception system with good stability and simple operation.

[0080] After the calibration of the sensor system is achieved, the conversion parameters can be used to fuse the sensing results. Taking the sensing result of the sensor system for a target object as an example, the target object is sensed by the first sensor and the second sensor, so that the target object has first original detection information in the first sensing coordinate system, and the target object has second original detection information in the second sensing coordinate system. If the sensor system includes a third sensor, the target object has third original detection information in the third sensing coordinate system. Among them, the first original detection information includes first original position information and first original motion information, the second original detection information includes second original position information and second original motion information, and the third original detection information includes third original position information and third original motion information.

[0081] In the embodiments of the present application, the first original detection information can be converted into the first converted detection information of the target object in the world coordinate system according to the first conversion parameter, and the second original detection information can be converted into the second converted detection information of the target object in the world coordinate system according to the first conversion parameter in combination with the second conversion parameter, so as to determine the fused detection information of the target object according to the first converted detection information and the second converted detection information. That is to say, the first conversion parameter is used to convert the first original detection information of the target object in the first sensing coordinate system into the first converted detection information of the target object in the world coordinate system, and in combination with the second conversion parameter, to convert the second original detection information of the target object in the second sensing coordinate system into the second converted detection information of the target object in the world coordinate system, so as to determine the fused detection information of the target object according to the first converted detection information and the second converted detection information.

[0082] Among them, converting the second original detection information into the second converted detection information of the target object in the world coordinate system according to the first conversion parameter in combination with the second conversion parameter can be specifically that the second original detection information is converted into the first intermediate detection information of the target object in the first sensing coordinate system according to the second conversion parameter, and then the first intermediate detection information is converted into the second converted detection information of the target object in the world coordinate system according to the first conversion parameter, or it can be specifically that the fourth conversion parameter is determined according to the first conversion parameter and the second conversion parameter, and then the second original detection information is converted into the second converted detection information of the target object in the world coordinate according to the fourth conversion parameter. That is to say, the first conversion parameter and the second conversion parameter are used to determine the fourth conversion parameter of the second sensing coordinate system relative to the world coordinate system, and the fourth conversion parameter is used to convert the second original detection information of the target object in the second sensing coordinate system into the second converted detection information of the target object in the world coordinate system.

[0083] In the embodiments of the present application, the third original detection information can also be converted into the third converted detection information of the target object in the world coordinate system according to the first conversion parameter in combination with the third conversion parameter, so as to determine the fused detection information of the target object according to the first converted detection information, the second converted detection information and the third converted detection information. That is to say, the third conversion parameter is used to combine with the first conversion parameter to convert the third original detection information of the target object in the third sensing coordinate system into the third converted detection information of the target object in the world coordinate system, so as to determine the fused detection information of the target object according to the first converted detection information, the second converted detection information and the third converted detection information.

[0084] Among them, according to the first conversion parameter in combination with the third conversion parameter, converting the third original detection information into the third converted detection information of the target object in the world coordinate system can be specifically that, according to the third conversion parameter, converting the third original detection information into the second intermediate detection information of the target object in the first sensing coordinate system, and then according to the first conversion parameter, converting the second intermediate detection information into the third converted detection information of the target object in the world coordinate system, or specifically, determining the fifth conversion parameter according to the first conversion parameter and the third conversion parameter, and then according to the fifth conversion parameter, converting the third original detection information into the third converted detection information of the target object in the world coordinate. That is to say, the first conversion parameter and the third conversion parameter are used to determine the fifth conversion parameter of the third sensing coordinate system relative to the world coordinate system, and the fifth conversion parameter is used to convert the third original detection information of the target object in the third sensing coordinate system into the third converted detection information of the target object in the world coordinate system.

[0085] The first original detection information is denoted as G lo (x, y, v, a, …), and the first converted detection information is denoted as G lw (x, y, v, a, …). The first conversion parameter includes the first translation parameter T l and the first rotation parameter R l ; the second original detection information is denoted as G co (x, y, v, a, …), and the second converted detection information is denoted as G cw (x, y, v, a, …). The fourth conversion parameter includes the fourth translation parameter T c and the fourth rotation parameter R c ; the third original detection information is denoted as G ro (x, y, v, a, …), and the third converted detection information is denoted as G rw (x, y, v, a, …). The fifth conversion parameter includes the fifth translation parameter T r and the fifth rotation parameter R r , then the conversion of the monitoring information can be carried out through the following formula:

[0086] G lw (x, y, v, a, …) = R l G lo (x, y, v, a, …) + T l , (7)

[0087] G cw (x, y, v, a, …) = R c G co (x, y, v, a, …) + T c , (8)

[0088] Grw (x, y, v, a, …) = R r G ro (x, y, v, a, …) + T r , (9)

[0089] In this way, the detection information of each sensor can be converted into a unified world coordinate system.

[0090] That is, assuming that G is the vector composed of the target set sensed by the sensor, the target vector in the original coordinate system of the sensor needs to be transformed into the world coordinate system through the rotation matrix R and the translation matrix T, G lw , G cw , G rw are the target vectors of the lidar, camera, and millimeter-wave radar in the world coordinate system respectively, R l , T l , R c , T c , R r , T r are the corresponding rotation and translation matrices. The relative distances of the sensors are determined during the device design stage, that is, R l , T l , R c , T c , R r , T r The relative relationship has been determined. Therefore, during installation and calibration, only one sensor needs to be calibrated, and the parameters of other sensors can be calculated through the conversion parameters between the sensing coordinate systems, avoiding the need to perform the calculations of formulas (4)-(6) for each sensor's calibration process, and the calibration results are less affected by the environment, reducing calibration errors.

[0091] In the fusion algorithm, Kalman filters, factor graph-based, non-maximum suppression, and other fusion methods will be used to achieve the fusion of different converted detection information. The fusion process can refer to the following two relational expressions:

[0092] G fw (x, y, v, a, …) = f(G lw (x, y, v, a, …), G cw (x, y, v, a, …)), (10)

[0093] G fw (x, y, v, a, …) = f(G lw (x, y, v, a, …), G cw (x, y, v, a, …), G rw (x, y, v, a, …)), (11)

[0095] Among them, f is the fusion algorithm, and G fw is the fusion detection result. It can be seen from this that the fusion detection result is strongly correlated with the conversion parameter. On the premise that the accuracy and reliability of the conversion parameter are improved, the accuracy and reliability of the fusion detection result are also correspondingly improved.

[0096] In the actual operation process, the fusion detection result can also be converted into longitude and latitude for output, and the conversion method can refer to formula (3).

[0097] The embodiment of the present application provides a sensor calibration method. By using a first sensor and a second sensor to sense the same calibration object, the second conversion coefficient of the second sensing coordinate system corresponding to the second sensor relative to the first sensing coordinate system corresponding to the first sensor can be obtained, and the relative positions of the first sensor and the second sensor are fixed. By using the first sensor to sense the calibration object, the coordinates of the calibration object in the first sensing coordinate system can be obtained. According to the coordinates of the calibration object in the world coordinate system and the coordinates of the calibration object in the first sensing coordinate system, the first conversion parameter of the first sensing coordinate system relative to the world coordinate system can be determined. In this way, according to the first conversion parameter, the first original detection information of the target object in the first sensing coordinate system can be converted into the first converted detection information of the target object in the world coordinate system. According to the first conversion parameter and the second conversion parameter, the second original detection information of the target object in the second sensing coordinate system can be converted into the second converted detection information of the target object in the world coordinate system. Furthermore, according to the first converted detection information and the second converted detection information, the fusion detection information of the target object can be determined, realizing multi-sensor fusion perception. In the calibration process of determining the first conversion parameter and the second conversion parameter, since the relative positions of the first sensor and the second sensor are fixed, the second conversion parameter is fixed. After the sensor system is installed, and when the position of the sensor system composed of the first sensor and the second sensor changes, only the first conversion parameter needs to be determined and updated to achieve the calibration of multiple sensors, without calibrating each sensor one by one, simplifying the overall calibration process of the sensor system, with small workload, good robustness, high precision, and improving the stability of the environmental perception scheme.

[0098] Based on the above sensor calibration method, the embodiment of the present application further provides a sensor calibration device, referring to Figure 4 As shown, it is a structural block diagram of a sensor calibration device provided by the embodiment of the present application, which may include:

[0099] A second conversion parameter determination unit 110, configured to sense the same calibration object through a first sensor and a second sensor, and obtain a second conversion parameter of the second sensing coordinate system corresponding to the second sensor relative to the first sensing coordinate system corresponding to the first sensor, where the relative positions of the first sensor and the second sensor are fixed;

[0100] The first sensing unit 120 is configured to sense a calibration object through a first sensor to obtain detection information of the calibration object in the first sensing coordinate system;

[0101] The first conversion parameter determination unit 130 is configured to determine a first conversion parameter of the first sensing coordinate system relative to the world coordinate system according to the actual information of the calibration object in the world coordinate system and the detection information of the calibration object in the first sensing coordinate system;

[0102] The first conversion parameter is used to convert first original detection information of a target object in the first sensing coordinate system into first converted detection information of the target object in the world coordinate system, and in combination with the second conversion parameter, convert second original detection information of the target object in the second sensing coordinate system into second converted detection information of the target object in the world coordinate system, so as to determine fused detection information of the target object according to the first converted detection information and the second converted detection information.

[0103] Optionally, the device further includes:

[0104] A third conversion parameter determination unit, configured to sense the same calibration object through the first sensor and a third sensor to obtain a third conversion parameter of the third sensing coordinate system corresponding to the third sensor relative to the first sensing coordinate system, where the relative positions of the first sensor and the third sensor are fixed;

[0105] The third conversion parameter is used to, in combination with the first conversion parameter, convert third original detection information of the target object in the third sensing coordinate system into third converted detection information of the target object in the world coordinate system, so as to determine fused detection information of the target object according to the first converted detection information, the second converted detection information, and the third converted detection information.

[0106] Optionally, the first conversion parameter and the second conversion parameter are used to determine a fourth conversion parameter of the second sensing coordinate system relative to the world coordinate system, and the fourth conversion parameter is used to convert second original detection information of the target object in the second sensing coordinate system into second converted detection information of the target object in the world coordinate system;

[0107] The first conversion parameter and the third conversion parameter are used to determine a fifth conversion parameter of the third sensing coordinate system relative to the world coordinate system, and the fifth conversion parameter is used to convert third original detection information of the target object in the third sensing coordinate system into third converted detection information of the target object in the world coordinate system.

[0108] Optionally, the first original detection information includes first original position information and first original motion information, the second original detection information includes second original position information and second original motion information, and the third original detection information includes third original position information and third original motion information.

[0109] Optionally, the first sensor includes a lidar, the second sensor is one of a camera and a millimeter-wave radar, and the third sensor is the other of the camera and the millimeter-wave radar.

[0110] Optionally, the apparatus further includes:

[0111] An actual information determination unit, configured to calculate actual information of the calibration object in a world coordinate system according to the longitude, latitude, and elevation of the calibration object.

[0112] Optionally, the apparatus further includes:

[0113] A correction unit, configured to correct the first conversion parameter according to a correction matrix.

[0114] Through the description of the above embodiments, those skilled in the art can clearly understand that all or part of the steps in the above embodiment methods can be implemented by means of software plus a general hardware platform. Based on such an understanding, the technical solution of the present application can be embodied in the form of a software product, and the computer software product can be stored in a storage medium, such as a read-only memory (ROM) / RAM, a magnetic disk, an optical disc, etc., including several instructions for causing a computer device (which may be a personal computer, a server, or a network communication device such as a router) to execute the methods described in each embodiment or some parts of the embodiments of the present application.

[0115] It should be noted that the embodiments in this specification are described in a progressive manner, and the key points of each embodiment are the differences from other embodiments. The same and similar parts between the embodiments can be referred to each other. For the apparatus disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple, and the relevant parts can be referred to the description of the method part.

[0116] It should also be noted that in this text, relational terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the term "comprising", "including" or any other variant thereof is intended to cover non-exclusive inclusion, such that a process, method, article or device comprising a series of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article or device. Without further limitation, an element defined by the statement "comprising an..." does not exclude the presence of additional identical elements in the process, method, article or device comprising the said element.

[0117] The above are only the preferred embodiments of the present application. Although the present application has been disclosed above with preferred embodiments, it is not intended to limit the present application. Any person skilled in the art can make many possible changes and modifications to the technical solution of the present application, or modify it into equivalent embodiments with equivalent changes, without departing from the scope of the technical solution of the present application. Therefore, any simple modification, equivalent change and modification made to the above embodiments according to the technical essence of the present application without departing from the content of the technical solution of the present application still fall within the scope of protection of the technical solution of the present application.

Claims

1. A sensor calibration method, characterized in that, The method includes: Perceiving the same calibration object through a first sensor and a second sensor to obtain a second conversion parameter of a second sensing coordinate system corresponding to the second sensor relative to a first sensing coordinate system corresponding to the first sensor, wherein the relative positions of the first sensor and the second sensor are fixed; Perceiving a calibration object through the first sensor to obtain detection information of the calibration object in the first sensing coordinate system; Determining a first conversion parameter of the first sensing coordinate system relative to the world coordinate system according to the actual information of the calibration object in the world coordinate system and the detection information of the calibration object in the first sensing coordinate system; The first conversion parameter is used to convert first original detection information of a target object in the first sensing coordinate system into first converted detection information of the target object in the world coordinate system, and in combination with the second conversion parameter, convert second original detection information of the target object in the second sensing coordinate system into second converted detection information of the target object in the world coordinate system, so as to determine fused detection information of the target object according to the first converted detection information and the second converted detection information.

2. The method according to claim 1, wherein The method further includes: Perceiving the same calibration object through the first sensor and a third sensor to obtain a third conversion parameter of a third sensing coordinate system corresponding to the third sensor relative to the first sensing coordinate system, wherein the relative positions of the first sensor and the third sensor are fixed; The third conversion parameter is used to, in combination with the first conversion parameter, convert third original detection information of the target object in the third sensing coordinate system into third converted detection information of the target object in the world coordinate system, so as to determine fused detection information of the target object according to the first converted detection information, the second converted detection information, and the third converted detection information.

3. The method according to claim 2, wherein The first conversion parameter and the second conversion parameter are used to determine a fourth conversion parameter of the second sensing coordinate system relative to the world coordinate system, and the fourth conversion parameter is used to convert second original detection information of the target object in the second sensing coordinate system into second converted detection information of the target object in the world coordinate system; The first conversion parameter and the third conversion parameter are used to determine a fifth conversion parameter of the third sensing coordinate system relative to the world coordinate system, and the fifth conversion parameter is used to convert third original detection information of the target object in the third sensing coordinate system into third converted detection information of the target object in the world coordinate system.

4. The method according to claim 2, wherein The first original detection information includes first original position information and first original motion information, the second original detection information includes second original position information and second original motion information, and the third original detection information includes third original position information and third original motion information.

5. The method according to claim 2, wherein The first sensor includes a lidar, the second sensor is one of a camera and a millimeter-wave radar, and the third sensor is the other of the camera and the millimeter-wave radar.

6. The method according to any one of claims 1-5, characterized in that, The method further includes: Based on the longitude, latitude and elevation of the calibration object, the actual information of the calibration object in the world coordinate system is calculated.

7. The method according to any one of claims 1-5, characterized in that, The method further includes: Correcting the first conversion parameter according to the correction matrix.

8. A sensor calibration device, characterized in that, The device includes: A second conversion parameter determination unit, configured to sense the same calibration object through a first sensor and a second sensor, and obtain a second conversion parameter of a second sensing coordinate system corresponding to the second sensor relative to a first sensing coordinate system corresponding to the first sensor, wherein the relative positions of the first sensor and the second sensor are fixed; A first sensing unit, configured to sense a calibration object through a first sensor to obtain detection information of the calibration object in the first sensing coordinate system; A first conversion parameter determination unit, configured to determine a first conversion parameter of the first sensing coordinate system relative to the world coordinate system according to the actual information of the calibration object in the world coordinate system and the detection information of the calibration object in the first sensing coordinate system; The first conversion parameter is used to convert first original detection information of a target object in the first sensing coordinate system into first converted detection information of the target object in the world coordinate system, and in combination with the second conversion parameter, convert second original detection information of the target object in the second sensing coordinate system into second converted detection information of the target object in the world coordinate system, so as to determine fused detection information of the target object according to the first converted detection information and the second converted detection information.

9. The device according to claim 8, wherein The device further includes: A third conversion parameter determination unit, configured to sense the same calibration object through the first sensor and a third sensor, and obtain a third conversion parameter of a third sensing coordinate system corresponding to the third sensor relative to the first sensing coordinate system, wherein the relative positions of the first sensor and the third sensor are fixed; The third conversion parameter is used to convert third original detection information of the target object in the third sensing coordinate system into third converted detection information of the target object in the world coordinate system in combination with the first conversion parameter, so as to determine fused detection information of the target object according to the first converted detection information, the second converted detection information and the third converted detection information.

10. The device according to claim 9, wherein, The first conversion parameter and the second conversion parameter are used to determine a fourth conversion parameter of the second sensing coordinate system relative to the world coordinate system, and the fourth conversion parameter is used to convert second original detection information of the target object in the second sensing coordinate system into second converted detection information of the target object in the world coordinate system; The first conversion parameter and the third conversion parameter are used to determine a fifth conversion parameter of the third sensing coordinate system relative to the world coordinate system, and the fifth conversion parameter is used to convert third original detection information of the target object in the third sensing coordinate system into third converted detection information of the target object in the world coordinate system.

Images