A method and apparatus for calibrating sensor data

Abstract

The specification discloses a sensor data calibration method and device, image data, radar data and inertial data at multiple moments during movement of a test device are acquired, a motion trajectory of the test device is determined according to the inertial data and the image data, a movement speed of the test device under an image reference system is determined according to the motion trajectory, a Doppler speed during movement of the test device is determined according to the radar data, and the Doppler speed and the movement speed of the test device under the image reference system are registered to calibrate the sensor data. The scheme can also be applied when the collection ranges of the radar sensor and the image sensor do not overlap, and does not require a calibration object that moves in an overlapping field of view, so that the calibration process is more convenient and the calibration efficiency of the sensor data is improved.

Description

Technical Field

[0001] This specification relates to the field of computer technology, and in particular to a method and apparatus for calibrating sensor data. Background Technology

[0002] Currently, with the development of autonomous driving technology, the driving safety of autonomous vehicles is becoming increasingly important. Obstacle detection and classification methods based on the fusion of image and radar data are widely used in obstacle detection and classification scenarios due to their relatively accurate detection and classification results. However, the prerequisite for image and radar data fusion is the calibration of both the radar and image sensors. Summary of the Invention

[0003] This specification provides a method and apparatus for calibrating sensor data, in order to partially solve the aforementioned problems existing in the prior art.

[0004] The following technical solution is adopted in this specification:

[0005] This specification provides a method for calibrating sensor data, including:

[0006] Acquire sensor data at multiple moments during the movement of the test equipment, wherein the sensor data includes at least image data, radar data, and inertial data;

[0007] Based on the inertial data and the image data, the motion trajectory of the test device is determined, and based on the motion trajectory, the moving speed of the test device in the image reference frame is determined.

[0008] Based on the radar data, the Doppler velocity during the movement of the test equipment is determined, and the Doppler velocity and the movement velocity of the test equipment in the image reference frame are registered to calibrate the sensor data.

[0009] Optionally, the motion trajectory of the test device is determined based on the inertial data and the image data, specifically including:

[0010] Based on the inertial data and the image data, determine the angular velocity, acceleration, and observation position of the image markers at the multiple moments in the inertial reference frame for the test equipment.

[0011] The trajectory of the test device in the world reference frame is solved based on the angular velocity, acceleration, and position of the image markers at each time point.

[0012] Optionally, based on the angular velocity, acceleration, and observation position of the image markers corresponding to the multiple moments, the trajectory of the test equipment in the world reference frame is solved, specifically including:

[0013] Based on the motion trajectory of the test device in the world reference frame, determine the first parameters to be solved for the plurality of times respectively, and the first parameters are used to solve the motion trajectory;

[0014] For each of the plurality of moments, based on the first parameter to be solved corresponding to that moment, the transformation relationship between the world reference frame and the inertial reference frame is determined, and the estimated angular velocity, estimated acceleration, and estimated position of the image markers of the test device in the inertial reference frame are determined;

[0015] Using the angular velocity and estimated angular velocity, acceleration and estimated acceleration corresponding to that moment, and the fact that the observed position and estimated position of the image marker are the same, constraints are constructed to solve for the motion trajectory.

[0016] Optionally, based on the first parameter to be solved at that moment, the transformation relationship between the world reference frame and the inertial reference frame is determined, as well as the estimated angular velocity, estimated acceleration, and estimated position of the image markers of the test equipment in the inertial reference frame are determined, specifically including:

[0017] Based on the pose of the test device in the first parameter to be solved at that moment, the transformation relationship between the world reference frame and the inertial reference frame to be solved at that moment is determined, wherein the first parameter includes the pose of the test device, acceleration offset, angular velocity offset, and observation position of image markers;

[0018] Based on the transformation relationship to be solved, the angular velocity offset to be solved in the first parameter, the acceleration offset to be solved, and the observation position of the image marker to be solved, the estimated acceleration, estimated angular velocity, and estimated position of the image marker of the test equipment are determined respectively.

[0019] Optionally, based on the motion trajectory, the moving speed of the test device in the image reference frame is determined, specifically including:

[0020] Based on the motion trajectory, determine the velocity of the test device at each of the multiple moments in the world reference frame;

[0021] Based on the pose of the test device at the multiple moments, determine the transformation relationship between the world reference frame and the inertial reference frame corresponding to the multiple moments respectively;

[0022] Based on the velocity of the test device at multiple moments in the world reference frame, the transformation relationship between the world reference frame and the inertial reference frame corresponding to the multiple moments, and the preset transformation relationship between the inertial reference frame and the image reference frame, the moving speed of the test device at the multiple moments in the image reference frame is determined.

[0023] Optionally, the Doppler velocity and the moving velocity of the test device in the image reference frame are registered to calibrate the sensor data, specifically including:

[0024] For each of the plurality of moments, the Doppler velocity in each direction component at that moment is determined based on the acquired radar data and the preset direction components.

[0025] Based on the moving speed of the test device in the image reference frame at that moment, the calibration relationship between the radar reference frame and the image reference frame to be solved, and the preset directional components, the moving speed of the test device to be solved in each directional component is determined.

[0026] The moving velocity and Doppler velocity in each directional component are registered, and the calibration relationship is calculated to calibrate the sensor data.

[0027] Optionally, based on the moving speed of the test device in the image reference frame at that moment, the calibration relationship between the radar reference frame and the image reference frame to be solved, and the preset directional components, the moving speed of the test device to be solved in each directional component is determined, specifically including:

[0028] Based on the unsolved time difference between the internal clocks of the radar sensor and the image sensor set on the test equipment, the unsolved moving speed of the test equipment in the image reference frame at that moment is determined, and the unsolved moving speed includes the unsolved time difference;

[0029] Based on the unsolved moving speed of the test device in the image reference frame at that moment, and the unsolved transformation relationship between the radar reference frame and the image reference frame, the unsolved moving speed of the test device in the radar reference frame is determined.

[0030] Based on the preset directional components, determine the moving speed of the test device in each directional component.

[0031] The calibration relationship is calculated by registering the unsolved moving velocities and Doppler velocities in each directional component, and then solving for the calibration relationship. Specifically, this includes:

[0032] The moving velocity and Doppler velocity in each directional component are registered, and the transformation relationship and time difference are calculated as the calibration relationship.

[0033] Optionally, the method further includes:

[0034] Based on the established calibration relationship and the collected sensor data, the pose difference of the test device in the image reference system and the radar reference system at each time point is determined, and it is determined whether the pose difference is greater than a preset error threshold.

[0035] If so, then the sensor data needs to be calibrated, and the sensor data is stored;

[0036] If not, then the sensor data does not require calibration.

[0037] This specification provides a sensor data calibration device, comprising:

[0038] The acquisition module is used to acquire sensor data at multiple moments during the movement of the test equipment. The sensor data includes at least image data, radar data, and inertial data.

[0039] The trajectory determination module is used to determine the motion trajectory of the test device based on the inertial data and the image data, and to determine the moving speed of the test device in the image reference frame based on the motion trajectory.

[0040] The calibration module is used to determine the Doppler velocity of the test equipment during its movement based on the radar data, and to register the Doppler velocity with the moving velocity of the test equipment in the image reference frame to calibrate the sensor data.

[0041] This specification provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the above-described method for calibrating sensor data.

[0042] The above-mentioned technical solutions adopted in this specification can achieve the following beneficial effects:

[0043] In the sensor data calibration method provided in this specification, image data, radar data, and inertial data at multiple moments during the movement of the test equipment are acquired. Based on the inertial data and image data, the motion trajectory of the test equipment is determined. Based on the motion trajectory, the moving speed of the test equipment in the image reference frame is determined. Then, based on the radar data, the Doppler velocity during the movement of the test equipment is determined. The Doppler velocity and the moving speed of the test equipment in the image reference frame are registered to calibrate the sensor data.

[0044] As can be seen from the above method, this method and scheme can also be applied when the acquisition ranges of radar sensors and image sensors do not overlap, and no calibration object needs to be moved within the overlapping field of view, making the calibration process more convenient and improving the calibration efficiency for determining sensor data. Attached Figure Description

[0045] The accompanying drawings, which are included to provide a further understanding of this specification and form part of this specification, illustrate exemplary embodiments and are used to explain this specification, but do not constitute an undue limitation thereof. In the drawings:

[0046] Figure 1 This is a flowchart illustrating the calibration method for the sensor data provided in this manual.

[0047] Figure 2 This is a schematic diagram of the sensor calibration scenario provided in this manual;

[0048] Figure 3 A schematic diagram of the calibration device for the sensor data provided in this manual;

[0049] Figure 4 The corresponding information provided in this specification Figure 1 A schematic diagram of an electronic device. Detailed Implementation

[0050] To make the objectives, technical solutions, and advantages of this specification clearer, the technical solutions of this specification will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of this specification, and not all of them. Based on the embodiments in this specification, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this specification.

[0051] In the field of sensor data calibration, commonly used sensor data calibration methods are based on the fact that the acquisition ranges of image sensors and radar sensors overlap.

[0052] Specifically, the radar sensor and image sensor are first kept stationary while the calibration object is moved. Then, based on the radar data collected during the movement of the calibration object, the first position of the calibration object at each moment is determined, and based on the collected image data, the second position of the calibration object at each moment is determined. Finally, for each moment, the calibration parameters of the radar sensor and image sensor are determined according to the same constraints as the first and second positions.

[0053] However, existing technologies are based on the overlap between the acquisition ranges of image sensors and radar sensors. If there is no overlap between the acquisition ranges of image sensors and radar sensors, it will be impossible to determine the calibration parameters of image sensors and radar sensors in autonomous driving equipment, resulting in poor calibration efficiency of existing technologies.

[0054] Therefore, a new method for calibrating sensor data is urgently needed.

[0055] The technical solutions provided in the various embodiments of this specification are described in detail below with reference to the accompanying drawings.

[0056] Figure 1 This is a flowchart illustrating the calibration method for the sensor data provided in this manual, which specifically includes the following steps:

[0057] S100: Acquire sensor data at multiple moments during the movement of the test equipment, wherein the sensor data includes at least image data, radar data, and inertial data.

[0058] Unlike existing technologies that rely on overlapping areas between the image and radar sensor acquisition ranges and use moving calibration objects within these overlapping areas for calibration, this specification provides a novel sensor data calibration method. This method eliminates the need for overlapping areas between the image and radar sensors. Instead, both sensors are placed within a test device, and the device is moved to determine image, radar, and inertial data at multiple time points. The calibration relationships between these sensor data are then determined based on the sensor data at these multiple time points.

[0059] Based on this, sensor data at multiple moments during the movement of the test equipment can be acquired first. This sensor data includes image data, radar data, and inertial data.

[0060] In one or more embodiments provided in this specification, during the operation of the autonomous driving device, the device can acquire sensor data at a preset frequency. This sensor data is required for calibrating the relationship between the sensor data and includes at least image data, inertial data, and radar data. Alternatively, the testing device can send the acquired calibration data to a server, which will then perform subsequent steps to determine the calibration relationship between the radar sensor and the image sensor. For ease of description, the following explanation will use the calibration process of the sensor data performed by the testing device as an example.

[0061] Specifically, the testing equipment can acquire image data, inertial data, and radar data it collects. This testing equipment can be an unmanned vehicle, a manned vehicle, or a handheld device, allowing for controlled movement or handheld movement of the testing equipment, while simultaneously collecting sensor data during the movement.

[0062] Furthermore, in this specification, the sensor data calibration method is applied to a scenario where a calibration object is placed on the ground, and the test equipment is moved under control. The image sensor acquires the position of the calibration object, the inertial sensor determines inertial data at multiple moments, and the radar sensor determines Doppler velocity at multiple moments. For example... Figure 2 As shown.

[0063] Figure 2 This is a schematic diagram of the sensor calibration scenario provided in this manual. In the diagram, the white cube represents the test equipment. The three gray cubes mounted on the white cube are the radar sensor, inertial sensor, and image sensor, respectively. Image markers are fixed on the ground. The image sensor can acquire image data including the markers. The radar sensor can acquire Doppler velocities in various directions through the Doppler effect of the wall. The inertial sensor can acquire inertial data corresponding to multiple moments. The image markers can be QR codes, checkerboard patterns, etc. The test equipment, radar sensor, image sensor, and inertial sensor are all simplified forms; their specific forms and fixing methods can be set as needed, and this manual does not impose any restrictions on them.

[0064] It should be noted that the chrome plating times mentioned above in this specification are multiple consecutive times, so as to ensure that the motion trajectory of the test equipment can be determined based on the acquired inertial data and image data, without needing to acquire sensor data at every moment during the movement of the test equipment.

[0065] S102: Determine the motion trajectory of the test device based on the inertial data and the image data, and determine the moving speed of the test device in the image reference frame based on the motion trajectory.

[0066] In one or more embodiments provided in this specification, during the movement of the test equipment, its trajectory in the image reference frame and its trajectory in the radar reference frame are actually the same. Therefore, if the trajectory of the test equipment in the image reference frame and its trajectory in the radar reference frame are registered, the calibration relationship between the image reference frame and the radar reference frame can be determined.

[0067] Based on this, the testing equipment can determine its own trajectory by acquiring inertial and image data.

[0068] The acquired inertial and image data correspond to multiple moments. The purpose is to determine the trajectory of the test device in the world reference frame based on the inertial and image data acquired at multiple moments during the device's movement. This trajectory corresponds to the inertial and image data at multiple moments during the device's movement; that is, the determined trajectory is not the entire trajectory of the device's movement, but rather a continuous trajectory determined by sensor data acquired at multiple moments during the device's movement.

[0069] Furthermore, while determining the trajectory of the test equipment in the radar reference frame based on radar data requires a large amount of computation and is difficult, determining the Doppler velocity of the test equipment based on the Doppler effect of the acquired radar data requires less computation and is less difficult. Therefore, the test equipment can determine its own Doppler velocity during its movement and its own movement velocity in the image reference frame. By registering the Doppler velocity and the movement velocity, a relatively accurate calibration relationship can be obtained.

[0070] Based on this, the testing equipment can determine its moving speed in the image reference frame according to the determined motion trajectory.

[0071] Specifically, the testing equipment can determine the displacement between each adjacent moment in multiple moments based on the determined motion trajectory, and then determine the velocity corresponding to each moment based on the displacement.

[0072] Furthermore, the steps for determining the motion trajectory of the testing equipment described above can be performed as follows:

[0073] Specifically, the testing equipment can determine the angular velocity, acceleration, and observation position of image markers at multiple moments in the inertial reference frame based on the acquired inertial data and image data.

[0074] Then, the test equipment can solve for the trajectory of the test equipment in the world reference frame based on the angular velocity, acceleration and the observation position of the image markers at multiple times.

[0075] Finally, based on the calculated motion trajectory, the testing device can determine its own motion trajectory in the world reference frame.

[0076] In addition, since the trajectory of an object can be determined by the pose of the object at multiple times, the testing device can determine its own trajectory based on the pose at multiple times.

[0077] Specifically, firstly, since the pose of the test device at multiple moments is unknown, the test device can determine the first parameter to be solved at multiple moments based on its pose at multiple moments and its motion trajectory to be solved in the world reference frame.

[0078] Then, for each of the multiple moments, the testing device can determine the transformation relationship between the world reference frame and the inertial reference frame based on the first parameter to be solved at that moment, as well as the angular velocity and acceleration of the testing device in the inertial reference frame, and determine the estimated angular velocity, estimated acceleration, and estimated position of the image markers of the testing device in the inertial reference frame.

[0079] Finally, the server can construct constraints based on the angular velocity and estimated angular velocity, acceleration and estimated acceleration at the given moment, and the observed and estimated positions of the image markers being the same, and solve for the motion trajectory of the test equipment.

[0080] Furthermore, since the estimated angular velocity, estimated acceleration, and estimated position of the image markers are all in the inertial reference frame, the test equipment needs to determine the estimated angular velocity, estimated acceleration, and estimated position of the image markers at multiple moments based on the conversion relationship between the world reference frame and the inertial reference frame.

[0081] Specifically, the testing equipment can determine the transformation relationship between the world reference frame and the inertial reference frame at that moment based on its position and attitude in the first parameter to be solved. Then, based on the transformation relationship, the angular velocity offset, acceleration offset, and the observation position of the image marker in the first parameter, the predicted acceleration, predicted angular velocity, and predicted position of the image marker in the inertial reference frame are determined respectively.

[0082] Furthermore, the steps for calculating the motion trajectory of the test equipment described above can be determined in the following manner:

[0083] Taking the modeling of the motion trajectory of the test equipment using the B-spline algorithm as an example, the first parameter can be constructed first: x = [x q T ,x p T ,b a T ,b w T ,l T ]. Among them, for each of the multiple time points, x q For the attitude of the test device in the world reference frame at that moment, x qb represents the position of the testing equipment in the world reference frame at that moment. a b is the acceleration bias corresponding to that moment. w Let l be the angular velocity offset at that moment, and l be the observation position of the image marker at that moment. The observation position of the image marker can be characterized by the positions of various image feature points within the image marker, such as the position of the center point or the position of the edge points. The image marker can be of various types, such as a checkerboard or QR code, and its shape can be various shapes, such as triangles, rectangles, circles, or polygons. The specific shape and type of the image marker can be set as needed.

[0084] Therefore, the predicted acceleration can be determined as follows: Where p is the estimate and k is the k-th time step. Let k be the predicted acceleration at time k. Let be the rotation matrix from the world reference frame to the inertial reference frame at time k. For the two derivatives of the translation matrix from the world reference frame to the inertial reference value at time k, that is, the acceleration of the test equipment in the world reference frame at time k, g w Let b be the gravitational acceleration at the current moment. Since its direction is downward, a negative sign is used to indicate the influence of its direction. a The bias is for acceleration.

[0085] Similarly, the predicted angular velocity can be determined. in, Let k be the predicted angular velocity at time k. Let be the first derivative of the rotation matrix from the world reference frame to the inertial reference value at time k. The specific predicted angular velocity can be derived from the formula for the derivative of the rotation matrix: It is determined that, based on the characteristics of the antisymmetric matrix, the formula for determining the predicted angular velocity can be established. Similarly, this testing device can also determine the predicted position of image markers. Where j is the time when the image sensor acquires the image, p j This represents the image acquired at time j. For each image acquired at a given time, Let be the translation matrix between the world reference frame and the image reference frame at that moment. Let be the translation matrix between the world reference frame and the image reference frame at that moment, l be the observed position of the image marker in the world reference frame at that moment, ∏() denotes normalization of the content within the parentheses, and w(,ζ) denotes the transformation of the observed position of the image marker in the image reference frame to the pixel reference frame, determining the position of the pixel where the image marker is located. Here, the image reference frame is the reference frame corresponding to the image sensor (e.g., camera), and the pixel reference frame is the reference frame corresponding to the image marker, such as the coordinates of the pixels where each image feature point of the image marker is located.

[0086] Therefore, based on the predicted acceleration, predicted angular velocity, predicted position of image markers, and the predicted acceleration corresponding to multiple moments in the inertial reference frame as determined above, angular velocity and the observation location of image markers Constraints can be constructed in,

[0087] That is, the test equipment can calculate the first parameter of the test equipment based on the constraint of minimizing the difference between the predicted acceleration and acceleration, the predicted angular velocity and angular velocity at multiple times, and the observed position and predicted position of the image marker. This will determine the motion trajectory of the test equipment.

[0088] It should be noted that the transformation relationship between the inertial reference frame and the image reference frame is predetermined. The rotation and translation matrices between the world reference frame and the inertial reference frame can be determined based on the pose of the test device in the world reference frame at multiple moments. The specific method of determining the rotation and translation matrices between the world reference frame and the inertial reference frame based on the pose is a relatively mature existing technology, and will not be elaborated on here.

[0089] In addition, the method of modeling motion trajectories in this manual can also be Bézier curves and other modeling methods. The specific method of constructing motion trajectory can be set as needed, based on the inertial data and image data corresponding to multiple time points. This manual does not impose any restrictions on this.

[0090] S104: Based on the radar data, determine the Doppler velocity during the movement of the test equipment, and register the Doppler velocity with the moving speed of the test equipment in the image reference frame to calibrate the sensor data.

[0091] In one or more embodiments provided in this specification, as previously described, the testing device can solve for the transformation relationship between the image reference frame and the radar reference frame based on the Doppler velocity and the movement velocity. Therefore, the testing device can determine the Doppler velocity in each directional component.

[0092] Specifically, the testing equipment can determine the Doppler velocity of the testing equipment in each of the multiple time moments based on the acquired radar data and the preset radar directional components.

[0093] For each directional component, which consists of pitch and azimuth angles, the Doppler velocity in that directional component is the Doppler velocity corresponding to that pitch and azimuth angle. For example, assuming a directional component consists of a pitch angle of 30° and an azimuth angle of 60°, the Doppler velocity in that directional component is the velocity component corresponding to the pitch angle of 30° and the azimuth angle of 60°.

[0094] It should be noted that the Doppler velocity mentioned above is determined by the radar sensor based on the Doppler effect. In other words, for each directional component, the Doppler velocity in that directional component can be measured.

[0095] Furthermore, since the Doppler velocity is the velocity of a stationary object relative to the test equipment, the Doppler velocity and the velocity of the test equipment are opposite in direction and equal in magnitude. Therefore, once the Doppler velocity is determined, the server can register the Doppler velocity and the velocity of the test equipment in the image reference frame to determine the calibration relationship between the radar reference frame and the image reference frame.

[0096] Specifically, the testing equipment can determine the Doppler velocity in each directional component as the Doppler velocity of the testing equipment in the same direction as the moving velocity, based on the direction of the moving velocity at multiple moments. Then, the Doppler velocity and the moving velocity are registered to determine the calibration relationship between the radar reference system and the image reference system.

[0097] Therefore, after determining the calibration relationship between the image sensor and the radar sensor, the testing equipment can calibrate the acquired sensor data according to the determined calibration relationship. For example, based on the calibration relationship, the radar data in the acquired sensor data can be converted into an image reference frame, and steps such as target recognition can be performed. Then, based on the recognition results, the motion strategy of the testing equipment can be determined.

[0098] Of course, after the calibration relationship is determined, it can also be used in various scenarios such as obstacle detection and obstacle classification. The specific application of the calibration relationship can be set as needed, and this manual does not limit it.

[0099] The calibrated sensor data can be sensor data collected during the current movement of the test equipment, or sensor data collected during subsequent movements of the test equipment. Alternatively, it can be sensor data collected by unmanned equipment with a similar structure to the test equipment during delivery tasks. The specific type of sensor data to be calibrated can be set as needed; this manual does not impose any restrictions on this.

[0100] Furthermore, since the calibration relationship between the radar reference frame and the image reference frame is to be solved, the direction of the test device's movement speed in the radar reference frame, determined based on the direction of the test device's movement speed in the image reference frame, may contain errors.

[0101] Based on this, the server can use the directional components of Doppler velocity as a reference, determine the moving speed of the test device in each directional component according to the preset directional components, and then solve the calibration relationship by minimizing the difference between the moving speed in each directional component and the Doppler velocity in each directional component.

[0102] Specifically, the test equipment can determine the moving speed of the test equipment in multiple moment image reference frames based on the motion trajectory of the test equipment in the world reference frame determined in step S102.

[0103] Then, for each of the multiple time points, the test equipment can determine the unsolved movement velocity in each directional component of the radar reference frame based on the unsolved calibration relationship between the image reference frame and the radar reference frame, and the movement velocity of the test equipment in the image reference frame. The unsolved movement velocity includes the unsolved calibration relationship. This calibration relationship includes at least the rotation matrix and translation matrix between the image reference frame and the radar reference frame.

[0104] Finally, the testing equipment can register the unsolved moving velocity and the Doppler velocity in each directional component to solve the calibration relationship. The registration method can be optimized by minimizing the difference between the unsolved moving velocity and the Doppler velocity in each directional component, and the optimization objective is then calculated.

[0105] Furthermore, the steps for calculating the moving speed and Doppler speed of the test equipment described above can be determined in the following manner:

[0106] Specifically, through The moving speed of the test equipment under the image reference can be determined. Among them, Let i be the speed at which the testing device moves in the image reference frame. Let v be the rotation matrix between the world reference frame and the image reference frame at time i. w (t iLet be the moving velocity of the test equipment in the world reference frame at time i, which can be determined by the first derivative of the displacement change of the test equipment at that time. It can be determined based on the transformation relationship from the world reference frame to the inertial reference frame at time i, and the predetermined transformation relationship between the inertial reference frame and the image reference frame.

[0107] Based on the calibration relationship between the image reference frame and the radar reference frame, the moving velocity in the radar reference frame at that moment can be determined. Among them, v r (t i Let be the moving velocity of the test equipment in the radar reference frame at time i. Let w be the rotation matrix between the radar reference frame and the image reference frame at time i. c (t i The angular velocity of the test device in the image reference frame at time i is indicated. Represents an antisymmetric matrix. Let be the translation matrix between the image reference frame and the radar reference frame at time i.

[0108] Then, the testing equipment can determine the unsolved moving velocity in each direction component under the radar reference frame according to the preset direction components. Based on the Doppler velocity determined in step S104, the following cost function can be determined: in, θ is the pitch angle, and θ is the azimuth angle.

[0109] The objective is to minimize the cost function, which means solving for the transformation relationship between the radar reference frame and the image reference frame.

[0110] Furthermore, since the radar sensor and the image sensor each have their own internal clock system, the time difference between the internal clock system of the radar sensor and the internal clock system of the image sensor can also be determined when determining the calibration relationship.

[0111] Specifically, for each time t among multiple times in the radar reference frame of the test equipment... s The time difference t between the internal clocks of the radar sensor and the image sensor, set on the test equipment, is to be solved. d Determine the speed at which the test device moves in the image reference frame at that moment, i.e., t. i =t s -t d The speed of movement at any given moment.

[0112] Secondly, the testing equipment can determine the unsolved moving speed of the testing equipment in the radar reference frame based on the moving speed of the testing equipment in the image reference frame at that moment, and the transformation relationship between the radar reference frame and the image reference frame.

[0113] Then, the testing device can determine the moving speed of the testing device in each directional component according to the preset directional components.

[0114] Finally, the moving velocities to be solved in each directional component and the Doppler velocities in each directional component are registered, the transformation relationship and the time difference are calculated, and the calculated transformation relationship and time difference are used as the calibration relationship. Wherein, t d This is the time difference between the internal clock system of the radar sensor and the internal clock system of the image sensor.

[0115] Among them, the moving speed of the test equipment under the image reference frame includes the time difference to be solved. When there is a time difference, the moving speed of the test equipment under the radar reference frame includes the time difference to be solved, as well as the translation matrix of the rotation matrix to be solved between the image reference frame and the radar reference frame.

[0116] based on Figure 1 The provided sensor data calibration method acquires image data, radar data, and inertial data at multiple moments during the movement of the test equipment. Based on the inertial and image data, the motion trajectory of the test equipment is determined. Based on the motion trajectory, the moving speed of the test equipment in the image reference frame is determined. Then, based on the radar data, the Doppler velocity of the test equipment during its movement is determined. The Doppler velocity and the moving speed of the test equipment in the image reference frame are registered to calibrate the sensor data. This scheme can also be applied when the acquisition ranges of the radar sensor and the image sensor do not overlap, and it does not require a calibration object moving within the overlapping field of view, making the calibration process more convenient and improving the calibration efficiency for determining sensor data.

[0117] In addition, due to factors such as vibration of the testing equipment, the positional relationship between the image sensor and the radar sensor may change. Therefore, the testing equipment can also determine whether the calibration relationship needs to be redefined during the movement of the equipment.

[0118] Specifically, the testing equipment can determine the pose difference of the testing equipment in the image reference frame and the radar reference frame at multiple time points based on the established calibration relationship and the collected sensor data, and determine whether the pose difference is greater than a preset error threshold. If so, the testing equipment can determine that the sensor data needs to be calibrated and store the sensor data. If not, the testing equipment can determine that the sensor data does not need to be calibrated.

[0119] Of course, to avoid the need to collect sensor data for a period of time before calibration when there is a large difference, resulting in low calibration efficiency, the testing equipment can also store sensor data within a preset time period. When the pose difference exceeds a preset error threshold, the calibration relationship is determined based on the pre-stored sensor data for the preset time period. The specific preset time period can be set as needed, and this manual does not impose any restrictions on it.

[0120] Furthermore, during the movement of the testing equipment, issues such as equipment vibration may occur, leading to significant differences in the determined pose. These vibrations typically disappear within a short time. Therefore, to improve the accuracy of the judgment, the testing equipment can record the number of times the pose difference exceeds an error threshold. When the pose difference is less than the error threshold, the calibration relationship is considered correct. Conversely, when the number of times the pose difference exceeds the error threshold reaches a preset threshold, the determined calibration relationship is considered unreliable.

[0121] In addition, after determining the calibration relationship, the testing equipment can determine the point cloud data and the projection of the point cloud data in the image reference system based on the acquired sensor data and the calibration relationship. Then, it can fuse the projection and the image data in the sensor data to determine the fusion result, and perform obstacle detection on the fusion result to determine the location of the obstacle.

[0122] Of course, the above method of fusing image data and point cloud data to determine the location of obstacles is only one of the uses of the calibration relationship. After the calibration relationship is determined, it can also be used for various scenarios such as obstacle detection and obstacle classification. The specific application of the calibration relationship can be set as needed, and this manual does not limit it.

[0123] The above describes a sensor data calibration method provided by one or more embodiments of this specification. Based on the same idea, this specification also provides a corresponding sensor data calibration device, such as... Figure 3 As shown.

[0124] Figure 3 The calibration device for the sensor data provided in this specification includes:

[0125] The acquisition module 200 is used to acquire sensor data at multiple moments during the movement of the test equipment. The sensor data includes at least image data, radar data, and inertial data.

[0126] The trajectory determination module 202 is used to determine the motion trajectory of the test device based on the inertial data and the image data, and to determine the moving speed of the test device in the image reference frame based on the motion trajectory.

[0127] The calibration module 204 is used to determine the Doppler velocity of the test equipment during its movement based on the radar data, and to register the Doppler velocity with the moving velocity of the test equipment in the image reference frame to determine the calibration relationship between the radar reference frame and the image reference frame, so as to calibrate the sensor data.

[0128] Optionally, the trajectory determination module 202 is used to determine the angular velocity, acceleration, and observation position of the image markers of the test device at multiple moments in the inertial reference frame based on the inertial data and the image data, and to solve the motion trajectory of the test device in the world reference frame based on the angular velocity, acceleration, and observation position of the image markers at the multiple moments.

[0129] Optionally, the trajectory determination module 202 is used to determine the first parameters to be solved for each of the plurality of time points based on the motion trajectory to be solved by the test device in the world reference frame. The first parameters are used to solve the motion trajectory. For each of the plurality of time points, based on the first parameters to be solved for that time point, the conversion relationship between the world reference frame and the inertial reference frame is determined, as well as the estimated angular velocity, estimated acceleration, and estimated position of the image marker of the test device in the inertial reference frame are determined. Constraints are constructed based on the angular velocity and the estimated angular velocity, the acceleration and the estimated acceleration corresponding to that time point, and the observed position and the estimated position of the image marker are the same, and the motion trajectory is solved.

[0130] Optionally, the trajectory determination module 202 is configured to determine the velocity of the test device at the multiple moments in the world reference frame based on the motion trajectory, determine the conversion relationship between the world reference frame and the inertial reference frame at the multiple moments based on the pose of the test device at the multiple moments, and determine the moving speed of the test device at the multiple moments in the image reference frame based on the velocity of the test device at the multiple moments in the world reference frame, the conversion relationship between the world reference frame and the inertial reference frame at the multiple moments, and a preset conversion relationship between the inertial reference frame and the image reference frame.

[0131] Optionally, the trajectory determination module 202 is configured to, for each of the plurality of time moments, determine the Doppler velocity in each directional component at that time moment based on the acquired radar data and preset directional components, determine the unsolved moving speed of the test device in each directional component based on the moving speed of the test device in the image reference frame at that time moment, the calibration relationship to be solved between the radar reference frame and the image reference frame, and the preset directional components, register the unsolved moving speed in each directional component and the Doppler velocity in each directional component, and solve the calibration relationship to calibrate the sensor data.

[0132] Optionally, the calibration module 204 is configured to determine the unsolved moving speed of the test device in the image reference frame at a given moment based on the unsolved time difference between the internal clocks of the radar sensor and the image sensor set on the test device, wherein the unsolved moving speed includes the unsolved time difference; determine the unsolved moving speed of the test device in the radar reference frame based on the unsolved moving speed of the test device in the image reference frame at that moment and the unsolved transformation relationship between the radar reference frame and the image reference frame; determine the unsolved moving speed of the test device in each direction component based on preset direction components; register the unsolved moving speed in each direction component and the Doppler velocity in each direction component; and calculate the transformation relationship and the time difference as the calibration relationship.

[0133] Optionally, the calibration module 204 is used to determine the pose difference of the test device in the image reference system and the radar reference system at each time according to the determined calibration relationship and the collected sensor data, and to determine whether the pose difference is greater than a preset error threshold. If so, it is determined that the sensor data needs to be calibrated and the sensor data is stored. If not, it is determined that the sensor data does not need to be calibrated.

[0134] This specification also provides a computer-readable storage medium storing a computer program that can be used to execute the above-described... Figure 1 The calibration method for the provided sensor data.

[0135] This specification also provides a computer-readable storage medium storing a computer program that can be used to execute the above-described... Figure 1 The calibration method for the provided sensor data.

[0136] This instruction manual also provides Figure 4 The diagram shows a schematic structural representation of the electronic device. Figure 4At the hardware level, the electronic device includes a processor, internal bus, network interface, memory, and non-volatile memory, and may also include other hardware required for the business operations. The processor reads the corresponding computer program from the non-volatile memory into memory and then runs it to achieve the above-mentioned functions. Figure 1 The method for calibrating sensor data is described above. Of course, besides software implementation, this specification does not exclude other implementation methods, such as logic devices or a combination of hardware and software, etc. That is to say, the execution entity of the following processing flow is not limited to individual logic units, but can also be hardware or logic devices.

[0137] In the 1990s, improvements to a technology could be clearly distinguished as either hardware improvements (e.g., improvements to the circuit structure of diodes, transistors, switches, etc.) or software improvements (improvements to the methodology). However, with technological advancements, many methodological improvements today can be considered direct improvements to the hardware circuit structure. Designers almost always obtain the corresponding hardware circuit structure by programming the improved methodology into the hardware circuit. Therefore, it cannot be said that a methodological improvement cannot be implemented using hardware physical modules. For example, a Programmable Logic Device (PLD) (such as a Field Programmable Gate Array (FPGA)) is such an integrated circuit whose logic function is determined by the user programming the device. Designers can program and "integrate" a digital system onto a PLD themselves, without needing chip manufacturers to design and manufacture dedicated integrated circuit chips. Furthermore, nowadays, instead of manually manufacturing integrated circuit chips, this programming is mostly implemented using "logic compiler" software. Similar to the software compiler used in program development, the original code before compilation must be written in a specific programming language, called a Hardware Description Language (HDL). There are many HDLs, such as ABEL (Advanced Boolean Expression Language), AHDL (Altera Hardware Description Language), Confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), Lava, Lola, MyHDL, PALASM, and RHDL (Ruby Hardware Description Language). Currently, the most commonly used are VHDL (Very-High-Speed ​​Integrated Circuit Hardware Description Language) and Verilog. Those skilled in the art should understand that by simply performing some logic programming on the method flow using one of these hardware description languages ​​and programming it into an integrated circuit, the hardware circuit implementing the logical method flow can be easily obtained.

[0138] The controller can be implemented in any suitable manner. For example, it can take the form of a microprocessor or processor and a computer-readable medium storing computer-readable program code (e.g., software or firmware) executable by the (micro)processor, logic gates, switches, application-specific integrated circuits (ASICs), programmable logic controllers, and embedded microcontrollers. Examples of controllers include, but are not limited to, the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20, and Silicon Labs C8051F320. A memory controller can also be implemented as part of the control logic of the memory. Those skilled in the art will also recognize that, in addition to implementing the controller in purely computer-readable program code form, the same functionality can be achieved by logically programming the method steps to make the controller take the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers. Therefore, such a controller can be considered a hardware component, and the means included therein for implementing various functions can also be considered as structures within the hardware component. Alternatively, the means for implementing various functions can be considered as both software modules implementing the method and structures within the hardware component.

[0139] The systems, devices, modules, or units described in the above embodiments can be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, a computer can be, for example, a personal computer, a laptop computer, a cellular phone, a camera phone, a smartphone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or any combination of these devices.

[0140] For ease of description, the above devices are described in terms of function, divided into various units. Of course, in implementing this specification, the functions of each unit can be implemented in one or more software and / or hardware.

[0141] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0142] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0143] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0144] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0145] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0146] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0147] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0148] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0149] Those skilled in the art will understand that the embodiments of this specification can be provided as methods, systems, or computer program products. Therefore, this specification may take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this specification may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0150] This specification can be described in the general context of computer-executable instructions that are executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform a specific task or implement a specific abstract data type. This specification can also be practiced in distributed computing environments, where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0151] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to interchangeably. Each embodiment focuses on describing the differences from other embodiments. In particular, the system embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments.

[0152] The above description is merely an embodiment of this specification and is not intended to limit this specification. Various modifications and variations can be made to this specification by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this specification should be included within the scope of the claims of this specification.

Claims

1. A method for calibrating sensor data, characterized in that, The method includes: Acquire sensor data at multiple moments during the movement of the test equipment, wherein the sensor data includes at least image data, radar data, and inertial data; Based on the inertial data and the image data, the motion trajectory of the test device is determined, and based on the motion trajectory, the moving speed of the test device in the image reference frame is determined. Based on the radar data, the Doppler velocity during the movement of the test equipment is determined, and the Doppler velocity and the movement velocity of the test equipment in the image reference system are registered to calibrate the sensor data. Specifically, registering the Doppler velocity with the moving velocity of the test device in the image reference frame to calibrate the sensor data includes: For each of the plurality of moments, the Doppler velocity in each direction component at that moment is determined based on the acquired radar data and the preset direction components. Based on the moving speed of the test device in the image reference frame at that moment, the calibration relationship between the radar reference frame and the image reference frame to be solved, and the preset directional components, the moving speed of the test device to be solved in each directional component is determined. The moving velocity and Doppler velocity in each directional component are registered, and the calibration relationship is calculated to calibrate the sensor data.

2. The method as described in claim 1, characterized in that, Determining the motion trajectory of the testing device based on the inertial data and the image data specifically includes: Based on the inertial data and the image data, determine the angular velocity, acceleration, and observation position of the image markers at the multiple moments in the inertial reference frame for the test equipment. The trajectory of the test equipment in the world reference frame is solved based on the angular velocity, acceleration, and observation position of the image markers corresponding to the multiple moments.

3. The method as described in claim 2, characterized in that, Based on the angular velocity, acceleration, and observed positions of image markers at each time point, the trajectory of the test equipment in the world reference frame is calculated, specifically including: Based on the motion trajectory of the test device in the world reference frame, determine the first parameters to be solved for the plurality of times respectively, and the first parameters are used to solve the motion trajectory; For each of the plurality of moments, based on the first parameter to be solved corresponding to that moment, the transformation relationship between the world reference frame and the inertial reference frame is determined, and the estimated angular velocity, estimated acceleration, and estimated position of the image markers of the test device in the inertial reference frame are determined; Using the angular velocity and estimated angular velocity, acceleration and estimated acceleration corresponding to that moment, and the fact that the observed position and estimated position of the image marker are the same, constraints are constructed to solve for the motion trajectory.

4. The method as described in claim 3, characterized in that, Based on the first parameter to be solved at that moment, the transformation relationship between the world reference frame and the inertial reference frame is determined, as well as the estimated angular velocity, estimated acceleration, and estimated position of the image markers of the test equipment in the inertial reference frame, specifically including: Based on the pose of the test device in the first parameter to be solved at that moment, the transformation relationship between the world reference frame and the inertial reference frame to be solved at that moment is determined, wherein the first parameter includes the pose of the test device, acceleration offset, angular velocity offset, and observation position of image markers; Based on the transformation relationship to be solved, the angular velocity offset to be solved in the first parameter, the acceleration offset to be solved, and the observation position of the image marker to be solved, the estimated acceleration, estimated angular velocity, and estimated position of the image marker of the test equipment are determined respectively.

5. The method as described in claim 2, characterized in that, Based on the motion trajectory, the moving speed of the test device in the image reference frame is determined, specifically including: Based on the motion trajectory, determine the velocity of the acquisition device at each of the multiple moments in the world reference frame; Based on the pose of the acquisition device at the multiple moments, determine the transformation relationship between the world reference frame and the inertial reference frame corresponding to the multiple moments respectively; Based on the velocity of the acquisition device at multiple moments in the world reference frame, the conversion relationship between the world reference frame and the inertial reference frame corresponding to the multiple moments, and the preset conversion relationship between the inertial reference frame and the image reference frame, the moving speed of the test device at the multiple moments in the image reference frame is determined.

6. The method as described in claim 1, characterized in that, Based on the moving speed of the test equipment in the image reference frame at that moment, the calibration relationship between the radar reference frame and the image reference frame to be solved, and the preset directional components, the moving speed of the test equipment to be solved in each directional component is determined, specifically including: Based on the unsolved time difference between the internal clocks of the radar sensor and the image sensor set on the test equipment, the unsolved moving speed of the test equipment in the image reference frame at that moment is determined, and the unsolved moving speed includes the unsolved time difference; Based on the unsolved moving speed of the test device in the image reference frame at that moment, and the unsolved transformation relationship between the radar reference frame and the image reference frame, the unsolved moving speed of the test device in the radar reference frame is determined. Based on the preset directional components, determine the moving speed of the test device in each directional component. The calibration relationship is calculated by registering the unsolved moving velocities and Doppler velocities in each directional component, and then solving for the calibration relationship. Specifically, this includes: The moving velocity and Doppler velocity in each directional component are registered, and the transformation relationship and time difference are calculated as the calibration relationship.

7. The method as described in claim 1, characterized in that, The method further includes: Based on the established calibration relationship and the collected sensor data, the pose difference of the test device in the image reference system and the radar reference system at each time point is determined, and it is determined whether the pose difference is greater than a preset error threshold. If so, then the sensor data needs to be calibrated, and the sensor data is stored; If not, then the sensor data does not require calibration.

8. A computer-readable storage medium, characterized in that, The storage medium stores a computer program, which, when executed by a processor, implements the method described in any one of claims 1 to 7.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the method described in any one of claims 1 to 7.

Images