Ground parameter calibration method, device and equipment

Abstract

The invention provides a ground parameter calibration method, device and equipment. The ground parameter calibration method comprises the following steps: acquiring map data acquired by acquisition equipment when the acquisition equipment runs on a target road; vIO processing is carried out on the map data, intermediate processing data is obtained, and the intermediate processing data at least comprises three-dimensional coordinates and pixel coordinates of road sign points located on the road surface of the target road; obtaining a target road sign point from the road sign points; performing plane fitting based on the three-dimensional coordinates of the target road sign point to obtain initial ground parameters of the acquisition equipment; calibrating ground parameters of the acquisition equipment according to the three-dimensional coordinates, the pixel coordinates and the initial ground parameters of the target road sign point and a three-dimensional coordinate error function constructed based on the ground parameters; the error function is used for obtaining the error between the three-dimensional coordinate of the target road sign point and the three-dimensional observation coordinate of the target road sign point obtained by using the ground parameter and the pixel coordinate corresponding to the target road sign point. The calibration efficiency of the ground parameters can be improved.

Description

Technical Field

[0001] The present application relates to the technical field of map data processing, and in particular to a ground parameter calibration method, device and equipment. Background Art

[0002] In recent years, with the development of basic capabilities such as sensors, on-end computing, and network communications, it has become a trend to use map data collected in compliance with consumer devices to produce and update maps.

[0003] At present, when some consumer devices collect map data, they estimate the pose of the camera carried by the consumer device through the Visual Inertial Odometry (VIO) system. Among them, the VIO system introduces constraints on ground parameters so that when the acceleration of the consumer device remains unchanged during driving, the travel scale can still be observed, thereby improving positioning accuracy and robustness. In addition, when subsequently making maps using map data collected by consumer devices, the position of map elements on the ground (such as lane lines, road signs, etc.) also needs to be calculated in combination with ground parameters.

[0004] At present, calibration equipment is mainly used to calibrate ground parameters offline, which leads to low calibration efficiency of ground parameters. Summary of the invention

[0005] The present application provides a ground parameter calibration method, device and equipment, which can improve the calibration efficiency of ground parameters.

[0006] In a first aspect, the present application provides a ground parameter calibration method, the method comprising:

[0007] Obtain map data collected by the collection device when driving on the target road;

[0008] Performing visual inertial odometer (VIO) processing on the map data to obtain intermediate processing data, wherein the intermediate processing data at least includes: three-dimensional coordinates and pixel coordinates of landmark points located on the road surface of the target road;

[0009] Acquire a target landmark point from the landmark points;

[0010] Performing plane fitting based on the three-dimensional coordinates of the target landmark points to obtain initial ground parameters of the acquisition device;

[0011] The ground parameters of the acquisition device are calibrated according to the three-dimensional coordinates, pixel coordinates, initial ground parameters and a three-dimensional coordinate error function constructed based on the ground parameters of the target landmark point; the three-dimensional coordinate error function is used to obtain the error between the three-dimensional coordinates of the target landmark point and the three-dimensional observation coordinates, and the three-dimensional observation coordinates are the three-dimensional coordinates of the target landmark point obtained by using the ground parameters and the pixel coordinates corresponding to the target landmark point.

[0012] In a second aspect, the present application provides a ground parameter calibration device, the device comprising:

[0013] An acquisition module is used to acquire map data collected by the acquisition device when driving on the target road;

[0014] A first processing module is used to perform visual inertial odometer (VIO) processing on the map data to obtain intermediate processing data, wherein the intermediate processing data at least includes: three-dimensional coordinates and pixel coordinates of landmark points located on the road surface of the target road;

[0015] A second processing module, used for acquiring a target landmark point from the landmark points;

[0016] A third processing module is used to perform plane fitting based on the three-dimensional coordinates of the target landmark point to obtain the initial ground parameters of the acquisition device;

[0017] A calibration module is used to calibrate the ground parameters of the acquisition device according to the three-dimensional coordinates of the target landmark point, the pixel coordinates, the initial ground parameters, and the three-dimensional coordinate error function constructed based on the ground parameters; the three-dimensional coordinate error function is used to obtain the error between the three-dimensional coordinates of the target landmark point and the three-dimensional observation coordinates, and the three-dimensional observation coordinates are the three-dimensional coordinates of the target landmark point obtained by using the ground parameters and the pixel coordinates corresponding to the target landmark point.

[0018] In a third aspect, the present application provides an electronic device, comprising: a processor and a memory; the processor is communicatively connected to the memory;

[0019] The memory stores computer-executable instructions;

[0020] The processor executes the computer-executable instructions stored in the memory to implement the method as described in any one of the first aspects.

[0021] In a fourth aspect, the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer-executable instructions, and when the computer-executable instructions are executed by a processor, they are used to implement the method as described in any one of the first aspects.

[0022] In a fifth aspect, the present application provides a computer program product, including a computer program, which implements the method described in any one of the first aspects when executed by a processor.

[0023] The ground parameter calibration method, device and equipment provided in the present application can obtain the data of the landmark points on the road surface of the target road based on the intermediate processed data obtained by VIO processing of the map data collected by the acquisition device when driving on the target road, and then automatically calibrate the ground parameters based on the data of the landmark points. Compared with the offline calibration method using the calibration equipment in the prior art, the calibration efficiency of the ground parameters can be improved. In addition, the calibration method of the present application does not require manual participation, so it can save labor costs. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] In order to more clearly illustrate the technical solutions in the present application or the prior art, a brief introduction will be given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0025] Figure 1 is a schematic diagram of a camera coordinate system;

[0026] Figure 2 A schematic diagram of a flow chart of a ground parameter calibration method provided in an embodiment of the present application;

[0027] Figure 3 A schematic diagram of a flow chart of another ground parameter calibration method provided in an embodiment of the present application;

[0028] Figure 4 A schematic diagram of a flow chart of another ground parameter calibration method provided in an embodiment of the present application;

[0029] Figure 5 A schematic diagram of a flow chart of another ground parameter calibration method provided in an embodiment of the present application;

[0030] Figure 6 A schematic diagram of the structure of a ground parameter calibration device provided in an embodiment of the present application;

[0031] Figure 7 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application.

[0032] The above drawings have shown clear embodiments of the present application, which will be described in more detail later. These drawings and text descriptions are not intended to limit the scope of the present application in any way, but to illustrate the concept of the present application to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION

[0033] In order to make the purpose, technical solutions and advantages of this application clearer, the technical solutions in this application will be clearly and completely described below in conjunction with the drawings in this application. Obviously, the described embodiments are part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0034] The following first explains some of the noun concepts involved in this application:

[0035] Map data includes: image data, IMU data and trajectory data (data consisting of a series of location points), etc. The aforementioned data can be collected in compliance with consumer devices, or collected through professional equipment. Among them, consumer devices can be mobile devices deployed on vehicles traveling on the road. The above mobile devices can be, for example, smartphones, driving recorders, or customized lightweight visual devices deployed in vehicles such as taxis and private cars.

[0036] In this application, the mobile device may be equipped with a camera (also referred to as a camera), inertial measurement unit (IMU) data, and a positioning receiver. Among them, the positioning receiver can receive any positioning signal that can realize a positioning system, such as the Global Positioning System (GPS), the Beidou positioning system, and the Real-time kinematic (RTK) system. The camera can be a monocular camera, a binocular camera, or a multi-eye camera.

[0037] The camera can collect images of the road on which the vehicle is traveling, the IMU can collect the IMU data of the vehicle, and the positioning signal received by the positioning signal machine can determine the positioning data of the vehicle, that is, the position data. The positioning data may include: the positioning observation data of the positioning system, and the positioning reliability, etc.

[0038] The VIO system, also known as the visual inertial system, is an algorithm that integrates camera and inertial measurement unit (IMU) data to achieve simultaneous localization and mapping (SLAM), which can output the camera's position and posture based on map data. This position and posture can also be regarded as the position and posture of a mobile device, or as the position and posture of a vehicle.

[0039] Map makers mainly collect map data by dispatching their own professional collection vehicles (equipped with professional sensors), and produce and update maps based on the collected map data. However, this processing method is too expensive and cannot meet actual usage needs. In recent years, with the development of basic capabilities such as sensors, on-end computing, and network communications, it has become a trend to use map data collected in compliance with consumer devices to produce and update maps. This can solve the problem of small coverage of maps due to insufficient coverage of map data, and slow map update frequency due to insufficient frequency of map data collection.

[0040] At present, when some consumer devices collect map data, they need to use the collected images and IMU data to estimate the position and posture of the camera of the consumer device through the VIO system. However, when the acceleration of the consumer device remains unchanged during driving, the travel scale is unobservable, which is prone to serious scale drift. Therefore, it is necessary to introduce constraints on ground parameters in the VIO system so that when the acceleration of the consumer device remains unchanged during driving, the travel scale can still be observed, thereby improving positioning accuracy and robustness. In addition, when subsequently making maps using map data collected by consumer devices, the position of map elements on the ground (such as lane lines, road signs, etc.) also needs to be calculated in combination with ground parameters.

[0041] The ground parameters mentioned here are the representation of the visible ground in the camera coordinate system of the acquisition device (such as a consumer device). The visible ground can be understood as the ground range captured by the camera in a certain posture.

[0042] It should be noted that this application does not limit the definition of the camera coordinate system of the acquisition device, nor the expression form of the ground parameters.

[0043] Figure 1 is a schematic diagram of a camera coordinate system, such as Figure 1 As shown, exemplarily, the camera coordinate system can be a coordinate system constructed in the following manner: the optical center of the camera of the acquisition device is defined as the coordinate origin, the front direction of the vehicle where the acquisition device is located is defined as the z-axis of the camera coordinate system, the downward direction perpendicular to the vehicle body is defined as the y-axis of the camera coordinate system, and the right side of the vehicle body is defined as the x-axis of the camera coordinate system.

[0044] The visible ground can be represented in the camera coordinate system by the ground normal vector It is represented by the vertical distance d from the origin of the camera coordinate system to the fitting plane. That is, the plane equation of the visible ground in the camera coordinate system is shown in the following formula (1):

[0045] n x *x+n y *y+n z*zd=0

[0046] Among them, n x , n y , n z is the ground normal vector n x *n x +n y *n y +n z *n z =1.

[0047] Therefore, for example, the ground can be parameterized as shown in the following formula (2):

[0048]

[0049] Among them, param is the ground parameter, that is, the representation of the visible ground in the camera coordinate system.

[0050] At present, for ground parameters, offline calibration using calibration equipment is mainly used. The calibration equipment mentioned here can be, for example, a calibration board in the form of a checkerboard. Specifically, the calibration equipment is manually placed on the ground of the road, and the acquisition equipment uses a camera to capture an image containing the calibration equipment. Then, the image containing the calibration equipment is processed by a calibration algorithm to obtain the calibrated ground parameters.

[0051] This calibration method requires offline calibration and manual participation, resulting in low calibration efficiency of ground parameters.

[0052] In view of this, the present application provides a ground parameter calibration method, which can automatically calibrate the ground parameters based on the intermediate processing data in the VIO processing process of the map data collected by the acquisition equipment. Compared with the offline calibration method of the calibration equipment used in the prior art, the calibration efficiency of the ground parameters can be improved, and the labor cost can be saved. The calibrated ground parameters can be used for map production, and can also be used in any scenario that requires the use of ground parameters, such as VIO processing.

[0053] In specific implementation, the ground parameters can be calibrated during the process of the acquisition device collecting map data. That is, the ground parameters are calibrated based on the map data collected in real time. Alternatively, the ground parameters can be calibrated based on the collected map data after the acquisition device has collected the map data.

[0054] It should be understood that when the ground parameters are calibrated during the process of the acquisition device collecting map data, the execution subject of the method can be the computing device of the vehicle where the acquisition device is located, or a server with more powerful performance. After the acquisition device has collected the map data, the computing device or the server can calibrate the ground parameters based on the collected map data.

[0055] It should be noted that the method of the embodiment of the present application can be used in scenarios where map data collected by consumer devices are used to create and update maps, and can also be used in scenarios where map data collected by a self-owned professional collection vehicle (equipped with professional sensors) are used to create and update maps. In other words, the collection device can be a consumer device or a professional collection vehicle.

[0056] It should be noted that this application does not limit the accuracy of the map. It can be a standard precision map or a high precision map.

[0057] The technical solution of the present application is described in detail below in conjunction with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described in detail in some embodiments.

[0058] Figure 2 A flow chart of a ground parameter calibration method provided in an embodiment of the present application. Figure 2 As shown, the method may include the following steps:

[0059] S101. Acquire map data collected by a collection device when the collection device is driving on a target road.

[0060] As mentioned above, the map data may include: images of the target road, IMU data and positioning data. The positioning data may include: positioning observation data of the positioning system, and positioning reliability, etc.

[0061] In the present application, the method of obtaining the map data collected by the acquisition device is related to the method of calibrating the ground parameters in the present application. For the method of realizing the calibration of ground parameters in the process of the acquisition device collecting map data, the map data collected in real time can be obtained from the acquisition device. For the method of calibrating the ground parameters based on the collected map data after the acquisition device has collected the map data, the map data collected in real time can be obtained from the acquisition device, the collected map data data can be obtained from the acquisition device periodically, or the map data collected in this collection process can be obtained from the acquisition device after the acquisition device completes the collection of the map data.

[0062] S102: Perform VIO processing on the map data to obtain intermediate processing data.

[0063] For example, VIO processing can be performed on the images and IMU data in the map data. For how to perform VIO processing, please refer to the prior art. To save space, this application will not go into details.

[0064] The image in the map data includes the road surface of the target road, so in the process of VIO processing, the intermediate processing data of VIO can be obtained. The intermediate processing data mentioned here refers to the intermediate processing results obtained in the process of VIO processing.

[0065] In this embodiment, the intermediate processing data at least includes: three-dimensional (3D) coordinates and pixel coordinates of the landmark points located on the road surface of the target road. The pixel coordinates are used to represent the position of the landmark points in the corresponding image.

[0066] The landmark points mentioned here can be, for example, feature points with large pixel gradient changes in map elements on the ground. For example, the corner points of traffic signs on the ground. For example, whether the feature point is a corner point of a map element and / or the pixel gradient of the feature point can be used to determine whether it is a landmark point. The behavior of filtering landmark points can be a behavior in the VIO processing process, or it can be obtained by filtering feature points from the intermediate processing results of VIO using the above method.

[0067] S103: Acquire a target landmark point from the landmark points.

[0068] The method of obtaining the target landmark points from the landmark points is related to the method of subsequent ground parameter calibration.

[0069] For example, the target landmark points used for calibrating the ground parameters can be firstly selected from the acquired landmark points, and then the ground parameters of the acquisition device are initialized and calibrated based on the selected target landmark points. That is, the data used for calibration is firstly selected, and then the ground parameters are initialized, and then the calibration is performed.

[0070] Alternatively, the target landmark point includes a first target landmark point and a second target landmark point, wherein the first target landmark point is used for ground parameter initialization, and the second target landmark point is used for ground parameter calibration. For example, the first target landmark point from the same frame image can be obtained from the landmark points to initialize the ground parameters of the acquisition device, and then the initialized ground parameters are used to filter the second target landmark point used for calibrating the ground parameters from the remaining landmark points. That is, the data corresponding to the single frame image is first used to initialize the ground parameters, and then the data used for calibration is filtered for calibration.

[0071] Optionally, a threshold value for the number of required target landmarks may be preset, so that subsequent operations are performed after target landmarks that meet the threshold value are screened out.

[0072] S104: Perform plane fitting based on the three-dimensional coordinates of the target landmark point to obtain initial ground parameters of the acquisition device.

[0073] For example, any plane fitting algorithm can be used to perform plane fitting on the target landmark point according to the three-dimensional coordinates of the target landmark point. The plane fitting algorithm mentioned here can be, for example, a Random Sample Consensus (RANSAC) algorithm or a least squares fitting algorithm.

[0074] It should be understood that after the plane is fitted, whether the fitted plane is valid can be determined based on whether the number of target landmark points included in the fitted plane is greater than or equal to the number. If valid, the ground parameters corresponding to the fitted plane are used as the initial ground parameters. If invalid, the target landmark points can be re-acquired based on the implementation method of step S103 to initialize the ground parameters.

[0075] It should be understood that re-obtaining the target landmark point from the landmark points may be obtaining the target landmark point from the remaining landmark points in the map data.

[0076] S105 , calibrating the ground parameters of the acquisition device according to the three-dimensional coordinates, pixel coordinates, initial ground parameters of the target landmark point, and a three-dimensional coordinate error function constructed based on the ground parameters.

[0077] The error function is used to obtain the error between the 3D coordinates of the target landmark point and the 3D observation coordinates. For example, the following formula (3) shows:

[0078]

[0079] Among them, p is the 3D coordinate of the landmark point obtained by VIO, K is the camera internal parameter, and K -1 is the inverse of K, obs is the pixel coordinate of the landmark point, K -1 obs is the coordinate of the landmark point in the normalized camera coordinate system, param·(K -1 obs) represents the ground parameters param and K -1 The dot product of obs, Represents the 3D observation coordinates, that is, the three-dimensional coordinates of the target landmark point obtained by using the ground parameters and the pixel coordinates corresponding to the target landmark point.

[0080] Error is the error term that needs to be optimized during the calibration process. The physical meaning of this error term in the calibration process is to minimize the error between the 3D coordinates of the landmark point obtained by VIO and the 3D observation coordinates calculated by pixel coordinates and ground parameters.

[0081] For the implementation method of first using the data corresponding to a single frame image to initialize the ground parameters and then screening the data for calibration, the ground parameters of the acquisition device can be calibrated according to the three-dimensional coordinates, pixel coordinates, camera intrinsic parameters of the acquisition device, initial ground parameters, and the three-dimensional coordinate error function constructed based on the ground parameters of the screened second target landmark point. Then, the calibrated ground parameters are used to continue to acquire the second target landmark point, and the ground parameters of the acquisition device are calibrated again using the three-dimensional coordinates and pixel coordinates of the newly acquired second target landmark point and the previously acquired second target landmark point (i.e., the second target landmark point used in the last calibration), the ground parameters of the last calibration, the camera intrinsic parameters of the acquisition device, and the three-dimensional coordinate error function, until the calibrated ground parameters make the three-dimensional coordinate error of the second target landmark point less than or equal to the preset value, and / or the number of calibrations reaches the preset number. That is, the ground parameters of the acquisition device are calibrated by iterative loop calibration. The preset number mentioned here can be, for example, 2, which can be specifically set according to the requirements of calibration accuracy.

[0082] The calibration of ground parameters is performed by an iterative cycle calibration method, which can improve the calibration accuracy and robustness of the ground parameters.

[0083] For the implementation method of first screening the data for calibration and then initializing the ground parameters, since the ground parameter initialization uses the screened landmark points, the accuracy of the initialized ground parameters is relatively high. Therefore, a calibration process can be performed based on the three-dimensional coordinates, pixel coordinates, initial ground parameters of the target landmark points, and the three-dimensional coordinate error function constructed based on the ground parameters to obtain the ground parameters of the calibration acquisition device. Of course, the aforementioned iterative cycle calibration method can also be used to calibrate the ground parameters of the acquisition device, which can be specifically set according to the calibration accuracy requirements.

[0084] It should be noted that in the implementation scenario of ground parameter calibration during the process of map data collection by the acquisition device, the above-mentioned ground parameter calibration can be that during the process of map data collection by the acquisition device, the collected map data is continuously subjected to VIO processing, and based on the continuously acquired intermediate processing data of VIO, target landmarks are collected, as well as ground parameter initialization operations are performed until the calibration process is completed.

[0085] The ground parameter calibration method provided in the embodiment of the present application can obtain the data of the landmark points on the road surface of the target road based on the intermediate processed data obtained by VIO processing of the map data collected by the acquisition device when driving on the target road, and then can automatically calibrate the ground parameters based on the data of the landmark points. Compared with the offline calibration method using the calibration equipment in the prior art, the calibration efficiency of the ground parameters can be improved. In addition, the calibration method of the present application does not require manual participation, so it can save labor costs.

[0086] The following describes how to obtain target landmark points from landmark points, initialize the ground parameters of the acquisition device, and calibrate the ground parameters. The acquisition device is integrated with a camera.

[0087] The first method is to first filter the data used for calibration, then initialize the ground parameters, and then calibrate the ground parameters.

[0088] Figure 3 A flow chart of another ground parameter calibration method provided in an embodiment of the present application. Figure 3 As shown, the method may include:

[0089] S201. Acquire map data collected by a collection device when the collection device is driving on a target road.

[0090] S202: Perform VIO processing on the map data to obtain intermediate processing data.

[0091] The intermediate processing data includes: the 3D coordinates of the landmark points on the road surface of the target road, the pixel coordinates, the distance from the landmark points to the optical center of the camera, and the speed and angular velocity corresponding to the camera when capturing the landmark points.

[0092] It should be understood that the distance from the landmark point to the optical center of the camera mentioned here refers to the straight-line distance between the landmark point and the optical center of the camera. The speed and angular velocity corresponding to the camera capturing the landmark point can be the speed and angular velocity corresponding to the image to which the landmark point belongs. In other words, when the camera captures the image to which the landmark point belongs, the speed and angular velocity of the acquisition device, or the speed and angular velocity of the camera of the acquisition device.

[0093] When two landmark points come from the same frame of image, the speeds corresponding to the two landmark points are the same, and the corresponding angular velocities are also the same.

[0094] S203: Obtain the positioning reliability corresponding to the landmark point from the map data.

[0095] For example, the positioning confidence corresponding to the landmark point can be the positioning confidence corresponding to the image to which the landmark point belongs. That is, when two landmark points come from the same frame of image, the positioning confidences corresponding to the two landmark points are the same. The positioning confidence is positively correlated with the positioning accuracy. That is, the higher the confidence, the higher the positioning accuracy.

[0096] For example, the intermediate processing data also includes: the image to which the landmark point belongs. Therefore, the positioning confidence corresponding to the positioning data with the same timestamp as the landmark point can be obtained from the positioning data of the map data through the timestamp of the image to which the landmark point belongs, and the confidence is used as the positioning confidence corresponding to the landmark point.

[0097] S204. According to the distance from the landmark point to the optical center of the camera, the speed and angular velocity corresponding to the landmark point, a target landmark point is obtained from the landmark points whose positioning confidence is greater than a preset first confidence threshold.

[0098] For example, the distance from the landmark point to the optical center of the camera, the speed corresponding to the landmark point, and the angular velocity each correspond to a filtering condition. Therefore, from the landmark points whose positioning confidence is greater than the first confidence threshold, the landmark points that meet the corresponding filtering conditions can be selected as the target landmark points.

[0099] Exemplarily, the filtering condition corresponding to the distance from the landmark point to the optical center of the camera may be, for example, that the distance from the landmark point to the optical center of the camera is greater than 1 meter and less than 20 meters. The filtering condition corresponding to the speed may be, for example, that the modulus of the speed is greater than 4 meters / second and less than 20 meters / second. The filtering condition corresponding to the angular velocity may be, for example, that the component of the angular velocity output by the VIO along the x-axis and y-axis of the vehicle coordinate system is less than 1 radian / second. The examples shall not be regarded as limitations on the present application.

[0100] By using the distance from the landmark point to the optical center of the camera, landmark points that are far from the optical center of the camera can be filtered out to ensure the accuracy of the target landmark points obtained by screening. By using the speed and angular velocity, it can be ensured that the selected target landmark points are landmark points where the vehicle where the acquisition device is located is in a stable driving state, and the speed and angular velocity output by the VIO are within a reasonable range. In this way, the quality of the obtained target landmark points can be ensured, thereby ensuring the robustness of subsequent calibration.

[0101] As for how many target landmarks to be screened, it can be set according to actual needs. When the number of the screened target landmarks reaches the set value, the subsequent ground parameter initialization process can be executed.

[0102] It should be understood that when the vehicle computer uses the acquisition equipment to collect map data to implement the calibration of ground parameters, a smaller threshold can be set during the screening process. In this way, when the number of screened target landmarks reaches the threshold, while the screening continues, the data of this part of the screened target landmarks are stored in the storage medium of the vehicle computer. That is, segmented screening and storage, in this way, can ensure the reliability of the screened data, and avoid the problem of loss of screened data due to interruption of collection and / or power failure of the vehicle computer.

[0103] S205 . Perform plane fitting on the target landmark point according to the three-dimensional coordinates of the target landmark point to obtain ground parameters.

[0104] For example, any plane fitting algorithm may be used to perform plane fitting on the target landmark point according to the three-dimensional coordinates of the target landmark point. The plane fitting algorithm mentioned here may be, for example, a RANSAC algorithm or a least squares fitting algorithm.

[0105] Exemplarily, for example, formula (2) may be used to obtain ground parameters corresponding to the fitting plane. The specific implementation method may be related to the method of ground parameterization.

[0106] S206: Determine whether the number of target landmark points included in the fitting plane is greater than or equal to a preset number.

[0107] Exemplarily, the distance between the target landmark point and the fitting plane can be calculated to determine whether the target landmark point is located in the fitting plane. That is, the distance from the point to the plane is used to determine whether the point is located in the plane. For example, when the absolute value of the distance is less than or equal to the set distance threshold, it can be considered that the target landmark point is in the fitting plane, and when the absolute value of the distance is greater than the set distance threshold, it can be considered that the target landmark point is not in the fitting plane.

[0108] If the number of target landmark points included in the fitting plane is greater than or equal to the preset number, it means that the fitting plane is valid, and then S209 is executed. If the number of target landmark points included in the fitting plane is less than the preset number, it means that the fitting plane is invalid, and then S207 is executed.

[0109] S207: Perform VIO processing on the data in the map data that has not been subjected to VIO processing to obtain new landmarks.

[0110] S208: Acquire the target landmark point from the new landmark points.

[0111] Regarding how to obtain the target landmark point from the new landmark point, please refer to the description of S203-S204.

[0112] After executing step S208, the process returns to executing step S205 to perform plane fitting based on the re-acquired target landmark points.

[0113] S209: Use the ground parameters corresponding to the fitting plane as initial ground parameters.

[0114] For how to obtain ground parameters based on a plane, please refer to the implementation methods of the prior art.

[0115] S210, obtaining a depth error corresponding to the target landmark point according to initial ground parameters, the three-dimensional coordinates of the target landmark point, pixel coordinates, camera intrinsic parameters of the acquisition device, and a three-dimensional coordinate error function.

[0116] For example, the initial ground parameters, the three-dimensional coordinates of the target landmark point, the pixel coordinates, and the camera intrinsic parameters of the acquisition device can be substituted into the three-dimensional coordinate error function to obtain the error term error. Then, the error term error is modulo-ed and projected to the depth direction of the image where the target landmark point is located to obtain the depth error corresponding to the target landmark point.

[0117] S211 , filter out target landmark points whose depth errors are greater than a depth error threshold.

[0118] Exemplarily, the depth error threshold may be 3, for example.

[0119] The depth error can be used to filter out landmark points with too large depth errors. In addition, the error function is positively correlated with the distance of the landmark point from the camera optical center to a certain extent. Therefore, by filtering out landmark points with too large depth errors, it is possible to prevent subsequent calibration results from being overly dependent on distant landmark points, thereby ensuring the quality of the obtained target landmark points and thus ensuring the robustness of subsequent calibration.

[0120] S212, obtaining the three-dimensional observation coordinates of the retained target landmark point according to the pixel coordinates of the retained target landmark point, the camera intrinsic parameters of the acquisition device, and the initial ground parameters.

[0121] The retained target landmark points mentioned here are the target landmark points remaining after filtering out the target landmark points whose depth errors are greater than the depth error threshold.

[0122] For example, the relationship between the 3D observation coordinates in the above formula (3) can be used: Get the 3D observation coordinates of the reserved target landmark point.

[0123] Optionally, the pixel coordinates of the target landmark point used here may be the pixel coordinates of the target landmark point obtained for the first time during the VIO processing, or may be multiple pixel coordinates of the target landmark point obtained based on multiple images, or all pixel coordinates, etc., which can be set according to actual needs.

[0124] S213, obtaining the three-dimensional coordinate error of the retained target landmark point according to the three-dimensional coordinate of the retained target landmark point, the three-dimensional observation coordinate and the three-dimensional coordinate error function.

[0125] For example, the above formula (3) can be used to obtain the value of the error term error.

[0126] S214. Using the Gauss-Newton method, determine whether the three-dimensional coordinate error meets a preset error convergence condition.

[0127] The above error convergence condition can be set according to actual needs, for example, the error can be less than or equal to a preset value, and / or the number of iterations reaches a preset number of iterations, etc. Regarding how to use the Gauss-Newton method to determine whether the three-dimensional coordinate error meets the preset error convergence condition, reference can be made to the method of using the Gauss-Newton method to optimize the error term in the error function in the prior art, which will not be repeated here.

[0128] If the preset error convergence condition is not met, it means that the accuracy of the currently calibrated ground parameters is insufficient, and step S215 is executed. If the preset error convergence condition is met, it means that the accuracy of the calibrated ground parameters meets the standard, and step S216 is executed.

[0129] S215: Correct the initial ground parameters of the acquisition equipment to obtain corrected initial ground parameters.

[0130] The corrected initial ground parameters are used as new initial ground parameters, and the process returns to step S210.

[0131] S216: Use the corrected initial ground parameters as ground parameters for calibration of acquisition equipment.

[0132] At this point, the ground parameters of the acquisition equipment have been calibrated once.

[0133] In this embodiment, since the ground parameter initialization uses the selected landmark points, the accuracy of the ground parameters obtained by initialization is relatively high. Therefore, the ground parameters of the calibration acquisition device can be obtained by executing a calibration process. In this way, the ground parameter calibration efficiency can be further improved while ensuring the accuracy of the ground parameters.

[0134] The second method is to first use the data corresponding to a single frame image to initialize the ground parameters, then filter the data used for calibration, and then calibrate the ground parameters.

[0135] Figure 4 A flow chart of another method for calibrating ground parameters provided in an embodiment of the present application is shown below. Figure 4 As shown, the method may include:

[0136] S301. Acquire map data collected by a collection device when the collection device is driving on a target road.

[0137] S302: Perform VIO processing on the map data to obtain intermediate processing data.

[0138] The intermediate processing data includes: the 3D coordinates of the landmark points on the road surface of the target road, the pixel coordinates, the distance from the landmark points to the optical center of the camera, the speed and angular velocity corresponding to the landmark points when the camera photographs them, and the image to which the landmark points belong. The image to which the landmark points belong can be represented by the semantic recognition label of the landmark points.

[0139] S303. According to the image to which the landmark point belongs, obtain the first target landmark point from the landmark points in the same frame of image.

[0140] S304: Perform plane fitting based on the three-dimensional coordinates of the first target landmark point to obtain initial ground parameters of the acquisition device.

[0141] For example, the first target landmark points from the same frame image can be screened from the landmark points according to the image to which the landmark points belong. When the number of first target landmark points from the same frame image screened out reaches a set number, these first target landmark points can be used for plane fitting to obtain ground parameters. The plane fitting algorithm mentioned here can be, for example, a RANSAC algorithm, or a least squares fitting algorithm, etc.

[0142] The ground parameters corresponding to the fitting plane are obtained, and the number of first target landmark points included in the fitting plane is obtained according to the distance between the first target landmark point and the fitting plane. That is, the distance from the point to the plane is used to determine whether the point is located in the plane, thereby obtaining the number of first target landmark points included in the fitting plane.

[0143] If the number of first target landmark points included in the fitting plane is greater than or equal to the preset number, it means that the fitting plane is valid, and the ground parameters corresponding to the fitting plane are used as the initial ground parameters. If the number of first target landmark points included in the fitting plane is less than the preset number, it means that the fitting plane is invalid, and the first target landmark points from the same frame image are re-acquired from the landmark points to initialize the ground parameters, until a fitting plane is obtained in which the number of first target landmark points included is greater than or equal to the preset number.

[0144] That is, when the fitting plane obtained by using the first target landmark point of the current frame image is invalid, the first target landmark point of the next frame image is found from the result after VIO processing to perform plane fitting processing until the obtained fitting plane is valid. It should be understood that the next frame image mentioned here may refer to the first image after the current frame image and including the first target landmark points reaching the set number, and is not necessarily an image frame adjacent to the current image.

[0145] S305: Obtain the positioning reliability of the landmark points other than the first target landmark point from the map data.

[0146] It should be understood that the landmark points other than the first target landmark point mentioned here may be landmark points located after the first target landmark point during VIO processing.

[0147] Regarding how to obtain the positioning reliability, please refer to the description of step S204, the implementation method is similar and will not be described in detail.

[0148] S306: Acquire, from the landmark points other than the first target landmark point, landmark points whose positioning confidence is greater than a first confidence threshold.

[0149] S307. Obtain the depth error corresponding to the landmark point whose positioning confidence is greater than the first confidence threshold according to the initial ground parameters, the three-dimensional coordinates of the landmark point whose positioning confidence is greater than the first confidence threshold, the pixel coordinates, the camera intrinsic parameters, and the three-dimensional coordinate error function.

[0150] Regarding how to obtain the depth error corresponding to the landmark point, please refer to the description of step S210, which will not be repeated here.

[0151] S308. Obtain a second target landmark point from the landmark points whose positioning confidence is greater than the first confidence threshold according to the distance from the landmark point to the optical center of the camera, the speed and angular velocity corresponding to when the camera photographs the landmark point, and the depth error.

[0152] For example, the distance from the landmark point to the optical center of the camera, the speed corresponding to the landmark point, the angular velocity, and the depth error each correspond to a screening condition. Therefore, from the landmark points whose positioning confidence is greater than the first confidence threshold, the landmark points that meet the corresponding screening conditions can be selected as the second target landmark points.

[0153] Exemplarily, the screening condition of the depth error may be less than or equal to 3 (i.e., the depth error threshold). Regarding the screening conditions corresponding to the distance from the landmark point to the optical center of the camera, the speed corresponding to the landmark point, and the angular velocity, as well as the corresponding technical effects, please refer to the description of the aforementioned step S204, which will not be repeated here.

[0154] The depth error can be used to filter out landmark points with too large depth errors. In addition, the error function is positively correlated with the distance of the landmark point from the camera optical center to a certain extent. Therefore, by filtering out landmark points with too large depth errors, it is possible to prevent subsequent calibration results from being overly dependent on distant landmark points, thereby ensuring the quality of the obtained second target landmark points and thus ensuring the robustness of subsequent calibration.

[0155] As for how many second target landmarks are screened, it can be set according to actual needs. When the number of the screened second target landmarks reaches a set value, the subsequent ground parameter initialization process can be executed.

[0156] It should be understood that when the vehicle computer uses the acquisition equipment to collect map data to implement the calibration of ground parameters, a smaller threshold can be set during the screening process. In this way, when the number of the screened second target landmarks reaches the threshold, while the screening continues, the data of the screened second target landmarks are stored in the storage medium of the vehicle computer. That is, segmented screening and storage, in this way, can ensure the reliability of the screened data, and avoid the problem of loss of the screened data due to interruption of collection and / or power failure of the vehicle computer.

[0157] S309, using the Gauss-Newton method to obtain ground parameters calibrated by the acquisition device according to the three-dimensional coordinates, pixel coordinates, camera intrinsic parameters, initial ground parameters, and three-dimensional coordinate error function of the second target landmark point.

[0158] That is, a calibration process is performed on the ground parameters of the acquisition device. For example, the three-dimensional observation coordinates of the second target landmark point are obtained according to the pixel coordinates of the second target landmark point, the camera internal parameters of the acquisition device, and the initial ground parameters. The three-dimensional coordinate error of the second target landmark point is obtained according to the three-dimensional coordinates, three-dimensional observation coordinates, and three-dimensional coordinate error function of the second target landmark point; the Gauss-Newton method is used to determine whether the three-dimensional coordinate error meets the preset error convergence condition. If not, the initial ground parameters of the acquisition device are corrected to obtain the corrected initial ground parameters; the corrected initial ground parameters are used as new initial ground parameters, and the process returns to step S305 until the three-dimensional coordinate error meets the preset error convergence condition.

[0159] S310: Determine whether the calibrated ground parameters make the three-dimensional coordinate error of the second target landmark point less than or equal to a preset value, and / or the number of calibrations reaches a preset number.

[0160] If at least one condition is satisfied, S312 is executed. If none of the conditions are satisfied, S311 is executed.

[0161] S311, performing VIO processing on the data in the map data that has not been subjected to VIO, and obtaining new landmarks.

[0162] S312: Using the calibrated ground parameters, obtain a new second target landmark point from the new landmark points.

[0163] For example, continuing to acquire the second target landmark point using the calibrated ground parameters means acquiring the depth error of the new landmark point using the calibrated ground parameters, and then acquiring the second target landmark point from the new landmark points whose positioning confidence is greater than the first confidence threshold based on the depth error of the acquired new landmark point, the positioning confidence, the distance from the landmark point to the optical center of the camera, the speed and angular velocity corresponding to the camera when photographing the landmark point.

[0164] Use the newly acquired second target landmark point and the previously acquired second target landmark point to return to S309. That is, when the acquired second target landmark point reaches the set value, first use the acquired second target landmark point for calibration. Then, according to the calibrated ground parameters, calculate the depth error of the landmark point to improve the acquisition accuracy of the depth error, and then be able to select the second target landmark point from the landmark points based on the accurate depth error for calibration of the ground parameters.

[0165] That is, starting from the second calibration cycle, the second target landmark points used in each calibration include the second target landmark points used in the previous calibration cycle in addition to the new second target landmark points obtained by using the ground parameters of the previous calibration. That is, by continuously expanding and improving the data range used for calibrating ground parameters, the calibration accuracy of ground parameters can be improved.

[0166] Optionally, the number of new second target landmarks obtained using the ground parameters of the last calibration may be consistent with the number required to trigger the first calibration, or may be less than or greater than the number, which may be set according to actual needs.

[0167] Optionally, as the calibration cycle increases, the screening condition corresponding to the depth error used when screening the second target landmark point can be continuously changed. The screening condition of the depth error can be less than or equal to the depth error threshold. Then, as the calibration cycle increases, the depth error threshold corresponding to the depth error used when screening the second target landmark point can be continuously reduced. Taking the first calibration cycle and the second calibration cycle as an example, the depth error threshold used in the first calibration cycle can be greater than the depth error threshold used in the second calibration cycle.

[0168] Optionally, in this embodiment, the pixel coordinates of the second target landmark point used in each calibration cycle of the iterative loop calibration may be the pixel coordinates of the second target landmark point obtained for the first time during the VIO processing. Of course, in this implementation, it is also possible to choose to use multiple pixel coordinates of the second target landmark point obtained based on multiple images, or all pixel coordinates, etc., which can be set according to actual needs.

[0169] S313: Use the calibrated ground parameters as ground parameters calibrated by the acquisition device.

[0170] At this point, the calibration of the ground parameters of the acquisition equipment is completed.

[0171] In this embodiment, the ground parameters can be calibrated by iterative cycle calibration, which can improve the calibration accuracy and robustness of the ground parameters. Compared with the offline calibration method of the calibration equipment in the prior art, the calibration efficiency of the ground parameters can be improved. In addition, the calibration method of the present application does not require manual participation, so it can save labor costs.

[0172] Optionally, in some embodiments, after the ground parameters of the acquisition device are calibrated, it is possible to further detect whether the calibrated ground parameters are valid.

[0173] For example, the following three items can be obtained:

[0174] 1. The absolute value of the error between the preset ground parameters of the acquisition equipment and the calibrated ground parameters.

[0175] The absolute value of the error between the two ground parameters may include the ground normal vector The absolute value of the error between , and the absolute value of the error between the vertical distance d from the camera to the ground.

[0176] The above-mentioned preset ground parameters may be obtained based on historical experience, or may be obtained based on calibrated ground parameters of the same type of vehicle.

[0177] 2. The average value of the depth error corresponding to the target landmark points used to calibrate the ground parameters.

[0178] 3. The proportion of target landmark points whose positioning confidence is greater than a second confidence threshold among the target landmark points used for calibrating ground parameters, wherein the second confidence threshold is greater than the first confidence threshold.

[0179] For the first method, the target landmarks mentioned in points 2 and 3 are the target landmarks used in the calibration process after the ground parameters are initialized. For the second method, the target landmarks mentioned in points 2 and 3 are the second target landmarks.

[0180] If the absolute value of the error between the ground parameters is within the corresponding first preset interval, the average value of the depth error is less than or equal to the first preset error threshold, and the proportion is greater than or equal to the preset proportion, the calibrated ground parameters are determined to be valid. Otherwise, if at least one of the conditions is not met, the calibrated ground parameters are determined to be invalid.

[0181] As mentioned above, the absolute value of the error between two ground parameters can include the ground normal vector The absolute value of the error between the ground normal vector and the absolute value of the error between the vertical distance d from the camera to the ground. Therefore, the first preset interval mentioned here may include the ground normal vector The interval range corresponding to the absolute value of the error between , and the interval range corresponding to the absolute value of the error between the vertical distance d from the camera to the ground.

[0182] If the calibrated ground parameters are invalid, the calibrated ground parameters and the corresponding map data are cleared, the acquisition device is instructed to re-acquire the map data, and the ground parameters are re-calibrated using the re-acquired map data.

[0183] If the calibrated ground parameters are valid, the calibrated ground parameters can be applied to the scene using the ground parameters.

[0184] For example, when the execution subject of the above calibration process is the server, the server can use the calibrated ground parameters and the corresponding map data to make maps, and / or use the calibrated ground parameters to detect whether the previously calibrated ground parameters are abnormal. When the execution subject of the above calibration process is the vehicle computer, the vehicle computer can return the map data and the calibrated ground parameters to the server for subsequent map making. And / or, the vehicle computer can use the calibrated ground parameters to detect whether the previously calibrated ground parameters are abnormal.

[0185] Exemplarily, the calibrated ground parameters may be used to detect whether the previously calibrated ground parameters are abnormal in the following manner.

[0186] For example, the following three items can be obtained:

[0187] 1. The average value of the depth error corresponding to the target landmark points used by the acquisition device to calibrate the ground parameters.

[0188] 2. The absolute value of the error between the ground parameters calibrated before and the ground parameters calibrated this time.

[0189] 3. The distance error between the ground represented by the ground parameters previously calibrated by the acquisition device and the target landmark point used for this calibration of the ground parameters.

[0190] For the first method, the target landmarks mentioned in points 2 and 3 are the target landmarks used in the calibration process after the ground parameters are initialized. For the second method, the target landmarks mentioned in points 2 and 3 are the second target landmarks.

[0191] If the average value of the depth error is less than or equal to the second preset error threshold, the absolute value of the error between the ground parameters is within the corresponding second preset interval, and the distance error is less than or equal to the preset distance error threshold, then it is determined that the ground parameters previously calibrated by the acquisition device are normal; conversely, if at least one of the conditions is not satisfied, then it is determined that the ground parameters previously calibrated by the acquisition device are abnormal. That is, if the average value of the depth error is greater than the second preset error threshold, and / or the absolute value of the error between the ground parameters is not within the corresponding second preset interval, and the distance error is greater than the preset distance error threshold, then it is determined that the ground parameters previously calibrated by the acquisition device are abnormal. When it is determined that the ground parameters previously calibrated by the acquisition device are abnormal, an operation of recalibrating the previously calibrated ground parameters can be triggered.

[0192] As mentioned above, the absolute value of the error between two ground parameters can include the ground normal vector The absolute value of the error between the ground normal vector and the absolute value of the error between the vertical distance d from the camera to the ground. Therefore, the second preset interval mentioned here may include the ground normal vector The interval range corresponding to the absolute value of the error between , and the interval range corresponding to the absolute value of the error between the vertical distance d from the camera to the ground.

[0193] By using the calibrated ground parameters to detect whether the previously calibrated ground parameters are abnormal, it is possible to determine whether there were calibration errors or whether the acquisition equipment has been moved, etc., so that the ground parameters can be calibrated in time to ensure the accuracy of the ground parameters used, thereby ensuring the robustness of the ground parameter calibration and the timeliness of the update.

[0194] It should be noted that in the aforementioned embodiments of the present application, all contents involving the use of thresholds or intervals for corresponding judgments or screening, the thresholds, set values ​​or intervals used can be set according to actual needs.

[0195] The ground parameter calibration method of the embodiment of the present application is illustrated below by using a specific example.

[0196] Figure 5 A flow chart of another ground parameter calibration method provided in an embodiment of the present application is shown as follows: Figure 5 As shown, the method includes:

[0197] S401. The vehicle computer (also understood as an on-board computing device) obtains map data collected by a collection device on the vehicle when the vehicle is driving on a target road.

[0198] S402: The vehicle computer performs VIO processing on the map data to obtain intermediate processing data.

[0199] S403: The vehicle computer screens the landmarks based on the intermediate processed data to obtain the target landmarks.

[0200] For example, the positioning confidence corresponding to the landmark point is obtained from the map data; and based on the distance from the landmark point to the optical center of the camera, the speed and angular velocity corresponding to the landmark point, the target landmark point is obtained from the landmark points whose positioning confidence is greater than a preset first confidence threshold.

[0201] S404: The vehicle computer determines whether the number of target landmarks reaches a set value. If yes, execute S406; if no, execute S405.

[0202] S405, the vehicle computer saves the screened target landmarks in sections, and continues to execute S403 to continue screening the target landmarks.

[0203] S406: The vehicle computer initializes ground parameters.

[0204] That is, plane fitting is performed based on the three-dimensional coordinates of the target landmark points to obtain the initial ground parameters of the acquisition equipment.

[0205] S407: The vehicle computer determines whether the ground parameter initialization is successful.

[0206] If yes, execute S408 , if no, return to execute S403 to reacquire the target landmark point.

[0207] For example, whether the initialization of the ground parameters is successful can be determined by determining whether the number of target landmark points included in the fitted plane is greater than or equal to a preset number.

[0208] S408: The vehicle computer uses the target landmarks used when initializing the ground parameters and the initial ground parameters to calibrate the ground parameters.

[0209] For example, according to the three-dimensional coordinates and pixel coordinates of the target landmark point, the camera intrinsic parameters of the acquisition device, the initial ground parameters, and the three-dimensional coordinate error function, the ground parameters of the acquisition device are calibrated using the Gauss-Newton method to obtain the calibrated ground parameters of the acquisition device.

[0210] S409: The vehicle computer determines whether the calibrated ground parameters are valid.

[0211] If yes, it is determined that the calibration is finished. That is, the calibration is completed. If no, S410 is executed.

[0212] S410: The vehicle computer clears the calibration data and returns to S403 to filter the target landmarks based on the VIO intermediate processing data of the new map data.

[0213] The calibration data mentioned here may include: calibrated ground parameters, and / or, the selected target landmark points, and / or, map data corresponding to the selected target landmark points.

[0214] The ground parameter calibration method, device and equipment provided by the present application can obtain the data of the landmark points on the road surface of the target road based on the intermediate processed data obtained by VIO processing of the map data collected by the acquisition device when driving on the target road, and then can automatically calibrate the ground parameters based on the data of the landmark points. Compared with the offline calibration method using the calibration equipment in the prior art, the calibration efficiency of the ground parameters can be improved. In addition, the calibration method of the present application does not require manual participation, so it can save labor costs.

[0215] Figure 6 A schematic diagram of the structure of a ground parameter calibration device provided in an embodiment of the present application is shown in FIG. Figure 6 As shown, the device includes: an acquisition module 11, a first processing module 12, a second processing module 13, a third processing module 14, and a calibration module 15. Optionally, the device may further include the following modules: a fourth processing module 16 and / or a sending module 17.

[0216] The acquisition module 11 is used to acquire the map data collected by the acquisition device when driving on the target road;

[0217] A first processing module 12 is used to perform visual inertial odometer (VIO) processing on the map data to obtain intermediate processing data, wherein the intermediate processing data at least includes: three-dimensional coordinates and pixel coordinates of landmark points located on the road surface of the target road;

[0218] A second processing module 13 is used to obtain a target landmark point from the landmark points;

[0219] The third processing module 14 is used to perform plane fitting based on the three-dimensional coordinates of the target landmark point to obtain the initial ground parameters of the acquisition device;

[0220] The calibration module 15 is used to calibrate the ground parameters of the acquisition device according to the three-dimensional coordinates, pixel coordinates, the initial ground parameters, and the three-dimensional coordinate error function constructed based on the ground parameters of the target landmark point; the three-dimensional coordinate error function is used to obtain the error between the three-dimensional coordinates of the target landmark point and the three-dimensional observation coordinates, and the three-dimensional observation coordinates are the three-dimensional coordinates of the target landmark point obtained by using the ground parameters and the pixel coordinates corresponding to the target landmark point.

[0221] In a possible implementation, the acquisition device integrates a camera, and the intermediate processing data also includes: the distance from the landmark point to the optical center of the camera, and the speed and angular velocity corresponding to the landmark point when the camera photographs the landmark point. The second processing module 13 is specifically used to: obtain the positioning confidence of the landmark point from the map data; and obtain the target landmark point from the landmark points whose positioning confidence is greater than the first confidence threshold according to the distance from the landmark point to the optical center of the camera, and the speed and angular velocity corresponding to the landmark point when the camera photographs the landmark point.

[0222] In this implementation, optionally, the third processing module 14 is specifically used to: perform plane fitting on the target landmark points according to the three-dimensional coordinates of the target landmark points to obtain ground parameters; if the number of target landmark points included in the fitting plane is less than a preset number, perform VIO processing on the data that has not been VIO processed in the map data to obtain new landmark points; obtain the target landmark points from the new landmark points, and return to the step of performing plane fitting until the number of target landmark points included in the fitting plane is greater than or equal to the preset number, and the ground parameters corresponding to the fitting plane are used as the initial ground parameters.

[0223] In this implementation, optionally, the calibration module 15 is specifically used to: obtain the depth error corresponding to the target landmark point according to the initial ground parameters, the three-dimensional coordinates of the target landmark point, the pixel coordinates, the camera intrinsic parameters of the acquisition device, and the three-dimensional coordinate error function; filter out the target landmark points whose depth errors are greater than the depth error threshold; obtain the three-dimensional observation coordinates of the target landmark point according to the pixel coordinates of the retained target landmark point, the camera intrinsic parameters of the acquisition device and the initial ground parameters; obtain the three-dimensional coordinate error of the retained target landmark point according to the three-dimensional coordinates of the retained target landmark point, the three-dimensional observation coordinates and the three-dimensional coordinate error function; use the Gauss-Newton method to determine whether the three-dimensional coordinate error meets the preset error convergence condition, if not, correct the initial ground parameters of the acquisition device to obtain the corrected initial ground parameters; use the corrected initial ground parameters as the new initial ground parameters, return to the step of obtaining the depth error, until the three-dimensional coordinate error meets the preset error convergence condition, and use the corrected initial ground parameters as the ground parameters calibrated by the acquisition device.

[0224] In a possible implementation, the acquisition device integrates a camera, the intermediate processed data further includes: an image to which the landmark point belongs, and the target landmark point includes: a first target landmark point and a second target landmark point.

[0225] The second processing module 13 is specifically configured to obtain, from the landmark points, a first target landmark point from the same frame of image according to the image to which the landmark point belongs.

[0226] The third processing module 14 is specifically used to perform plane fitting based on the three-dimensional coordinates of the first target landmark point to obtain initial ground parameters of the acquisition device.

[0227] The second processing module 13 is further configured to obtain a second target landmark point from the landmark points other than the first target landmark point.

[0228] For example, the intermediate processing data also includes: the distance from the landmark point to the optical center of the camera, and the speed and angular velocity corresponding to the landmark point when the camera photographs the landmark point. The second processing module 13 is specifically used to obtain the positioning confidence of the landmark points other than the first target landmark point from the map data; obtain the landmark points whose positioning confidence is greater than the first confidence threshold from the landmark points other than the first target landmark point; obtain the depth error corresponding to the landmark points whose positioning confidence is greater than the first confidence threshold according to the initial ground parameters, the three-dimensional coordinates and pixel coordinates of the landmark points whose positioning confidence is greater than the first confidence threshold, the camera internal parameters of the acquisition device, and the three-dimensional coordinate error function; obtain the second target landmark point from the landmark points whose positioning confidence is greater than the first confidence threshold according to the distance from the landmark point whose positioning confidence is greater than the first confidence threshold to the optical center of the camera, the speed and angular velocity corresponding to the landmark point when the camera photographs the landmark point, and the depth error.

[0229] The calibration module 15 is specifically used to calibrate the ground parameters of the acquisition device according to the three-dimensional coordinates, pixel coordinates, the initial ground parameters, and the three-dimensional coordinate error function of the second target landmark point.

[0230] For example, the calibration module 15 is specifically used to: obtain the ground parameters calibrated by the acquisition device using the Gauss-Newton method according to the three-dimensional coordinates, pixel coordinates, camera intrinsic parameters of the acquisition device, the initial ground parameters, and the three-dimensional coordinate error function of the second target landmark point; perform VIO processing on the data that has not been VIO processed in the map data to obtain new landmark points, and use the calibrated ground parameters to obtain new second target landmark points from the new landmark points; use the newly obtained second target landmark points and the previously obtained second target landmark points to return to the step of calibrating the ground parameters of the acquisition device until the calibrated ground parameters make the three-dimensional coordinate error of the second target landmark point less than or equal to a preset value, and / or, when the number of calibrations reaches a preset number, use the calibrated ground parameters as the ground parameters calibrated by the acquisition device.

[0231] A possible implementation method is that the fourth processing module 16 is used to detect whether the calibrated ground parameters are valid after the calibration module 15 calibrates the ground parameters of the acquisition device; if the calibrated ground parameters are valid, the map data and the calibrated ground parameters are transmitted back through the sending module 17; and / or, the calibrated ground parameters are used to detect whether the ground parameters previously calibrated by the acquisition device are abnormal; if the calibrated ground parameters are invalid, the acquisition device is instructed to re-collect the map data.

[0232] For example, the fourth processing module 16 is specifically used to obtain the absolute value of the error between the ground parameters preset by the acquisition device and the calibrated ground parameters, the average value of the depth errors corresponding to the target landmark points used for calibrating the ground parameters, and the proportion of target landmark points used for calibrating the ground parameters whose positioning confidence is greater than a second confidence threshold; the second confidence threshold is greater than the first confidence threshold; if the absolute value of the error between the ground parameters is within the corresponding first preset interval, the average value of the depth error is less than or equal to the first preset error threshold, and the proportion is greater than or equal to the preset ratio, then it is determined that the calibrated ground parameters are valid.

[0233] For example, the fourth processing module 16 is specifically used to obtain the average value of the depth error corresponding to the target landmark point used by the acquisition device to calibrate the ground parameters before, the absolute value of the error between the ground parameters calibrated before and the ground parameters calibrated this time, and the distance error between the ground represented by the ground parameters calibrated before and the target landmark point used to calibrate the ground parameters this time; if the average value of the depth error is less than or equal to the second preset error threshold, the absolute value of the error between the ground parameters is within the corresponding second preset interval, and the distance error is less than or equal to the preset distance error threshold, then it is determined that the ground parameters previously calibrated by the acquisition device are normal; if the average value of the depth error is greater than the second preset error threshold, and / or the absolute value of the error between the ground parameters is not within the corresponding second preset interval, and the distance error is greater than the preset distance error threshold, then it is determined that the ground parameters previously calibrated by the acquisition device are abnormal.

[0234] The ground parameter calibration device provided in the embodiment of the present application can execute the ground parameter calibration method in the above method embodiment, and its implementation principle and technical effect are similar, which will not be repeated here. Figure 6 The division of the modules shown is only a schematic diagram, and the present application does not limit the division of the modules and the naming of the modules.

[0235] Figure 7 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. Figure 7As shown, the electronic device 20 may include: at least one processor 21 and a memory 22 .

[0236] The memory 22 is used to store programs. Specifically, the programs may include program codes, and the program codes include computer operation instructions.

[0237] The memory 22 may include a high-speed RAM memory, and may also include a non-volatile memory (non-volatile memory), such as at least one disk memory.

[0238] The processor 21 is used to execute the computer-executable instructions stored in the memory 22 to implement the ground parameter calibration method described in the above method embodiment. The processor 21 may be a central processing unit (CPU), or an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present application.

[0239] The electronic device 20 may further include a communication interface 23, so as to communicate and interact with an external device through the communication interface 23. When the electronic device 20 is a vehicle computer of a vehicle to which the acquisition terminal belongs, the external device may be, for example, a acquisition terminal and / or a service terminal. When the electronic device 20 is a service terminal, the external device may be, for example, a acquisition terminal and / or a vehicle computer of a vehicle to which the acquisition terminal belongs.

[0240] In specific implementation, if the communication interface 23, the memory 22 and the processor 21 are implemented independently, the communication interface 23, the memory 22 and the processor 21 can be connected to each other through the bus 24 and communicate with each other. The bus 24 can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The bus 24 can be divided into an address bus, a data bus, a control bus, etc., but it does not mean that there is only one bus or one type of bus.

[0241] Optionally, in a specific implementation, if the communication interface 23, the memory 22 and the processor 21 are integrated on a chip, the communication interface 23, the memory 22 and the processor 21 can communicate through an internal interface.

[0242] The present application also provides a computer-readable storage medium, which may include: a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a disk or an optical disk, and other media that can store program codes. Specifically, the computer-readable storage medium stores program instructions, and the program instructions are used for the methods in the above embodiments.

[0243] The present application also provides a program product, which includes an execution instruction, which is stored in a readable storage medium. At least one processor of the electronic device can read the execution instruction from the readable storage medium, and at least one processor executes the execution instruction so that the electronic device implements the ground parameter calibration method provided by the various embodiments described above.

[0244] The term "plurality" in this article refers to two or more than two. The term "and / or" in this article is only a description of the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the previous and next associated objects are in an "or" relationship; in the formula, the character " / " indicates that the previous and next associated objects are in a "division" relationship. In addition, it should be understood that in the description of this application, words such as "first" and "second" are only used for the purpose of distinguishing descriptions, and cannot be understood as indicating or implying relative importance, nor can they be understood as indicating or implying order.

[0245] It should be understood that the various numerical numbers involved in the embodiments of the present application are only used for the convenience of description and are not used to limit the scope of the embodiments of the present application.

[0246] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A ground parameter calibration method, characterized in that: The method comprises: Obtain map data collected by the collection device when driving on the target road; Performing visual inertial odometer (VIO) processing on the map data to obtain intermediate processing data, wherein the intermediate processing data at least includes: three-dimensional coordinates and pixel coordinates of landmark points located on the road surface of the target road; Acquire a target landmark point from the landmark points; Performing plane fitting based on the three-dimensional coordinates of the target landmark points to obtain initial ground parameters of the acquisition device; The ground parameters of the acquisition device are calibrated according to the three-dimensional coordinates, pixel coordinates, initial ground parameters and a three-dimensional coordinate error function constructed based on the ground parameters of the target landmark point; the three-dimensional coordinate error function is used to obtain the error between the three-dimensional coordinates of the target landmark point and the three-dimensional observation coordinates, and the three-dimensional observation coordinates are the three-dimensional coordinates of the target landmark point obtained by using the ground parameters and the pixel coordinates corresponding to the target landmark point.

2. The method according to claim 1, characterized in that The acquisition device integrates a camera, and the intermediate processing data also includes: the distance from the landmark point to the optical center of the camera, the speed and angular velocity corresponding to the landmark point when the camera photographs the landmark point, and obtaining the target landmark point from the landmark point includes: Obtaining the location reliability of the landmark point from the map data; According to the distance from the landmark point to the optical center of the camera, the speed and angular velocity corresponding to the camera photographing the landmark point, the target landmark point is obtained from the landmark points whose positioning confidence is greater than a first confidence threshold.

3. The method according to claim 2, characterized in that The performing plane fitting based on the three-dimensional coordinates of the target landmark point to obtain the initial ground parameters of the acquisition device includes: According to the three-dimensional coordinates of the target landmark points, plane fitting is performed on the target landmark points to obtain ground parameters; If the number of target landmark points included in the fitting plane is less than a preset number, performing VIO processing on the data in the map data that has not been subjected to VIO processing to obtain new landmark points; From the new landmark points, the target landmark points are obtained, and the step of performing plane fitting is returned to until the number of target landmark points included in the fitted plane is greater than or equal to the preset number, and the ground parameters corresponding to the fitted plane are used as the initial ground parameters.

4. The method according to claim 3, characterized in that The method of calibrating the ground parameters of the acquisition device according to the three-dimensional coordinates and pixel coordinates of the target landmark point, the initial ground parameters, and the three-dimensional coordinate error function constructed based on the ground parameters includes: Obtaining a depth error corresponding to the target landmark point according to the initial ground parameters, the three-dimensional coordinates of the target landmark point, the pixel coordinates, the camera intrinsic parameters of the acquisition device, and the three-dimensional coordinate error function; Filter out target landmark points whose depth error is greater than the depth error threshold; Obtaining the three-dimensional observation coordinates of the target landmark point according to the retained pixel coordinates of the target landmark point, the camera intrinsic parameters of the acquisition device and the initial ground parameters; Obtaining a three-dimensional coordinate error of the retained target landmark point according to the three-dimensional coordinates of the retained target landmark point, the three-dimensional observation coordinates and the three-dimensional coordinate error function; Using the Gauss-Newton method, determine whether the three-dimensional coordinate error meets a preset error convergence condition. If not, correct the initial ground parameters of the acquisition device to obtain corrected initial ground parameters. The corrected initial ground parameters are used as new initial ground parameters, and the step of obtaining the depth error is returned to execute until the three-dimensional coordinate error meets the preset error convergence condition, and the corrected initial ground parameters are used as the ground parameters calibrated by the acquisition device.

5. The method according to claim 1, characterized in that The acquisition device integrates a camera, the intermediate processing data further includes: an image to which the landmark point belongs, the target landmark point includes: a first target landmark point and a second target landmark point, and acquiring the target landmark point from the landmark points includes: According to the image to which the landmark point belongs, acquiring a first target landmark point from the same frame of image from the landmark point; The performing plane fitting based on the three-dimensional coordinates of the target landmark point to obtain the initial ground parameters of the acquisition device includes: Performing plane fitting based on the three-dimensional coordinates of the first target landmark point to obtain initial ground parameters of the acquisition device; The method further comprises: Acquire a second target landmark point from the landmark points other than the first target landmark point; The method of calibrating the ground parameters of the acquisition device according to the three-dimensional coordinates and pixel coordinates of the target landmark point, the initial ground parameters, and the three-dimensional coordinate error function constructed based on the ground parameters includes: The ground parameters of the acquisition device are calibrated according to the three-dimensional coordinates, pixel coordinates, the initial ground parameters, and the three-dimensional coordinate error function of the second target landmark point.

6. The method according to claim 5, characterized in that The intermediate processing data also includes: the distance from the landmark point to the optical center of the camera, the speed and angular velocity corresponding to the landmark point when the camera photographs the landmark point, and the acquiring of the second target landmark point from the landmark points other than the first target landmark point includes: Acquiring the positioning reliability of the landmark points other than the first target landmark point from the map data; Acquire, from the landmark points other than the first target landmark point, landmark points whose positioning confidence is greater than a first confidence threshold; Obtaining a depth error corresponding to a landmark point whose positioning confidence is greater than a first confidence threshold according to the initial ground parameters, the three-dimensional coordinates of the landmark point whose positioning confidence is greater than a first confidence threshold, the pixel coordinates, the camera intrinsic parameters of the acquisition device, and the three-dimensional coordinate error function; According to the distance from the landmark point whose positioning confidence is greater than the first confidence threshold to the optical center of the camera, the speed and angular velocity corresponding to when the camera photographs the landmark point, and the depth error, the second target landmark point is obtained from the landmark points whose positioning confidence is greater than the first confidence threshold.

7. The method according to claim 6, characterized in that The step of calibrating the ground parameters of the acquisition device according to the three-dimensional coordinates, pixel coordinates, the initial ground parameters, and the three-dimensional coordinate error function of the second target landmark point includes: According to the three-dimensional coordinates and pixel coordinates of the second target landmark point, the camera intrinsic parameters of the acquisition device, the initial ground parameters, and the three-dimensional coordinate error function, the ground parameters calibrated by the acquisition device are obtained using the Gauss-Newton method; Performing VIO processing on the data in the map data that has not been subjected to VIO processing to obtain new landmarks, and using the calibrated ground parameters to obtain new second target landmarks from the new landmarks; Use the newly acquired second target landmark point and the previously acquired second target landmark point to return to the step of calibrating the ground parameters of the acquisition device until the calibrated ground parameters make the three-dimensional coordinate error of the second target landmark point less than or equal to a preset value, and / or, when the number of calibrations reaches a preset number, the calibrated ground parameters are used as the ground parameters calibrated by the acquisition device.

8. The method according to any one of claims 1 to 7, characterized in that: After calibrating the ground parameters of the acquisition device, the method further includes: Check whether the calibrated ground parameters are valid; If the calibrated ground parameters are valid, the map data and the calibrated ground parameters are transmitted back; and / or, the calibrated ground parameters are used to detect whether the ground parameters previously calibrated by the acquisition device are abnormal; If the calibrated ground parameters are invalid, the acquisition device is instructed to re-acquire the map data.

9. The method according to claim 8, characterized in that The detecting whether the calibrated ground parameters are valid includes: Obtaining the absolute value of the error between the ground parameters preset by the acquisition device and the calibrated ground parameters, the average value of the depth errors corresponding to the target landmark points used for calibrating the ground parameters, and the proportion of the target landmark points used for calibrating the ground parameters whose positioning confidence is greater than a second confidence threshold; the second confidence threshold is greater than the first confidence threshold; If the absolute value of the error between the ground parameters is within the corresponding first preset interval, the average value of the depth error is less than or equal to the first preset error threshold, and the proportion is greater than or equal to the preset proportion, it is determined that the calibrated ground parameters are valid.

10. The method according to claim 9, characterized in that The using the calibrated ground parameters to detect whether the ground parameters previously calibrated by the acquisition device are abnormal includes: Obtaining the average value of the depth error corresponding to the target landmark point used by the acquisition device to calibrate the ground parameters previously, the absolute value of the error between the ground parameters calibrated previously by the acquisition device and the ground parameters calibrated this time, and the distance error between the ground represented by the ground parameters calibrated previously by the acquisition device and the target landmark point used for calibrating the ground parameters this time; If the average value of the depth error is less than or equal to the second preset error threshold, the absolute value of the error between the ground parameters is within the corresponding second preset interval, and the distance error is less than or equal to the preset distance error threshold, it is determined that the ground parameters previously calibrated by the acquisition device are normal; If the average value of the depth error is greater than a second preset error threshold, and / or the absolute value of the error between the ground parameters is not within the corresponding second preset interval, and the distance error is greater than the preset distance error threshold, it is determined that the ground parameters previously calibrated by the acquisition device are abnormal.

11. A ground parameter calibration device, characterized in that: The device comprises: An acquisition module is used to acquire map data collected by the acquisition device when driving on the target road; A first processing module is used to perform visual inertial odometer (VIO) processing on the map data to obtain intermediate processing data, wherein the intermediate processing data at least includes: three-dimensional coordinates and pixel coordinates of landmark points located on the road surface of the target road; A second processing module, used for acquiring a target landmark point from the landmark points; A third processing module is used to perform plane fitting based on the three-dimensional coordinates of the target landmark point to obtain the initial ground parameters of the acquisition device; A calibration module is used to calibrate the ground parameters of the acquisition device according to the three-dimensional coordinates of the target landmark point, the pixel coordinates, the initial ground parameters, and the three-dimensional coordinate error function constructed based on the ground parameters; the three-dimensional coordinate error function is used to obtain the error between the three-dimensional coordinates of the target landmark point and the three-dimensional observation coordinates, and the three-dimensional observation coordinates are the three-dimensional coordinates of the target landmark point obtained by using the ground parameters and the pixel coordinates corresponding to the target landmark point.

12. An electronic device, characterized in that: include: Processor and memory; The processor is communicatively connected to the memory; The memory stores computer instructions; The processor executes the computer instructions stored in the memory to implement the method according to any one of claims 1 to 10.

Images