Information completion method, device, computer equipment and storage medium

Abstract

The present application relates to a method, device, computer equipment and storage medium for completing the above information, which obtains millimeter-wave radar data of a first area collected by a millimeter-wave radar and image data of a second area collected by a visual sensor; inputs the millimeter-wave radar data into a preset first model to obtain first feature information of road objects in the first area; obtains pixel coordinates of road objects in the second area based on the image data, and obtains second feature information of road objects in the second area according to a second model corresponding to the pixel coordinates and the second scanning area. The above method can avoid abnormal operation of the system due to failure of the laser radar, thereby improving the reliability of the system.

Description

Technical Field

[0001] The present application relates to the field of sensor technology, and in particular to an information completion method, device, computer equipment and storage medium. Background Art

[0002] With the development of technologies such as automated driving and vehicle-road collaboration, sensors have been used more and more widely. In automated driving systems or vehicle-road collaboration systems, it is necessary to collect different types of feature information of target objects so that the system can make more accurate judgments based on the feature information.

[0003] In order to collect different types of feature information, the above system can deploy multiple sensors. LiDAR, visual sensors and millimeter wave radar can be deployed in the system at the same time. The size and depth information of road objects can be collected by LiDAR; the color features of road objects can be collected by visual sensors, and the speed and other features of road objects can be collected by millimeter wave radar.

[0004] However, when the lidar fails, the system cannot obtain characteristics such as the size and positioning of road objects, which may cause the automated driving system or the vehicle-road cooperative system to operate abnormally or even cause a safety accident. Summary of the invention

[0005] Based on this, it is necessary to provide an information completion method, device, computer equipment and storage medium that can improve system reliability in response to the above technical problems.

[0006] A method for completing information, the method comprising:

[0007] Acquire millimeter-wave radar data of a first area collected by the millimeter-wave radar and image data of a second area collected by the visual sensor; the first area is an overlapping area of ​​scans of the laser radar and the millimeter-wave radar, and the second area is an area of ​​the overlapping area of ​​scans of the laser radar and the visual sensor that does not intersect with the first area;

[0008] Inputting the millimeter wave radar data into a preset first model to obtain first feature information of the road object in the first area; the first model is obtained by training based on data collected by the millimeter wave radar and the laser radar in a time period and the same scene; the first feature information includes at least one of size information, category information and depth information;

[0009] Based on the image data, pixel coordinates of the road object in the second area are obtained, and second feature information of the road object in the second area is obtained according to the second model corresponding to the pixel coordinates and the second scanning area; the second model includes a correspondence between the pixel coordinates and the second feature information; the second feature information includes at least one of depth information, size information and category information.

[0010] In one embodiment, the first model is a deep learning model, the second model is a fitting mapping model, and before obtaining the millimeter-wave radar data of the first area collected by the millimeter-wave radar and the image data of the second area collected by the visual sensor, the method further includes:

[0011] Acquire historical point cloud data collected by the laser radar and second historical data collected by the second sensor in the same time period and the same scene; the second sensor is a millimeter wave radar or a visual sensor; the second historical data is historical millimeter wave radar data collected by the millimeter wave radar, or historical image data collected by the visual sensor;

[0012] Performing spatiotemporal synchronization processing on the historical point cloud data and the second historical data to obtain a spatiotemporal correspondence between the historical point cloud data and the second historical data;

[0013] Based on the spatiotemporal correspondence between the historical point cloud data and the second historical data, a deep learning model or a fitting mapping model is established.

[0014] In one embodiment, the historical point cloud data includes a plurality of first data frames, the second historical data includes a plurality of second data frames, and the historical point cloud data and the second historical data are subjected to spatiotemporal synchronization processing to obtain a spatiotemporal correspondence between the historical point cloud data and the second historical data, including:

[0015] Time synchronization is performed on the historical point cloud data and the second historical data to obtain a plurality of time-synchronized data frame pairs; each data frame pair includes a first data frame and a second data frame synchronized at a sampling time;

[0016] Coordinate system conversion is performed on the first data frame and the second data frame in each data frame pair to obtain spatially synchronized data pairs, each data pair including first data in the spatially synchronized first data frame and second data in the second data frame.

[0017] In one embodiment, the second sensor is a millimeter wave radar; the second historical data is historical millimeter wave radar data collected by the millimeter wave radar; and establishing the deep learning model based on the spatiotemporal correspondence between the historical point cloud data and the second historical data includes:

[0018] Perform feature extraction on the historical point cloud data to obtain first feature information corresponding to each first data;

[0019] Using the second data as a training input sample, and using the first feature information associated with the first data corresponding to the second data as a sample label of the training input sample, to obtain a training data set;

[0020] Using the second data as input of the initial deep learning model, using the first feature information associated with the first data as a reference output of the initial deep learning model, training the initial deep learning model, and obtaining a deep learning model;

[0021] In one embodiment, the second sensor is a visual sensor; the second historical data is historical image data collected by the visual sensor; and establishing the fitting mapping model based on the spatiotemporal correspondence between the historical point cloud data and the second historical data includes:

[0022] Perform feature extraction on historical point cloud data to obtain second feature information;

[0023] Perform feature extraction on the historical image data collected by the visual sensor to obtain pixel coordinates;

[0024] A fitting mapping model is established based on the correspondence between the second feature information corresponding to the first data and the pixel coordinates corresponding to the second data in each data pair.

[0025] In one embodiment, the relative positions of the laser radar and the visual sensor are fixed; the second feature information is the depth information of the road object in the second area; based on the correspondence between the second feature information corresponding to the first data and the pixel coordinates corresponding to the second data in each data pair, a fitting mapping model is established, including:

[0026] According to the coordinates of the second data, the depth information is mapped to corresponding pixel coordinates in the image data to obtain a depth image, so that some pixel coordinates on the depth image have depth information;

[0027] Curve fitting is performed on some pixel coordinates in the depth image to obtain a plurality of depth fitting curves, and the plurality of depth fitting curves are determined as a fitting mapping model.

[0028] In one embodiment, obtaining the second characteristic information of the road object in the second area according to the fitting mapping model corresponding to the pixel coordinates and the second scanning area includes:

[0029] In the depth image, determining whether pixel coordinates of the road object in the second area fall on a plurality of depth fitting curves;

[0030] If yes, the depth information corresponding to the pixel coordinates of the road object is determined as the depth information of the road object in the second area.

[0031] In one embodiment, the method further comprises:

[0032] If not, obtaining a plurality of target pixel coordinates around the pixel coordinates of the road object in the second area; the target pixel coordinates are pixel coordinates having depth information;

[0033] A preset interpolation algorithm is used to interpolate the depth information corresponding to each target pixel coordinate to obtain the depth information of the road object in the second area.

[0034] In one embodiment, the above-mentioned use of a preset interpolation algorithm to interpolate the depth information corresponding to each target pixel coordinate to obtain the depth information of the road object in the second area includes:

[0035] Calculate the distance between the pixel coordinates of the road object in the second area and the pixel coordinates of each target;

[0036] According to the distance value, the depth information corresponding to each target pixel coordinate is interpolated to obtain the depth information of the road object in the second area.

[0037] In one embodiment, the acquiring of a plurality of target pixel coordinates around the pixel coordinates of the road object in the second area includes:

[0038] In the depth image, determining two target depth fitting curves that are closest to pixel coordinates of the road object in the second area;

[0039] Determine a calibration line passing through pixel coordinates of the road object in the second area; the calibration line intersects the two target depth fitting curves;

[0040] The pixel coordinates of the intersection of the calibration line and the two target depth fitting curves are determined as the target pixel coordinates.

[0041] In one embodiment, performing coordinate system conversion on the first data frame and the second data frame in each data frame pair to obtain a spatially synchronized data pair includes:

[0042] According to a preset conversion matrix, the first data in the data frame pair is converted into the coordinate system where the second sensor is located, and the mapping coordinates of each first data in the coordinate system where the second sensor is located are obtained;

[0043] In the data frame pair, obtaining second data corresponding to the mapping coordinates;

[0044] The second data corresponding to each mapping coordinate and the first data corresponding to the mapping coordinate are determined as a data pair.

[0045] In one embodiment, the above-mentioned time synchronization of the historical point cloud data and the second historical data to obtain multiple time-synchronized data frame pairs includes:

[0046] Convert historical point cloud data and second historical data to the same timeline;

[0047] Under the time axis, obtain the first sampling time of each first data frame in the historical point cloud data, and the second sampling time of each second data frame in the second historical data;

[0048] Calculating the difference between the first sampling time and the second sampling time;

[0049] If the difference is less than a preset threshold, it is determined that the first data frame corresponding to the first sampling moment and the second data frame corresponding to the second sampling moment are a data frame pair.

[0050] In one embodiment, the sampling frequencies of the historical point cloud data and the second historical data are in a multiple relationship.

[0051] In one of the embodiments, the method further comprises: performing feature extraction on the image data to obtain category information of road objects in the second area.

[0052] An information supplement device, the device comprising:

[0053] An acquisition module is used to acquire millimeter-wave radar data of a first area collected by the millimeter-wave radar and image data of a second area collected by the visual sensor; the first area is an overlapping area of ​​scans of the laser radar and the millimeter-wave radar, and the second area is an area of ​​the overlapping area of ​​scans of the laser radar and the visual sensor that does not intersect with the first area;

[0054] An input module, used to input the millimeter wave radar data into a preset first model to obtain first feature information of road objects in the first area; the first model is obtained by training based on data collected by the millimeter wave radar and the laser radar in a time period and the same scene; the first feature information includes at least one of size information, category information and depth information;

[0055] A fitting module is used to obtain pixel coordinates of road objects in the second area based on the image data, and obtain second feature information of the road objects in the second area according to a second model corresponding to the pixel coordinates and the second scanning area; the second model includes a correspondence between the pixel coordinates and the second feature information; the second feature information includes at least one of depth information, size information and category information.

[0056] A computer device includes a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the above-mentioned information completion method when executing the computer program.

[0057] A computer-readable storage medium stores a computer program, which implements the steps of the above-mentioned information completion method when executed by a processor.

[0058] The above-mentioned information completion method, device, computer equipment and storage medium obtain millimeter-wave radar data of a first area collected by a millimeter-wave radar, and image data of a second area collected by a visual sensor; wherein the above-mentioned first area is a scanning overlapping area of ​​a laser radar and a millimeter-wave radar, and the above-mentioned second area is an area in the scanning overlapping area of ​​a laser radar and a visual sensor that does not intersect with the first area; then, the computer equipment inputs the millimeter-wave radar data into a preset first model to obtain first feature information of road objects in the first area; obtains pixel coordinates of road objects in the second area based on the image data, and obtains second feature information of road objects in the second area according to a second model corresponding to the pixel coordinates and the second scanning area; wherein the above-mentioned first model is obtained by training based on data collected by the millimeter-wave radar and the laser radar in a time period and the same scene; the first feature information includes at least one of size information, category information and depth information; the above-mentioned second model contains a correspondence between pixel coordinates and second feature information; the second feature information includes at least one of depth information, size information and category information. Since the above-mentioned first model is obtained through training based on data collected by millimeter-wave radar and lidar, and the second model contains the correspondence between pixel coordinates and second feature information; therefore, when the lidar fails, the computer device can obtain the millimeter-wave radar data in the first area where the lidar and the above-mentioned millimeter-wave radar overlap, and then obtain the first feature information of each road object in the first area based on the millimeter-wave radar data and the first model; and obtain the image data in the second area where the lidar and the above-mentioned visual sensor overlap, and obtain the second feature information of each road object in the second area according to the image data and the second model, so that when the lidar fails, the computer device can supplement the feature information of the road objects in the first area and the second area through the first model and the second model, thereby avoiding abnormal system operation caused by the failure of the lidar and improving the reliability of the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] Figure 1 is an application environment diagram of an information completion method in an embodiment;

[0060] Figure 2 is a schematic diagram of a flow chart of an information completion method in an embodiment;

[0061] Figure 2A is a schematic diagram of an information completion method in an embodiment;

[0062] Figure 3 is a schematic diagram of a flow chart of an information completion method in an embodiment;

[0063] Figure 4 is a flow chart of an information completion method in another embodiment;

[0064] Figure 5 is a flow chart of an information completion method in another embodiment;

[0065] Figure 6 is a flow chart of an information completion method in another embodiment;

[0066] Figure 7 is a flow chart of an information completion method in another embodiment;

[0067] Fig. 7A is a schematic diagram of an information completion method in another embodiment;

[0068] Figure 7B is a schematic diagram of an information completion method in another embodiment;

[0069] Figure 8 is a flow chart of an information completion method in another embodiment;

[0070] Fig. 8A is a schematic diagram of an information completion method in another embodiment;

[0071] Fig. 9 is a flow chart of an information completion method in another embodiment;

[0072] Fig.10 is a flow chart of an information completion method in another embodiment;

[0073] Fig.11 is a structural block diagram of an information completion device in another embodiment;

[0074] Fig.12 is a structural block diagram of an information completion device in another embodiment;

[0075] Fig.13 is a structural block diagram of an information completion device in another embodiment;

[0076] Fig.14 is a structural block diagram of an information completion device in another embodiment;

[0077] Fig.15 is a structural block diagram of an information completion device in another embodiment;

[0078] Fig.16 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0079] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0080] The information completion method provided in this application can be applied to Figure 1 In the application environment shown. The computer device 100 is connected to the laser radar 200, the visual sensor 300 and the millimeter wave radar 400. The laser radar 200, the visual sensor 300 and the millimeter wave radar 400 can be sensors in the roadside system or sensors in the vehicle system, which are not limited here. The computer device 10 can be implemented by an independent server or a server cluster composed of multiple servers.

[0081] In one embodiment, Figure 2 As shown, an information completion method is provided, which is applied to Figure 1 This method is used as an example to illustrate the computer device in the example. It is applicable to situations where the laser radar cannot be used to obtain information about certain areas, such as laser radar failure, laser radar itself perception blind area, etc. It includes:

[0082] S101, obtaining millimeter-wave radar data of a first area collected by the millimeter-wave radar and image data of a second area collected by the visual sensor; the first area is an overlapping scanning area of ​​the laser radar and the millimeter-wave radar, and the second area is an area in the overlapping scanning area of ​​the laser radar and the visual sensor that does not intersect with the first area.

[0083] Among them, the above-mentioned laser radar, millimeter wave radar, and visual sensor can be sensors in the roadside system or sensors in the vehicle-mounted system, which is not limited here. The above-mentioned laser radar can be but not limited to 8-line laser radar, 16-line laser radar, 32-line laser radar, 64-line laser radar, 128-line laser radar, etc. The above-mentioned millimeter wave radar can correspond to multiple working frequency bands, such as 77GHz, 24GHz, etc. The above-mentioned visual sensor can be an image data acquisition device or a monitoring camera, which is not limited here.

[0084] The above three different sensors are arranged in the above system, and different characteristic information of road objects can be obtained. For example, the computer device can obtain the speed information of the road object through the millimeter wave radar, the color information or category information of the road object through the visual sensor, and the size information, category information and depth information of the road object through the laser radar. The above road objects can be traffic participants on the road, such as vehicles or pedestrians; in addition, the above road objects can also be obstacles on the road, etc.; the types of the above road objects are not limited here. Among them, the above size information can be the size of the target recognition box corresponding to the road object; the above category information can include whether the road object is a small vehicle, a large vehicle, a pedestrian, a road equipment, etc.; the above depth information can refer to the distance or relative position between the road object and the sensor; the above color information can be RGB information or grayscale value corresponding to the road object, etc.

[0085] Each of the above sensors may have different scanning ranges. Figure 2A As shown, the laser radar can achieve 360-degree scanning, and the corresponding scanning range is as follows Figure 2A In area A, the scanning range of the millimeter-wave radar may be area B, the scanning range of the visual sensor may be area C, the scanning overlapping area of ​​the lidar and the millimeter-wave radar is the first area, and in the scanning overlapping area of ​​the lidar and the visual sensor, the area that does not intersect with the first area is the second area.

[0086] Optionally, the computer device can detect the laser radar to determine whether the laser radar has failed. The above failure can be an abnormal connection status between the laser radar and the computer device, or it can be an abnormal data returned by the laser radar, such as missing data or distorted data; the above failure mode is not limited here. When determining whether the laser radar has failed, the computer device can determine it based on the status monitoring log of the laser radar, or detect whether the data returned by the laser radar has been received. If the data returned by the laser radar is not received, it can be determined that the laser radar has failed; the above failure determination method is not limited here.

[0087] When the computer device determines that the laser radar fails, the millimeter-wave radar data of the first area collected by the millimeter-wave radar and the image data of the second area collected by the visual sensor can be obtained.

[0088] Specifically, when the computer device acquires millimeter-wave radar data and image data, it can acquire the above data through a wired connection or a wireless connection, which is not limited here. The computer device can acquire the millimeter-wave radar data sent by the millimeter-wave radar and the image data sent by the visual sensor in real time, or it can acquire the data returned by the millimeter-wave radar and the visual sensor based on the acquisition instruction after sending the acquisition instruction to the millimeter-wave radar and the visual sensor, which is not limited here.

[0089] The computer device can store preset ranges of the first area and the second area. After receiving the data sent by the millimeter-wave radar and the visual sensor, the computer device can extract the millimeter-wave radar data in the first area and the image data in the second area according to the above preset ranges.

[0090] S102, input millimeter-wave radar data into a preset first model to obtain first feature information of road objects in the first area; the first model is obtained by training based on data collected by millimeter-wave radar and laser radar in a time period and the same scene; the first feature information includes at least one of size information, category information and depth information.

[0091] The first feature information may include at least one of size information, category information, and depth information. In the present application, the millimeter wave radar itself does not have the ability to obtain the first feature information of the road object. The computer device trains the first model based on the data collected by the millimeter wave radar and the lidar in the same time period and the same scene, so that the first model obtains the correlation between the millimeter wave radar data and the point cloud data, so that the first feature information of the road object in the first area can be obtained according to the millimeter wave radar data.

[0092] The above-mentioned first model can be a deep learning model; wherein, the above-mentioned deep learning model can be a supervised learning model, an unsupervised learning model, or a semi-supervised learning model; the above-mentioned deep learning model can be a convolutional neural network model or a recurrent neural network model, and the type of deep learning model is not limited here.

[0093] The scene for collecting the above historical millimeter wave radar data and historical point cloud data can be the scene where the current road object is located, or other scenes, which are not limited here. The above deep learning model can be applied only to the same scene, or can be migrated to other scenes for use.

[0094] Specifically, after the computer device obtains the millimeter-wave radar data, the millimeter-wave radar data can be directly input into the deep learning model, or the millimeter-wave radar data can be pre-processed and then input into the deep learning model to obtain the first feature information of the road object in the first area, which is not limited here.

[0095] S103. Obtain pixel coordinates of the road object in the second area based on the image data, and obtain second feature information of the road object in the second area according to a second model corresponding to the pixel coordinates and the second scanning area; the second model includes a correspondence between the pixel coordinates and the second feature information; the second feature information includes at least one of depth information, size information and category information.

[0096] In the present application, the visual sensor itself does not have the ability to obtain the second characteristic information of the road object. The computer device establishes a second model based on the historical data collected by the laser radar and the visual sensor for the current road scene in the same time period, so that the second model has the ability to associate the pixel coordinates in the current road scene with the second characteristic information, so that the second characteristic information of the road object in the second area can be obtained based on the image data collected by the visual sensor.

[0097] The second model may be a fitting mapping model, and the fitting mapping model may be a nonlinear relationship model, a linear relationship model, or a feature association table, and the type of the fitting mapping model is not limited herein. The fitting mapping model may associate multiple features in the pixel coordinates with the second feature information, or may associate a single feature in the pixel coordinates with the second feature information, and is not limited herein. For example, the fitting mapping relationship may also be a square relationship, a cubic relationship, etc. between the feature values ​​of the pixel coordinates and the feature values ​​of the second feature information.

[0098] Specifically, after the computer device obtains the pixel coordinates of the road object in the second area, it can obtain the second feature information of the road object in the second area according to the pixel coordinates of the road object and the fitting mapping model.

[0099] Furthermore, the computer device may perform feature extraction on the image data to obtain category information of the road objects in the second area.

[0100] The above-mentioned information completion method is applicable to situations where laser radar cannot be used to obtain information of certain areas. First, the millimeter-wave radar data of the first area collected by the millimeter-wave radar and the image data of the second area collected by the visual sensor are obtained; wherein the above-mentioned first area is the scanning overlapping area of ​​the laser radar and the millimeter-wave radar, and the above-mentioned second area is the area in the scanning overlapping area of ​​the laser radar and the visual sensor that does not intersect with the first area; then, the computer device inputs the millimeter-wave radar data into a preset first model to obtain the first feature information of the road object in the first area; based on the image data, the pixel coordinates of the road object in the second area are obtained, and according to the second model corresponding to the pixel coordinates and the second scanning area, the second feature information of the road object in the second area is obtained; wherein the above-mentioned first model is obtained by training based on the data collected by the millimeter-wave radar and the laser radar in a time period and the same scene; the first feature information includes at least one of size information, category information and depth information; the above-mentioned second model contains the correspondence between the pixel coordinates and the second feature information; the second feature information includes depth information and / or size information. Since the above-mentioned first model is obtained through training based on data collected by millimeter-wave radar and lidar, and the second model contains the correspondence between pixel coordinates and second feature information; therefore, when the lidar cannot obtain information of certain areas, the computer device can obtain the millimeter-wave radar data in the first area where the lidar and the above-mentioned millimeter-wave radar overlap, and then obtain the first feature information of each road object in the first area based on the millimeter-wave radar data and the first model; and obtain the image data in the second area where the lidar and the above-mentioned visual sensor overlap, and obtain the second feature information of each road object in the second area according to the image data and the second model, so that when the lidar fails, the computer device can supplement the feature information of the road objects in the first area and the second area through the first model and the second model, so as to avoid abnormal system operation caused by the failure of the lidar, and improve the reliability of the system.

[0101] Figure 3 is a flow chart of an information completion method in another embodiment, wherein the first model is a deep learning model, and the second model is a fitting mapping model. This embodiment relates to a method for obtaining a deep learning model and a fitting mapping model. Based on the above embodiment, Figure 3 As shown, the above method also includes:

[0102] S201. Acquire historical point cloud data collected by the laser radar and second historical data collected by a second sensor in the same time period and the same scene; the second sensor is a millimeter wave radar or a visual sensor; the second historical data is historical millimeter wave radar data collected by the millimeter wave radar, or historical image data collected by the visual sensor.

[0103] In one of the embodiments, the relative positions of the above-mentioned millimeter-wave radar, laser radar and visual sensor are fixed, for example, the laser radar, millimeter-wave radar and visual sensor can be installed in the same fixture. Specifically, the computer device can obtain the historical millimeter-wave radar data and historical point cloud data in the first area collected by the millimeter-wave radar and the laser radar in the same time period and the same scene in the database according to the collection time period and the collection scene. In addition, the computer device can also obtain the historical point cloud data and historical image data collected by the laser radar and the visual sensor in the same time period and the same scene in the database according to the collection time period and the collection scene. The above-mentioned historical point cloud data and historical image data can be data in the second area, or data in the scanning overlapping area of ​​the visual sensor and the laser radar.

[0104] S202: Performing spatiotemporal synchronization processing on the historical point cloud data and the second historical data to obtain a spatiotemporal correspondence between the historical point cloud data and the second historical data.

[0105] After obtaining the second historical data and the historical point cloud data, in order to obtain a deep learning model and a fitting mapping model, the computer device needs to perform spatiotemporal synchronization processing on the second historical data and the historical point cloud data.

[0106] Specifically, the computer device may first perform time synchronization on the second historical data and the historical point cloud data, and then perform space synchronization; or may first perform space synchronization on the second historical data and the historical point cloud data, and then perform time synchronization; the above-mentioned time-space synchronization processing method is not limited here. After the time-space synchronization processing is performed on the second historical data and the historical point cloud data, the obtained time-space correspondence relationship includes the correspondence between the second historical data and the historical point cloud data collected for the same target at the same time.

[0107] S203: Establish a deep learning model or a fitting mapping model based on the spatiotemporal correspondence between the historical point cloud data and the second historical data.

[0108] When the second sensor is a millimeter-wave radar and the second historical data is historical millimeter-wave radar data, after obtaining the spatiotemporal correspondence between the second historical data and the historical point cloud data, the computer device can obtain a training data set for the deep learning model based on the spatiotemporal correspondence. The training data set may include positive samples and negative samples. On the basis of obtaining the training data set, the computer device may perform model training based on the training data set to obtain a deep learning model. The computer device may perform data cleaning on each sample in the training data set, or may process each sample in the training data set to increase the number of samples. The sample processing method is not limited here.

[0109] When the second sensor is a visual sensor and the second historical data is historical image data, the computer device can establish a fitting mapping model based on the above-mentioned time-space correspondence. Specifically, when establishing the fitting mapping model, the computer device can substitute multiple sets of corresponding pixel coordinates and the second feature information into the initial model of the fitting mapping model, and establish a nonlinear model after obtaining the parameters of the fitting mapping model; or, the fitting mapping model can be a machine learning model, and the computer device can determine the model parameters of the machine learning model based on the multiple sets of corresponding pixel coordinates and the second feature information. The establishment process of the above-mentioned mapping model is not limited here. The above-mentioned fitting mapping model can also be a mapping curve or a mapping table. For example, the above-mentioned mapping curve can be obtained by fitting with the value of the pixel coordinate having a time-space correspondence as the horizontal coordinate and the value of the second feature information as the vertical coordinate. After the computer device obtains the pixel coordinates of the road object in the second area, it can search for the second feature information corresponding to the pixel coordinates according to the above-mentioned fitting mapping model and determine it as the second feature information of the road object in the second area.

[0110] In the above-mentioned information completion method, the computer device performs spatiotemporal synchronous processing on the second historical data and the historical point cloud data, thereby obtaining data collected by the second sensor and the lidar on the same target at the same time, and establishing a spatiotemporal correspondence between the second historical data and the historical point cloud data; further, through the above-mentioned spatiotemporal correspondence, the computer device obtains a deep learning model or a fitting mapping model, so that the feature information of the road objects in the first area and the second area can be completed through the deep learning model and the fitting mapping model, thereby avoiding abnormal system operation caused by failure of the lidar, and improving the reliability of the system.

[0111] Figure 4 FIG. 1 is a flow chart of an information completion method in another embodiment. This embodiment relates to a method for a computer device to perform spatiotemporal synchronization of historical point cloud data and second historical data. Based on the above embodiment, Figure 4 As shown, the historical point cloud data includes a plurality of first data frames, and the second historical data includes a plurality of second data frames. The above S202 includes:

[0112] S301 , time-synchronize the historical point cloud data and the second historical data to obtain a plurality of time-synchronized data frame pairs; each data frame pair includes a first data frame and a second data frame synchronized at sampling time.

[0113] The computer device may first synchronize the second historical data with the historical point cloud data in time, so as to correspond the second historical data and the historical point cloud data collected at the same time.

[0114] The second historical data includes multiple second data frames, and the historical point cloud data includes multiple first data frames. The computer device can determine whether the second data frame and the first data frame are time synchronized according to the acquisition time, and then determine the second data frame and the first data frame synchronized at the sampling time as a data frame pair.

[0115] Specifically, when the computer device performs time synchronization according to the sampling time, it can obtain the sampling time of each second data frame and each first data frame, and then compare the obtained sampling times to determine the synchronized data frames; in addition, the computer device can also first determine the first data frame synchronized with the first second data frame, and then calculate the positions of the first data frames synchronized with other second data frames in turn according to the sampling frequency of the second historical data and the historical point cloud data; the above-mentioned time synchronization method is not limited here.

[0116] The sampling frequency of the second historical data and the historical point cloud data may be the same or different. Optionally, the sampling frequency of the second historical data may be a multiple of the sampling frequency of the historical point cloud data. For example, the sampling frequency of the historical millimeter-wave radar data is twice the sampling frequency of the historical point cloud data, and the sampling frequency of the historical image data may be four times the sampling frequency of the historical point cloud data. After the computer device determines the data frame pair synchronized at the first sampling moment, it can quickly determine the remaining data frame pairs based on the multiple relationship of the sampling frequency, thereby improving the time synchronization efficiency of the second historical data and the historical point cloud data.

[0117] S302 , performing coordinate system conversion on the first data frame and the second data frame in each data frame pair to obtain spatially synchronized data pairs, each data pair including first data in the spatially synchronized first data frame and second data in the second data frame.

[0118] Each second data frame may include multiple second data, and the second data may be data of a certain sampling point in the second data frame, or data of sampling points in a certain area in the second data frame; in addition, the second data may also be millisecond historical data corresponding to a certain road object in the second data frame, which is not limited here. Each first data frame may include multiple first data; the first data may be data of a certain sampling point in the first data frame, or data of sampling points in a certain area in the first data frame; in addition, the first data may also be point cloud data corresponding to a certain road object in the first data frame, which is not limited here.

[0119] After the computer device obtains multiple data frame pairs synchronized with the sampling time, it can correspond the first data and the second data in each data frame pair to obtain the first data and the second data obtained by different sensors when collecting the same target.

[0120] Specifically, the computer device can perform spatial calibration by means of coordinate conversion, so that the data collected by the second sensor and the laser radar can be marked in the same coordinate system, thereby obtaining the corresponding relationship between the first data and the second data. The computer device can convert the first data to the coordinates where the second data is located, and can also convert the second data to the coordinates where the first data is located. In addition, the computer device can also convert both the first data and the second data to other coordinate systems, such as the earth coordinate system, etc.; the above-mentioned spatial calibration method is not limited here.

[0121] In the above information completion method, the computer equipment first synchronizes the time of the historical millimeter-wave radar data or the historical image data with the historical point cloud data, and then performs spatial correspondence, so that the spatiotemporal correspondence between the historical millimeter-wave radar data and the historical point cloud data, as well as the spatiotemporal correspondence between the historical image data and the historical point cloud data can be quickly established.

[0122] Figure 5 is a flow chart of an information completion method in another embodiment. This embodiment relates to a method for a computer device to obtain a deep learning model. The second sensor is a millimeter wave radar; the second historical data is historical millimeter wave radar data collected by the millimeter wave radar. Based on the above embodiment, Figure 5 As shown, the above S203 includes:

[0123] S401: extract features from historical point cloud data to obtain first feature information corresponding to each first data.

[0124] The computer device can perform feature extraction on the point cloud data to obtain first feature information corresponding to each first data in each point cloud data frame. Specifically, the computer device can input the historical point cloud data into a preset feature extraction model to obtain the first feature information corresponding to each first data through the feature extraction model.

[0125] S402: Use the second data as a training input sample, use the first feature information associated with the first data corresponding to the second data as a sample label of the training input sample, and obtain a training data set.

[0126] Furthermore, the computer device may use the second data as a training input sample, and use the first feature information associated with the first data corresponding to the second data in the above-mentioned spatiotemporal correspondence as a sample label of the training input sample to obtain a training data set.

[0127] Specifically, the computer device may obtain a training sample based on the first data and the second data in each data pair; or obtain a training sample based on the first data and the second data in some data pairs, which is not limited here. The computer device may set the training weight of the training sample based on the importance of the second data in each data pair in the historical millimeter wave radar data.

[0128] S403: Use the second data as input of an initial deep learning model, use the first feature information associated with the first data as a reference output of the initial deep learning model, train the initial deep learning model, and obtain a deep learning model.

[0129] Furthermore, the computer device may use the second data in the training data set as the input of the initial deep learning model, use the first feature information associated with the first data as the reference output of the initial deep learning model, train the initial deep learning model, and obtain the deep learning model. For example, the computer device may adjust the model's learning rate, sliding average coefficient, and other hyperparameters based on the reference output of the initial deep learning model to obtain the deep learning model.

[0130] For example, the computer device selects historical millimeter-wave radar data in a region in the second data frame as the second data, determines the first feature information of the point cloud data corresponding to the second data, and inputs the training sample set into the initial deep learning model after normalization, flipping, shuffling and other preprocessing; utilizes the downsampling, upsampling and multi-size feature splicing methods in the initial deep learning model to extract the multi-dimensional feature information of the historical millimeter-wave radar data to form multiple feature maps of different sizes; utilizes target detection to calculate the positioning loss, classification loss, foreground and background loss, etc.; and then uses the gradient descent algorithm to iterate the loss function so that the deep learning model is obtained when the loss function is lower than a certain threshold or reaches the iteration termination condition.

[0131] In the above information completion method, the computer device obtains the training data set through the correspondence between the first data and the second data, and trains the deep learning model, so that the deep learning model establishes the association between the historical millimeter-wave radar data and the historical point cloud data, so that the first feature information of the road object can be obtained according to the millimeter-wave radar data, thereby improving the stability and reliability of the system.

[0132] Figure 6 is a flow chart of an information completion method in another embodiment. This embodiment relates to a method for a computer device to obtain a fitting mapping model. The second sensor is a visual sensor; the second historical data is historical image data collected by the visual sensor. Based on the above embodiment, Figure 6 As shown, the above S203 includes:

[0133] S501. Perform feature extraction on historical point cloud data to obtain second feature information.

[0134] The computer device can perform feature extraction on the point cloud data to obtain second feature information corresponding to each first data in each point cloud data frame. Specifically, the computer device can input the historical point cloud data into a preset feature extraction model to obtain the second feature information corresponding to each first data through the feature extraction model.

[0135] S502: Extract features from historical image data collected by the visual sensor to obtain pixel coordinates.

[0136] Furthermore, the computer device may perform feature extraction on the historical image data collected by the visual sensor to obtain pixel coordinates corresponding to each second data.

[0137] S503: Establish a fitting mapping model based on the correspondence between the second feature information corresponding to the first data and the pixel coordinates corresponding to the second data in each data pair.

[0138] On the basis of the above steps, the computer device may establish a fitting mapping model based on the correspondence between the second feature information corresponding to the first data and the pixel coordinates corresponding to the second data in each data pair. For example, the computer device may perform curve fitting based on the second feature information corresponding to the first data and the pixel coordinates corresponding to the second data, and determine the obtained fitting curve as the fitting mapping model.

[0139] The above-mentioned information completion method, through the correspondence between the first data and the second data, the computer device establishes a fitting mapping model, so that the fitting mapping model has the ability to associate pixel coordinates with the second feature information, so that the second feature information of the road object can be obtained according to the pixel coordinates of the road object in the second area, thereby improving the stability and reliability of the system.

[0140] Figure 7 is a flow chart of an information completion method in another embodiment. This embodiment relates to a method for a computer device to obtain a fitting mapping model. On the basis of the above embodiment, the relative positions of the laser radar and the visual sensor are fixed; the second feature information is the depth information of the road object in the second area, such as Figure 7 As shown, the above S503 includes:

[0141] S601 . Map depth information to corresponding pixel coordinates in the image data according to coordinates of the second data to obtain a depth image, so that some pixel coordinates on the depth image have depth information.

[0142] Among them, the first data is the image data of the road object in the second area, and the second data is the point cloud data of the road object in the second area; the second characteristic information is the depth information of the road object in the second area. The pixel coordinates represent the position of each pixel point in the image data in the two-dimensional coordinate system where the image data is located, that is, the pixel coordinate system. The depth information is used to characterize the distance between the road object and the laser radar, and can be represented by the relative position coordinates of the road object and the laser radar. The relative position coordinates can be Cartesian coordinate values ​​or polar coordinate values; in addition, the depth information can also be represented by the distance value between the road object and the laser radar, which is not limited here.

[0143] For example, the coordinate system of the laser radar is the world coordinate system, and the conversion relationship between the above point cloud data and image data is:

[0144]

[0145] Among them, in this conversion relationship, is the internal parameter matrix of the laser radar, is the external parameter matrix of the visual sensor; where r 11 ,r 12 ,…r 33 Represents the rotation angle between the lidar and the visual sensor, and the above t1-t3 represents the translation amount between the lidar and the visual sensor. is the coordinate of the second data in the world coordinate system, is the coordinate of the second data in the pixel coordinate system.

[0146] Based on the correspondence between the first data and the second data obtained by the computer device, it can be determined which point cloud data each pixel coordinate in the image data corresponds to, thereby determining the depth information corresponding to each pixel coordinate. The computer device can set the corresponding pixel coordinates for each depth information, such as marking them on the image data, to obtain a depth image. The above-mentioned depth image can include pixel values ​​corresponding to each pixel coordinate, and each pixel value can also be binarized, and each pixel value can also be set to a preset value, such as setting them all to white, which is not limited here. Fig. 7A The depth image corresponding to a scene is shown as mapping the depth information onto the image data. Fig. 7A The points in represent pixel coordinates with depth information. For a road object in the figure, such as point M on a vehicle, the depth information mapped at point M in the above depth image can represent the distance between the vehicle and the lidar.

[0147] S602 , performing curve fitting connection on some pixel coordinates in the depth image to obtain a plurality of depth fitting curves, and determining the plurality of depth fitting curves as a fitting mapping model.

[0148] The density of the point cloud data obtained by the laser radar can be determined according to the number of laser radar lines. For example, the point cloud data obtained by a 64-line laser radar is denser than the point cloud data obtained by a 16-line laser radar. However, since the density of the point cloud data may not match the number of pixels in the image data, and there may be data loss during the laser radar scanning, the depth image obtained by the computer device may only have depth information on some pixel coordinates.

[0149] Based on the above depth image, the computer device can perform curve fitting connection on the partial pixel coordinates with depth information to obtain multiple depth fitting curves, such as Figure 7B As shown, some pixel coordinates in the depth image that originally did not have depth information obtain corresponding depth information, thereby enhancing the richness of the depth information on the depth image, so that the computer device can more accurately obtain the second feature information of the road object according to the fitting mapping relationship.

[0150] In addition, after obtaining the point cloud data, the computer device can also perform densification processing on the point cloud data, so that more depth fitting curves can be obtained. For example, a 16-line laser radar can obtain 16 depth fitting curves. After the point cloud data is densified, the computer device can obtain 32 or more depth fitting curves, further improving the accuracy of the fitting mapping relationship.

[0151] When the computer device performs curve fitting connection on the pixel coordinates with depth information, the depth information can be filtered according to the smoothness of the curve, and the pixel coordinates that are not successfully connected can be filtered out, so that the obtained depth fitting curve is smoother. The above-mentioned depth fitting curve can be a fitting circle or other types of curves, which are not limited here.

[0152] In the above-mentioned information completion method, the computer device obtains a depth image, and performs curve fitting connection on some pixel coordinates in the depth image, and determines the obtained multiple depth fitting curves as a fitting mapping relationship, so that after the computer device obtains the image data, it can determine the depth information corresponding to the pixel coordinates in the above-mentioned depth fitting curve according to the pixel coordinates of the road object in the image data, so that the depth information of the road object can be directly determined, thereby avoiding system operation abnormalities caused by the inability of the laser radar to obtain depth information, thereby improving the reliability of the system.

[0153] Figure 8 : is a flow chart of an information completion method in another embodiment. This embodiment relates to a method in which a computer device obtains second feature information of a road object in a second area according to pixel coordinates and a fitting mapping model. On the basis of the above embodiment, the above S103 includes:

[0154] S701: In a depth image, determine whether pixel coordinates of a road object in a second area fall on a plurality of depth fitting curves.

[0155] When establishing the fitting mapping relationship, the computer device may not necessarily cover all pixel coordinates, so after obtaining the pixel coordinates of the road object on the image data, it can first determine whether the pixel coordinates have completely corresponding depth information in the fitting mapping relationship. Specifically, the computer device can determine whether the pixel coordinates of the road object fall on the depth fitting curve in the depth image.

[0156] S702: If yes, determine the depth information corresponding to the pixel coordinates of the road object as the depth information of the road object in the second area.

[0157] The pixel coordinates of the road object may be pixel coordinates corresponding to the center point of the road object, or may be pixel coordinates corresponding to any point in the contour of the road object, which is not limited here.

[0158] If the pixel coordinates of the road object fall on the depth fitting curve, that is, the pixel coordinates of the road object have completely corresponding depth information, then the computer device can determine the depth information corresponding to the pixel coordinates as the depth information of the road object.

[0159] Of course, the pixel coordinates of the above road objects may not fall on multiple depth fitting curves, such as Fig. 8A , the computer can obtain multiple target pixel coordinates around the pixel coordinates of the road object in the second area; then, a preset interpolation algorithm is used to interpolate the depth information corresponding to each target pixel coordinate to obtain the depth information of the road object in the second area; wherein the target pixel coordinates are pixel coordinates with depth information. The number of the target pixel coordinates can be two or more, and this embodiment is mainly described by taking two target pixel coordinates as an example.

[0160] When acquiring multiple target pixel coordinates, the computer device calculates the distance between the pixel coordinates of the road object in the second area and the pixel coordinates on the depth fitting curve, and then determines a preset number of target pixel coordinates that are closer to the pixel coordinates of the road object in the second area based on the above distance; or the computer device can use the pixel coordinates of the road object in the second area as the center of the circle, and determine a circular area of ​​target pixel coordinates based on a preset radius, and all pixel coordinates with depth information in the circular area can be determined as target pixel coordinates; the method for determining the above target pixel coordinates is not limited here.

[0161] Optionally, the computer device can determine, in the depth image, two target depth fitting curves that are closest to the pixel coordinates of the road object in the second area; determine a calibration line that passes through the pixel coordinates of the road object in the second area; wherein the calibration line intersects with the two target depth fitting curves; and then, determine the pixel coordinates of the intersection of the calibration line and the two target depth fitting curves as the target pixel coordinates.

[0162] The calibration line is a line intersecting the two target depth fitting curves. The calibration line may be a line parallel to the coordinate axis or a normal to the depth fitting curve. The method for determining the calibration line is not limited herein.

[0163] Continue with Fig. 8A For example, the pixel coordinates of the road object in the second area in the figure are pixel coordinates A (x0, y0), and the two target depth fitting curves closest to pixel coordinate A are curve 1 and curve 2. The calibration line in the figure is a line perpendicular to the x-axis in the pixel coordinates of the depth image. The intersection points of the above calibration line with curve 1 and curve 2 are pixel coordinates B (x0, y1) and pixel coordinates C (x0, y2), respectively. The computer device can determine the above pixel coordinates B (x0, y1) and pixel coordinates C (x0, y2) as the target pixel coordinates.

[0164] Furthermore, when the computer device performs interpolation processing on the depth information corresponding to each target pixel coordinate, the distance value between the pixel coordinate of the road object in the second area and each target pixel coordinate can be calculated; then, according to each distance value, the depth information corresponding to each target pixel coordinate can be interpolated to obtain the depth information of the road object in the second area. Figure 7 For example, in the above target pixel coordinates, the depth information corresponding to the pixel coordinate B (x0, y1) is (X1, Y1, Z1), and the depth information corresponding to the pixel coordinate C (x0, y2) is (X2, Y2, Z2). The computer device can interpolate X1 and X2, Y1 and Y2, and Z1 and Z2 according to the ratio of y1-y0 and y0-y2 to obtain the depth information corresponding to the pixel coordinate A (x0, y0).

[0165] In the above-mentioned information completion method, after obtaining the pixel coordinate system of the road object in the second area, the computer device determines whether the pixel coordinates of the road object in the second area fall on multiple depth fitting curves. When the pixel coordinates fall on the depth fitting curve, the depth information corresponding to the pixel coordinates can be directly determined as the depth information of the road object, so that the computer device can quickly determine the depth information of the road object according to the pixel coordinates; further, when the pixel coordinates do not fall on the depth fitting curve, the computer device interpolates the depth information corresponding to each target pixel coordinate, so that the depth information of the road object in the second area obtained is more accurate.

[0166] Fig. 9 FIG. 1 is a flow chart of an information completion method in another embodiment. This embodiment relates to a method for a computer device to synchronize the second historical data with the historical point cloud data. Based on the above embodiment, Fig. 9 As shown, the above S301 includes:

[0167] S801, converting the historical point cloud data and the second historical data to the same time axis.

[0168] Millimeter-wave radar, visual sensor, and lidar are independent devices, and the time axes of the historical millimeter-wave radar data, historical image data, and historical point cloud data obtained may be different. For example, the time axis of the millimeter-wave radar is the Global Positioning System (GPS) time axis, while the time axis of the lidar is determined by the device itself, and there is a certain time axis difference. The computer device can convert the historical point cloud data and the second historical data to the same time axis so that the system can obtain the first data frame and the second data frame synchronized with the sampling time.

[0169] Specifically, the computer device can convert the historical point cloud data to the time axis of the second historical data, or convert the second historical data to the time axis of the historical point cloud data, or convert both the historical point cloud data and the second historical data to another time axis, for example, to the GPS time axis. The above conversion method is not limited here.

[0170] S802. Under the above time axis, obtain a first sampling time of each first data frame in the historical point cloud data and a second sampling time of each second data frame in the second historical data.

[0171] Furthermore, the computer device can obtain the first sampling moment of each first data frame and the second sampling moment of the second data frame. The above-mentioned first sampling moment can be the timestamp marked on the first data frame by the laser radar when collecting historical point cloud data, or the first sampling moment obtained according to the order of each first data frame and the start sampling moment. The method for obtaining the first sampling moment is not limited here. The above-mentioned second sampling moment can be the timestamp marked on the second data frame by the second sensor when collecting historical millimeter-wave radar data, or the second sampling moment obtained according to the order of each second data frame and the start sampling moment. The method for obtaining the second sampling moment is not limited here.

[0172] S803: Calculate the difference between the first sampling time and the second sampling time.

[0173] S804: If the difference is less than a preset threshold, determine that the first data frame corresponding to the first sampling moment and the second data frame corresponding to the second sampling moment are a data frame pair.

[0174] The computer device can calculate the absolute value of the difference between the first sampling moment and the second sampling moment; if the absolute value of the difference is less than a preset threshold, it is determined that the first data frame corresponding to the first sampling moment and the second data frame corresponding to the second sampling moment are a data frame pair.

[0175] In the above-mentioned information completion method, the computer device converts the historical point cloud data and the second historical data to the same time axis, so that the computer device can accurately synchronize the first data frame and the second data frame; further, the computer device obtains the absolute value of the difference between the first sampling moment and the second sampling moment, and when the absolute value of the above-mentioned difference is less than a preset threshold, it is determined that the first data frame and the second data frame are a data frame pair, thereby avoiding time synchronization failure caused by the first sampling moment and the second sampling moment not being completely the same due to differences in sampling frequency, etc., thereby improving the stability of the information completion process.

[0176] Fig.10 FIG. 1 is a flow chart of an information completion method in another embodiment. This embodiment relates to a method in which a computer device spatially corresponds the second historical data to the historical point cloud data. Based on the above embodiment, Fig.10 As shown, the above S302 includes:

[0177] S901. According to a preset conversion matrix, convert the first data in the data frame pair into the coordinate system where the second sensor is located, and obtain the mapping coordinates of each first data in the coordinate system where the second sensor is located.

[0178] The above-mentioned conversion matrix is ​​a matrix that converts the data collected by the laser radar into the coordinate system of the second sensor, which can be determined based on the relative posture between the laser radar and the second sensor. The above-mentioned relative posture can include the translation and rotation angle between the laser radar and the second sensor.

[0179] The above conversion matrix can be obtained by manual measurement by a staff member and then input into the computer device; it can also be obtained by automatic calibration of the computer device according to the current relative posture relationship between the laser radar and the second sensor, which is not limited here. The computer device converts the first data in the data pair to the coordinate system where the second sensor is located according to the preset conversion matrix, and the mapping coordinates of each first data in the coordinate system where the second sensor is located can be obtained.

[0180] When the first data is data of a sampling point, the mapping coordinates may be the coordinates of a sampling point; when the first data is a group of data, the mapping coordinates are a group of coordinates corresponding to each data in the first data.

[0181] S902: In the data frame pair, obtain second data corresponding to the mapping coordinates.

[0182] The computer device may determine the second data corresponding to each mapping coordinate in the second data frame according to each mapping coordinate. Specifically, the computer device may search for the second data corresponding to the mapping coordinate in the data pair; if the data pair does not contain the sampling point data corresponding to the mapping coordinate, the computer device may determine the second data corresponding to the mapping coordinate according to the distance between the mapping coordinate and the coordinate value corresponding to the data in the second data frame. For example, when the distance between the mapping coordinate and the coordinate value corresponding to the data in the second data frame is less than a preset distance threshold, the computer device may determine the data as the second data corresponding to the mapping coordinate.

[0183] S903: Determine the second data corresponding to each mapping coordinate and the first data corresponding to the mapping coordinate as a data pair.

[0184] Based on the above steps, the computer device determines the first data corresponding to each mapping coordinate and the second data corresponding to each mapping coordinate, and can associate the first data with the second data through the mapping coordinates, thereby establishing a corresponding relationship between the first data and the second data.

[0185] In the above information completion method, the computer device completes the spatial calibration of the data frame pair through the conversion matrix, so that the computer device can obtain the spatially corresponding data pairs, providing a data basis for the establishment of a deep learning model or a fitting mapping model.

[0186] It should be understood that although Figure 2-10The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. Moreover, Figure 2-10 At least part of the steps may include multiple steps or multiple stages. These steps or stages are not necessarily performed at the same time, but can be performed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed in turn or alternately with other steps or at least part of the steps or stages in other steps.

[0187] In one embodiment, Fig.11 As shown, an information completion device is provided, comprising: an acquisition module 10, an input module 20 and a fitting module 30, wherein:

[0188] The acquisition module 10 is used to acquire the millimeter-wave radar data of the first area collected by the millimeter-wave radar and the image data of the second area collected by the visual sensor; the first area is the scanning overlap area of ​​the laser radar and the millimeter-wave radar, and the second area is the area of ​​the scanning overlap area of ​​the laser radar and the visual sensor that does not intersect with the first area;

[0189] The input module 20 is used to input the millimeter wave radar data into a preset deep learning model to obtain first feature information of the road object in the first area; the deep learning model is obtained by training based on the data collected by the millimeter wave radar and the laser radar in a time period and the same scene; the first feature information includes at least one of size information, category information and depth information;

[0190] The fitting module 30 is used to obtain the pixel coordinates of the road object in the second area based on the image data, and obtain the second feature information of the road object in the second area according to the fitting mapping model corresponding to the pixel coordinates and the second scanning area; the fitting mapping model includes the correspondence between the pixel coordinates and the second feature information; the second feature information includes depth information and / or size information.

[0191] In one embodiment, based on the above embodiment, Fig.12 As shown, the above device further includes an establishment module 40, and the establishment module 40 includes:

[0192] The acquisition unit 401 is used to acquire historical point cloud data collected by the laser radar and second historical data collected by the second sensor in the same time period and the same scene; the second sensor is a millimeter wave radar or a visual sensor; the second historical data is historical millimeter wave radar data collected by the millimeter wave radar, or historical image data collected by the visual sensor;

[0193] A synchronization unit 402 is used to perform spatiotemporal synchronization processing on the historical point cloud data and the second historical data to obtain a spatiotemporal correspondence between the historical point cloud data and the second historical data;

[0194] The establishing unit 403 is used to establish a deep learning model or a fitting mapping model based on the spatiotemporal correspondence between the historical point cloud data and the second historical data.

[0195] In one embodiment, based on the above embodiment, the historical point cloud data includes a plurality of first data frames, and the second historical data includes a plurality of second data frames, such as Fig.13 As shown, the synchronization unit 402 includes:

[0196] The synchronization subunit 4021 is used to perform time synchronization on the historical point cloud data and the second historical data to obtain a plurality of time-synchronized data frame pairs; each data frame pair includes a first data frame and a second data frame synchronized at a sampling time;

[0197] The conversion subunit 4022 is used to perform coordinate system conversion on the first data frame and the second data frame in each data frame pair to obtain spatially synchronized data pairs, each data pair including first data in the spatially synchronized first data frame and second data in the second data frame.

[0198] In one embodiment, based on the above embodiment, the second sensor is a millimeter wave radar; the second historical data is historical millimeter wave radar data collected by the millimeter wave radar; the above establishment unit 403 is specifically used to: perform feature extraction on the historical point cloud data to obtain the first feature information corresponding to each first data; use the second data as a training input sample, and use the first feature information associated with the first data corresponding to the second data as the sample label of the training input sample to obtain a training data set; use the second data as the input of the initial deep learning model, and use the first feature information associated with the first data as the reference output of the initial deep learning model to train the initial deep learning model to obtain a deep learning model.

[0199] In one embodiment, based on the above embodiment, Fig.14 As shown, the second sensor is a visual sensor; the second historical data is historical image data collected by the visual sensor; the establishing unit 403 includes:

[0200] The first extraction subunit 4031 is used to extract features from the historical point cloud data to obtain second feature information;

[0201] The second extraction subunit 4032 is used to extract features from the historical image data collected by the visual sensor to obtain pixel coordinates;

[0202] The subunit 4033 is established to establish a fitting mapping model based on the correspondence between the second feature information corresponding to the first data and the pixel coordinates corresponding to the second data in each data pair.

[0203] In one embodiment, based on the above embodiment, the relative positions of the laser radar and the visual sensor are fixed; the second feature information is the depth information of the road object in the second area; a subunit 4033 is established specifically for: mapping the depth information to the corresponding pixel coordinates in the image data according to the coordinates of the second data to obtain a depth image so that some pixel coordinates on the depth image have depth information; performing curve fitting connection on some pixel coordinates in the depth image to obtain multiple depth fitting curves, and determining the multiple depth fitting curves as a fitting mapping model.

[0204] In one embodiment, based on the above embodiment, Fig.15 As shown, the fitting module 30 includes:

[0205] A first determining unit 301 is used to determine, in the depth image, whether the pixel coordinates of the road object in the second area fall on a plurality of depth fitting curves;

[0206] The second determining unit 302 is configured to determine the depth information corresponding to the pixel coordinates of the road object in the second area as the depth information of the road object in the second area when the pixel coordinates of the road object in the second area fall on a plurality of depth fitting curves.

[0207] In one embodiment, based on the above embodiment, the second determination unit 302 is specifically used to: when the pixel coordinates of the road object in the second area do not fall on multiple depth fitting curves, obtain multiple target pixel coordinates around the pixel coordinates of the road object in the second area; the target pixel coordinates are pixel coordinates with depth information; and use a preset interpolation algorithm to interpolate the depth information corresponding to each target pixel coordinate to obtain the depth information of the road object in the second area.

[0208] In one embodiment, based on the above embodiment, the second determination unit 302 is specifically used to: calculate the distance value between the pixel coordinates of the road object in the second area and the target pixel coordinates; and interpolate the depth information corresponding to each target pixel coordinate according to the distance value to obtain the depth information of the road object in the second area.

[0209] In one embodiment, based on the above embodiment, the second determination unit 302 is specifically used to: determine, in the depth image, two target depth fitting curves that are closest to the pixel coordinates of the road object in the second area; determine a calibration line passing through the pixel coordinates of the road object in the second area; the calibration line intersects the two target depth fitting curves; and determine the pixel coordinates of the intersection of the calibration line and the two target depth fitting curves as the target pixel coordinates.

[0210] In one embodiment, based on the above embodiment, the above conversion subunit 4022 is specifically used to: convert the first data in the data frame pair to the coordinate system where the second sensor is located according to a preset conversion matrix, and obtain the mapping coordinates of each first data in the coordinate system where the second sensor is located; in the data frame pair, obtain the second data corresponding to the mapping coordinates; and determine the second data corresponding to each mapping coordinate and the first data corresponding to the mapping coordinate as a data pair.

[0211] In one embodiment, based on the above embodiment, the synchronization subunit 4021 is specifically used to: convert the historical point cloud data and the second historical data to the same time axis; obtain the first sampling moment of each first data frame in the historical point cloud data and the second sampling moment of each second data frame in the second historical data on the time axis; calculate the difference between the first sampling moment and the second sampling moment; if the difference is less than a preset threshold, determine that the first data frame corresponding to the first sampling moment and the second data frame corresponding to the second sampling moment are a data frame pair.

[0212] In one embodiment, based on the above embodiment, the sampling frequencies of the historical point cloud data and the second historical data are in a multiple relationship.

[0213] In one embodiment, based on the above embodiment, the fitting module 30 is further used to: extract features from the image data to obtain category information of the road objects in the second area.

[0214] The information completion devices provided in the above embodiments can execute the above information completion method embodiments, and their implementation principles and technical effects are similar and will not be described in detail here.

[0215] For the specific definition of the information completion device, please refer to the definition of the information completion method above, which will not be repeated here. Each module in the above-mentioned information completion device can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0216] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Fig.16 As shown. The computer device includes a processor, a memory and a network interface connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store information completion data. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, an information completion method is implemented.

[0217] Those skilled in the art will understand that Fig.16 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0218] In one embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above-mentioned method embodiments when executing the computer program.

[0219] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0220] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory may include read-only memory (ROM), magnetic tape, floppy disk, flash memory or optical memory, etc. Volatile memory may include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM).

[0221] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0222] The above embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.

Claims

1. An information completion method, characterized in that: The method comprises: Acquire millimeter-wave radar data of a first area collected by the millimeter-wave radar, and image data of a second area collected by the visual sensor; the first area is an overlapping area of ​​scans of the laser radar and the millimeter-wave radar, and the second area is an area of ​​the overlapping area of ​​scans of the laser radar and the visual sensor that does not intersect with the first area; Inputting the millimeter-wave radar data into a preset first model to obtain first feature information of road objects in the first area; the first model is obtained by training based on data collected by the millimeter-wave radar and the laser radar in the same time period and the same scene; the first feature information includes at least one of size information, category information and depth information; obtaining pixel coordinates of the road object in the second area based on the image data, and obtaining second feature information of the road object in the second area according to the pixel coordinates and a second model corresponding to the second area; the second model includes a correspondence between the pixel coordinates and the second feature information; the second feature information includes at least one of depth information, size information and category information; The first model is a deep learning model, and the second model is a fitting mapping model; before acquiring the millimeter-wave radar data of the first area collected by the millimeter-wave radar and the image data of the second area collected by the visual sensor, the method further includes: Acquire historical point cloud data collected by the laser radar and second historical data collected by a second sensor in the same time period and the same scene; the second sensor is the millimeter wave radar or the visual sensor; the second historical data is the historical millimeter wave radar data collected by the millimeter wave radar, or the historical image data collected by the visual sensor; Performing spatiotemporal synchronization processing on the historical point cloud data and the second historical data to obtain a spatiotemporal correspondence between the historical point cloud data and the second historical data; Based on the spatiotemporal correspondence between the historical point cloud data and the second historical data, the deep learning model or the fitting mapping model is established.

2. The method according to claim 1, characterized in that The historical point cloud data includes a plurality of first data frames, the second historical data includes a plurality of second data frames, and the performing spatiotemporal synchronization processing on the historical point cloud data and the second historical data to obtain a spatiotemporal correspondence between the historical point cloud data and the second historical data includes: Performing time synchronization on the historical point cloud data and the second historical data to obtain a plurality of time-synchronized data frame pairs; each data frame pair includes a first data frame and a second data frame synchronized at a sampling time; Coordinate system conversion is performed on the first data frame and the second data frame in each data frame pair to obtain spatially synchronized data pairs, each data pair including first data in the first spatially synchronized data frame and second data in the second spatially synchronized data frame.

3. The method according to claim 2, characterized in that The second sensor is the millimeter wave radar; the second historical data is historical millimeter wave radar data collected by the millimeter wave radar; and the establishing of the deep learning model based on the spatiotemporal correspondence between the historical point cloud data and the second historical data includes: Performing feature extraction on the historical point cloud data to obtain first feature information corresponding to each of the first data; Using the second data as a training input sample, and using the first feature information associated with the first data corresponding to the second data as a sample label of the training input sample, to obtain a training data set; The second data is used as the input of an initial deep learning model, the first feature information associated with the first data is used as the reference output of the initial deep learning model, the initial deep learning model is trained, and the deep learning model is obtained.

4. The method according to claim 2, characterized in that: The second sensor is the visual sensor; the second historical data is historical image data collected by the visual sensor; and the establishing of the fitting mapping model based on the spatiotemporal correspondence between the historical point cloud data and the second historical data includes: Performing feature extraction on the historical point cloud data to obtain second feature information; Performing feature extraction on the historical image data collected by the visual sensor to obtain pixel coordinates; The fitting mapping model is established based on the corresponding relationship between the second feature information corresponding to the first data and the pixel coordinates corresponding to the second data in each of the data pairs.

5. The method according to claim 4, characterized in that The relative positions of the laser radar and the visual sensor are fixed; the second feature information is the depth information of the road object in the second area; the fitting mapping model is established based on the correspondence between the second feature information corresponding to the first data and the pixel coordinates corresponding to the second data in each of the data pairs, including: According to the coordinates of the second data, mapping the depth information to corresponding pixel coordinates in the image data to obtain a depth image, so that some pixel coordinates on the depth image have depth information; Curve fitting is performed on the partial pixel coordinates in the depth image to obtain a plurality of depth fitting curves, and the plurality of depth fitting curves are determined as the fitting mapping model.

6. The method according to claim 5, characterized in that The obtaining, according to the pixel coordinates and the fitting mapping model corresponding to the second area, second feature information of the road object in the second area comprises: In the depth image, determining whether pixel coordinates of road objects in the second area fall on the plurality of depth fitting curves; If so, the depth information corresponding to the pixel coordinates of the road object is determined as the depth information of the road object in the second area.

7. The method according to claim 6, characterized in that The method further comprises: If not, obtaining a plurality of target pixel coordinates around the pixel coordinates of the road object in the second area; the target pixel coordinates are pixel coordinates having depth information; A preset interpolation algorithm is used to interpolate the depth information corresponding to each of the target pixel coordinates to obtain the depth information of the road object in the second area.

8. The method according to claim 7, characterized in that The using a preset interpolation algorithm to perform interpolation processing on the depth information corresponding to each of the target pixel coordinates to obtain the depth information of the road object in the second area includes: Calculate the distance between the pixel coordinates of the road object in the second area and the pixel coordinates of each target; According to the distance value, interpolation processing is performed on the depth information corresponding to each of the target pixel coordinates to obtain the depth information of the road object in the second area.

9. The method according to claim 8, characterized in that The step of acquiring a plurality of target pixel coordinates around the pixel coordinates of the road object in the second area comprises: In the depth image, determining two target depth fitting curves that are closest to pixel coordinates of the road object in the second area; Determine a calibration line passing through the pixel coordinates of the road object in the second area; the calibration line intersects the two target depth fitting curves; The pixel coordinates of the intersection of the calibration line and the two target depth fitting curves are determined as the target pixel coordinates.

10. The method according to any one of claims 2 to 9, characterized in that: The step of performing coordinate system conversion on the first data frame and the second data frame in each data frame pair to obtain a spatially synchronized data pair comprises: According to a preset conversion matrix, the first data in the data frame pair is converted into the coordinate system where the second sensor is located, and the mapping coordinates of each of the first data in the coordinate system where the second sensor is located are obtained; In the data frame pair, obtaining second data corresponding to the mapping coordinates; The second data corresponding to each of the mapping coordinates and the first data corresponding to the mapping coordinates are determined as a data pair.

11. The method according to any one of claims 2 to 9, characterized in that: The step of performing time synchronization on the historical point cloud data and the second historical data to obtain a plurality of time-synchronized data frame pairs includes: Converting the historical point cloud data and the second historical data to the same time axis; Under the time axis, obtaining a first sampling time of each first data frame in the historical point cloud data, and a second sampling time of each second data frame in the second historical data; Calculating a difference between the first sampling time and the second sampling time; If the difference is smaller than a preset threshold, it is determined that the first data frame corresponding to the first sampling moment and the second data frame corresponding to the second sampling moment are a data frame pair.

12. The method according to claim 11, characterized in that The sampling frequencies of the historical point cloud data and the second historical data are in a multiple relationship.

13. The method according to claim 1, characterized in that The method further comprises: Feature extraction is performed on the image data to obtain category information of road objects in the second area.

14. An information completion device, characterized in that: The device comprises: An acquisition module, used to acquire millimeter-wave radar data of a first area collected by the millimeter-wave radar, and image data of a second area collected by the visual sensor; the first area is an overlapping area of ​​scans of the laser radar and the millimeter-wave radar, and the second area is an area of ​​the overlapping area of ​​scans of the laser radar and the visual sensor that does not intersect with the first area; an input module, configured to input the millimeter-wave radar data into a preset first model to obtain first feature information of road objects in the first area; the first model is obtained by training based on data collected by the millimeter-wave radar and the laser radar in the same time period and the same scene; the first feature information includes at least one of size information, category information and depth information; a fitting module, configured to obtain pixel coordinates of the road object in the second area based on the image data, and obtain second feature information of the road object in the second area according to a second model corresponding to the pixel coordinates and the second area; the second model includes a correspondence between the pixel coordinates and the second feature information; the second feature information includes at least one of depth information, size information and category information; The first model is a deep learning model, and the second model is a fitting mapping model; the device also includes an establishment module, which includes: An acquisition unit, used for acquiring historical point cloud data collected by the laser radar and second historical data collected by a second sensor in the same time period and the same scene; the second sensor is the millimeter wave radar or the visual sensor; the second historical data is the historical millimeter wave radar data collected by the millimeter wave radar, or the historical image data collected by the visual sensor; A synchronization unit, configured to perform spatiotemporal synchronization processing on the historical point cloud data and the second historical data to obtain a spatiotemporal correspondence between the historical point cloud data and the second historical data; An establishing unit is used to establish the deep learning model or the fitting mapping model based on the spatiotemporal correspondence between the historical point cloud data and the second historical data.

15. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 13 are implemented.

16. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 13 are implemented.

Images