Information collection method, device, computer equipment and storage medium

Abstract

The present application relates to the above-mentioned information collection method, device, computer equipment and storage medium, which obtains the first sensor data of the road object collected by the first sensor, and obtains the first feature information of the road object based on the first sensor data; obtains the second feature information of the road object under the perspective of the second sensor according to the first feature information of the road object and the fitting mapping model corresponding to the current road scene; the fitting mapping model includes the correspondence between the first feature information and the second feature information under the perspective of the second sensor, wherein the first feature information and the second feature information are of different types. The above-mentioned method can improve the system reliability and the flexibility of system sensor deployment.

Description

Technical Field

[0001] The present application relates to the field of sensor technology, and in particular to an information collection method, device, computer equipment and storage medium. Background Art

[0002] With the development of technologies such as automated driving and vehicle-road collaboration, sensors have been used more and more widely. In automated driving systems or vehicle-road collaboration systems, it is necessary to collect different types of feature information of target objects so that the system can make more accurate judgments based on the feature information.

[0003] In order to collect different types of feature information, the above system can deploy multiple sensors. For example, the system can deploy a camera to collect color features of the target object; the system can also deploy a lidar to collect distance features of the target object, etc.

[0004] However, when one of the sensors fails or the sensor installation is restricted, the system cannot obtain the corresponding type of feature information through the sensor, causing the automated driving system or the vehicle-road cooperative system to operate abnormally or even cause a safety accident. Summary of the invention

[0005] Based on this, it is necessary to provide an information collection method, device, computer equipment and storage medium that can improve system reliability in response to the above technical problems.

[0006] An information collection method, the method comprising:

[0007] Acquire first sensor data of the road object collected by the first sensor, and obtain first feature information of the road object based on the first sensor data;

[0008] According to the first feature information of the road object and the fitting mapping model corresponding to the current road scene, the second feature information of the road object under the perspective of the second sensor is obtained; the fitting mapping model includes the correspondence between the first feature information and the second feature information under the perspective of the second sensor, and the types of the first feature information and the second feature information are different.

[0009] In one embodiment, before obtaining the first sensor data of the road object collected by the first sensor, the method further includes:

[0010] Acquire first historical data collected by a first sensor and second historical data collected by a second sensor in the same time period and the same scene;

[0011] Performing spatiotemporal synchronization processing on first historical data collected by the first sensor and second historical data collected by the second sensor to obtain a spatiotemporal mapping relationship between the first historical data and the second historical data;

[0012] Extracting features from the first historical data to obtain first feature information, and extracting features from the second historical data to obtain second feature information;

[0013] Based on the spatiotemporal mapping relationship between the first historical data and the second historical data, the first feature information and the second feature information are fitted to establish a fitting mapping model.

[0014] In one embodiment, the first historical data includes a plurality of first data frames, the second historical data includes a plurality of second data frames, and the first historical data collected by the first sensor and the second historical data collected by the second sensor are subjected to spatiotemporal synchronization to obtain a spatiotemporal mapping relationship between the first historical data and the second historical data, including:

[0015] Performing time synchronization on the first historical data and the second historical data to obtain a plurality of time-synchronized data frame pairs; each data frame pair includes a first data frame and a second data frame synchronized at a sampling time;

[0016] Coordinate system conversion is performed on the first data frame and the second data frame in each data frame pair to obtain spatially synchronized data pairs, each data pair including first data in the spatially synchronized first data frame and second data in the second data frame.

[0017] Accordingly, based on the spatiotemporal mapping relationship between the first historical data and the second historical data, the first feature information and the second feature information are fitted to establish a fitting mapping model, including:

[0018] A fitting mapping model is established based on the correspondence between the first feature information corresponding to the first data and the second feature information corresponding to the second data in each data pair.

[0019] In one embodiment, the first sensor is an image sensor, the second sensor is a laser radar, the relative positions of the first sensor and the second sensor are fixed, the first data is a two-dimensional image of the current road scene, and the second data is point cloud data of the road object; the first feature information is the pixel coordinates of the road object in the image coordinate system of the two-dimensional image, and the second feature information is the depth information of the road object, and the depth information is used to characterize the distance between the road object and the laser radar;

[0020] Based on the correspondence between the first feature information corresponding to the first data and the second feature information corresponding to the second data in each data pair, a fitting mapping model is established, including:

[0021] According to the coordinates of the second data, the depth information is mapped to corresponding pixel coordinates in the two-dimensional image to obtain a depth image, so that some pixel coordinates on the depth image have depth information;

[0022] Curve fitting is performed on some pixel coordinates in the depth image to obtain a plurality of depth fitting curves, and the plurality of depth fitting curves are determined as a fitting mapping model.

[0023] In one embodiment, obtaining the second feature information of the road object under the perspective of the second sensor according to the first feature information of the road object and the fitting mapping model corresponding to the current road scene includes:

[0024] In the depth image, determining whether pixel coordinates of the road object fall on a plurality of depth fitting curves;

[0025] If so, the depth information corresponding to the pixel coordinates of the road object is determined as the depth information of the road object.

[0026] In one embodiment, the method further comprises:

[0027] If not, then obtaining multiple target pixel coordinates around the pixel coordinates of the road object; the target pixel coordinates are pixel coordinates with depth information;

[0028] The preset interpolation algorithm is used to interpolate the depth information corresponding to each target pixel coordinate to obtain the depth information of the road object.

[0029] In one embodiment, the above-mentioned use of a preset interpolation algorithm to interpolate the depth information corresponding to each target pixel coordinate to obtain the depth information of the road object includes:

[0030] Calculate the distance between the pixel coordinates of the road object and the pixel coordinates of each target;

[0031] According to the distance value, the depth information corresponding to each target pixel coordinate is interpolated to obtain the depth information of the road object.

[0032] In one embodiment, the step of obtaining a plurality of target pixel coordinates around a pixel coordinate of a road object includes:

[0033] In the depth image, determine two target depth fitting curves that are closest to the pixel coordinates of the road object;

[0034] Determine a calibration line passing through pixel coordinates of the road object; the calibration line intersects the two target depth fitting curves;

[0035] The pixel coordinates of the intersection of the calibration line and the two target depth fitting curves are determined as the target pixel coordinates.

[0036] In one embodiment, the coordinate system conversion of the first data frame and the second data frame in each data frame pair to obtain a spatially synchronized data pair includes:

[0037] According to a preset conversion matrix, the second data in the data frame pair is converted into the first sensor coordinate system to obtain the mapping coordinates of each second data in the first sensor coordinate system;

[0038] In the data frame pair, obtaining first data corresponding to the mapping coordinates;

[0039] The second data corresponding to each mapping coordinate and the first data corresponding to the mapping coordinate are determined as a data pair.

[0040] In one embodiment, the time synchronization of the first historical data and the second historical data to obtain a plurality of time-synchronized data frame pairs includes:

[0041] Convert the first historical data and the second historical data to the same time axis;

[0042] Under the time axis, obtaining the first sampling time of each first data frame in the first historical data and the second sampling time of each second data frame in the second historical data;

[0043] Calculating the difference between the first sampling time and the second sampling time;

[0044] If the difference is less than a preset threshold, it is determined that the first data frame corresponding to the first sampling moment and the second data frame corresponding to the second sampling moment are a data frame pair.

[0045] In one embodiment, the sampling frequencies of the first historical data and the second historical data are in a multiple relationship.

[0046] An information collection method, the method comprising:

[0047] According to the current sensor state, determine whether the sensor switching condition is met;

[0048] If the sensor switching condition is met, the current sensor is used as the second sensor, and the steps of the above information collection method are executed to collect information.

[0049] An information collection device, comprising:

[0050] an acquisition module, configured to acquire first sensor data of a road object collected by a first sensor, and obtain first feature information of the road object based on the first sensor data;

[0051] A fitting module is used to obtain second feature information of a road object under the perspective of a second sensor based on the first feature information of the road object and a fitting mapping model corresponding to a current road scene; the fitting mapping model includes a correspondence between the first feature information and the second feature information under the perspective of the second sensor, and the first feature information and the second feature information are of different types.

[0052] A computer device includes a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the above-mentioned information collection method when executing the computer program.

[0053] A computer-readable storage medium stores a computer program, which implements the steps of the above-mentioned information collection method when executed by a processor.

[0054] The above-mentioned information collection method, device, computer equipment and storage medium obtain the first sensor data of the road object collected by the first sensor, and obtain the first feature information of the road object based on the first sensor data; then, according to the first feature information of the road object and the fitting mapping model corresponding to the current road scene, obtain the second feature information of the road object under the perspective of the second sensor; wherein the fitting mapping model contains the correspondence between the first feature information and the second feature information under the perspective of the second sensor, and the first feature information and the second feature information are of different types. Since the fitting mapping model contains the correspondence between the first feature information and the second feature information under the perspective of the second sensor, after the computer equipment obtains the first feature information of the road object, it can obtain the second feature information of the road object according to the first feature information and the fitting mapping model, thereby avoiding the system operation abnormality caused by the second sensor being unable to obtain the second feature information, thereby improving the reliability of the system; or when there is a limit on the number or weight of sensors in the system, the first feature information and the second feature information of the road object can be obtained by deploying a first sensor, thereby improving the flexibility of system sensor deployment. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] Figure 1 An application environment diagram of an information collection method in an embodiment;

[0056] Figure 2 A schematic diagram of a flow chart of an information collection method in an embodiment;

[0057] Figure 3 A schematic diagram of a flow chart of an information collection method in an embodiment;

[0058] Figure 4 A schematic diagram of a flow chart of an information collection method in another embodiment;

[0059] Figure 5A schematic diagram of a flow chart of an information collection method in another embodiment;

[0060] Figure 6 is a schematic diagram of an information collection method in another embodiment;

[0061] Figure 7 is a schematic diagram of an information collection method in another embodiment;

[0062] Figure 8 A schematic diagram of a flow chart of an information collection method in another embodiment;

[0063] Fig. 9 is a schematic diagram of an information collection method in another embodiment;

[0064] Fig.10 A schematic diagram of a flow chart of an information collection method in another embodiment;

[0065] Fig.11 A schematic diagram of a flow chart of an information collection method in another embodiment;

[0066] Fig.12 A schematic diagram of a flow chart of an information collection method in another embodiment;

[0067] Fig.13 A schematic diagram of a flow chart of an information collection method in another embodiment;

[0068] Fig.14 is a structural block diagram of an information collection device in an embodiment;

[0069] Fig.15 It is a structural block diagram of an information collection device in another embodiment;

[0070] Fig.16 It is a structural block diagram of an information collection device in another embodiment;

[0071] Fig.17 It is a structural block diagram of an information collection device in another embodiment;

[0072] Fig.18 It is a structural block diagram of an information collection device in another embodiment;

[0073] Fig.19 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0074] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0075] The information collection method provided by this application can be applied to Figure 1 In the application environment shown. The computer device 10 is connected to the first sensor 200. The computer device 100 can be set in the first sensor 200 device, or it can be an independent device outside the first sensor 200; the first sensor 200 and the second sensor 300 can be sensors in the roadside system or sensors in the vehicle-mounted system, which is not limited here. The first sensor 200 and the second sensor 300 can be but are not limited to laser radars, cameras, millimeter wave radars, etc. The computer device 100 can be implemented with an independent server or a server cluster consisting of multiple servers.

[0076] In one embodiment, Figure 2 As shown, a method for collecting information is provided, which is applied to Figure 1 The computer equipment in the example is used as an example to illustrate, including:

[0077] S101. Acquire first sensor data of a road object collected by a first sensor, and obtain first feature information of the road object based on the first sensor data.

[0078] The first sensor may be a laser radar, a camera, or a millimeter wave radar, and the type of the first sensor is not limited here. The first sensor may be a sensor provided in a roadside system or a sensor provided in a vehicle-mounted system, and the application scenario of the first sensor is not limited here.

[0079] The first sensor data is data collected by the first sensor in the current scene, including information about road objects. Different types of first sensors can obtain different types of first sensor data, and the first sensor data can be point cloud data, image data, or video data, which is not limited here.

[0080] The first sensor has the ability to obtain the first characteristic information of the road object, and different types of first sensors can obtain different types of first characteristic information. The first characteristic information can be the color characteristic of the road object, the distance characteristic of the road object, the speed characteristic, the posture characteristic, the surface texture characteristic, etc. of the road object, and the type of the first characteristic information is not limited here. For example, when the first sensor is an image sensor, it has the ability to obtain the color characteristic of the road object, and when the first sensor is a laser radar, it has the ability to obtain the distance characteristic of the road object.

[0081] The first feature information may also include multiple types of features, including color features and surface texture features, or speed features and distance features, which are not limited here. For example, when the first sensor is a laser radar, it may have the ability to obtain distance features and speed features of road objects at the same time.

[0082] The above-mentioned road objects may be traffic participants on the road, such as vehicles or pedestrians; in addition, the above-mentioned road objects may also be obstacles on the road, road facilities, or signs such as zebra crossings, lane lines, etc. on the road; the types of the above-mentioned road objects are not limited here.

[0083] Specifically, the computer device obtains the first sensor data of the road object collected by the first sensor, and may receive the first sensor data sent by the first sensor through a wired connection, or may receive the first sensor data through a wireless connection, which is not limited here.

[0084] The computer device may acquire the first sensor data sent by the first sensor in real time, or may send an acquisition instruction to the first sensor and then acquire the first sensor data returned by the first sensor based on the acquisition instruction. The above data acquisition method is not limited herein.

[0085] Furthermore, after the computer device acquires the first sensor data, it can perform feature extraction on the first sensor data to acquire first feature information of the road object. Specifically, when the first sensor data is point cloud data, the computer device can input the point cloud data into the target recognition model to acquire the contour of the road object, and then acquire the distance feature of the road object; when the first sensor data is image data, the computer device can acquire the RGB value of the road object according to the image data.

[0086] S102. Obtain second feature information of the road object under the perspective of a second sensor according to the first feature information of the road object and a fitting mapping model corresponding to the current road scene; the fitting mapping model includes a correspondence between the first feature information and the second feature information under the perspective of the second sensor, and the first feature information and the second feature information are of different types.

[0087] Among them, the above-mentioned second sensor can be a laser radar, a camera, or a millimeter-wave radar, and the type of the above-mentioned second sensor is not limited here. The second sensor viewing angle refers to the range of information that can be perceived after the second sensor is installed and fixed. The second feature information under the second sensor viewing angle is feature information obtained by feature extraction of the perception information obtained after the second sensor is installed and fixed. Correspondingly, the second historical data collected by the second sensor can be image data, point cloud data, etc. The above-mentioned second feature information can also be color features, distance features, speed features, posture features, surface texture features, etc. of road objects, and the type of the above-mentioned second feature information is not limited here.

[0088] In the present application, the first sensor and the second sensor are of different types, and the first feature information and the second feature information are also of different types. The first sensor data itself does not have the ability to obtain the second feature information of the road object, and the second sensor itself does not have the ability to obtain the second feature information of the road object. For example, the first sensor is a laser radar, and the second sensor is an image sensor. Generally, the point cloud data obtained by the laser radar cannot directly obtain the color features of the road object, and the image data obtained by the image sensor cannot directly obtain the distance features of the road object.

[0089] For conventional roadside systems or vehicle-mounted systems, when the first characteristic information and the second characteristic information of a road object can be collected by the first sensor and the second sensor respectively, if the second sensor fails, the system cannot obtain the second characteristic information of the road object. In the present application, a computer device collects first historical data and second historical data in the current road scene, and the first historical data and the second historical data can be data collected simultaneously for the same scene; then, a fitting mapping model is established based on the first historical data collected by the first sensor and the second historical data collected by the second sensor. The fitting mapping model has the ability to associate the first characteristic information with the second characteristic information in the current road scene, so that the second characteristic information of the road object can be obtained according to the first sensor data.

[0090] The above-mentioned fitting mapping model can be a nonlinear relationship model, a linear relationship model, or a feature association table, and the type of the fitting mapping model is not limited here. The above-mentioned fitting mapping model can associate multiple features in the first feature information with the second feature information, or can associate a single feature in the first feature information with the second feature information, and is not limited here. For example, the above-mentioned fitting mapping model can also be a square relationship, a cubic relationship, etc. between the first feature information value and the second feature information value.

[0091] Specifically, after the computer device obtains the first feature information, it can directly input the first feature information into the fitting mapping model, or determine whether to input the first feature information into the fitting mapping model according to the working state of the second sensor. Optionally, the computer device can first determine whether the second sensor is invalid, and if so, perform the step of obtaining the second feature information of the road object according to the first feature information of the road object and the fitting mapping model corresponding to the current road scene.

[0092] The above failure may be caused by an abnormal connection state between the second sensor and the computer device, or abnormal data returned by the second sensor, such as missing data or distorted data; the above failure mode is not limited here.

[0093] When determining whether the second sensor has failed, the computer device can determine it based on the status monitoring log of the second sensor, or it can detect whether data returned by the second sensor has been received. If no data returned by the second sensor has been received, it can be determined that the second sensor has failed. The above-mentioned failure determination method is not limited here.

[0094] In the above information collection method, the computer device obtains the first sensor data of the road object collected by the first sensor, and obtains the first feature information of the road object based on the first sensor data; then, according to the first feature information of the road object and the fitting mapping model corresponding to the current road scene, the second feature information of the road object under the perspective of the second sensor is obtained; wherein the fitting mapping model contains the correspondence between the first feature information and the second feature information under the perspective of the second sensor, and the types of the first feature information and the second feature information are different. Since the fitting mapping model contains the correspondence between the first feature information and the second feature information under the perspective of the second sensor, after the computer device obtains the first feature information of the road object, the second feature information of the road object can be obtained according to the first feature information and the fitting mapping model, thereby avoiding the system operation abnormality caused by the second sensor being unable to obtain the second feature information, thereby improving the reliability of the system; or when there is a limit on the number or weight of sensors in the system, the first feature information and the second feature information of the road object can be obtained by deploying a first sensor, thereby improving the flexibility of system sensor deployment. In addition, it should be noted that the second feature information obtained in this embodiment can be directly used for multi-sensor data fusion, target detection, target tracking, etc., without the need to perform the corresponding feature extraction process. Of course, if the format of the second feature information does not conform to the subsequent multi-sensor data fusion, target detection, and target tracking data operations, corresponding format conversion is required.

[0095] Figure 3FIG. 1 is a flow chart of an information collection method in another embodiment. This embodiment relates to a method for establishing a fitting mapping model. Based on the above embodiment, Figure 3 As shown, the above method also includes:

[0096] S201 : Acquire first historical data collected by a first sensor and second historical data collected by a second sensor in the same time period and the same scene.

[0097] Specifically, the computer device may obtain, in a database, first historical data and second historical data collected by the first sensor and the second sensor in the same time period and in the same scene according to the collection time period and the collection scene.

[0098] S202 : Performing spatiotemporal synchronization processing on first historical data collected by the first sensor and second historical data collected by the second sensor to obtain a spatiotemporal mapping relationship between the first historical data and the second historical data.

[0099] After obtaining the first historical data and the second historical data, the computer device may perform spatiotemporal synchronization processing on the first historical data and the second historical data.

[0100] Specifically, the computer device may first perform time synchronization on the first historical data and the second historical data, and then perform space synchronization; or may first perform space synchronization on the first historical data and the second historical data, and then perform time synchronization; the above-mentioned time-space synchronization processing method is not limited here. After the time-space synchronization processing is performed on the first historical data and the second historical data, the obtained time-space mapping relationship includes the corresponding relationship between the first historical data and the second historical data collected for the same target at the same time.

[0101] S203: Extract features from the first historical data to obtain first feature information, and extract features from the second historical data to obtain second feature information.

[0102] Specifically, the computer device may input the first historical data and the second historical data into corresponding feature extraction models respectively to obtain first feature information corresponding to the first historical data and second feature information corresponding to the second historical data.

[0103] S204: Based on the spatiotemporal mapping relationship between the first historical data and the second historical data, fit the first feature information and the second feature information to establish a fitting mapping model.

[0104] The above-mentioned fitting mapping model can be a nonlinear model, and the computer device can substitute multiple sets of corresponding first feature information and second feature information into the nonlinear initial model to obtain the parameters of the nonlinear initial model and obtain the nonlinear model; or, the above-mentioned fitting mapping model can be a machine learning model, and the computer device can determine the model parameters of the machine learning model according to the multiple sets of corresponding first feature information and second feature information. The establishment process of the above-mentioned mapping model is not limited here.

[0105] The above-mentioned fitting mapping model may be a mapping curve or a mapping table, and the form of the above-mentioned fitting mapping model is not limited herein. For example, the above-mentioned fitting mapping model may be a mapping curve, and the above-mentioned mapping curve may be obtained by fitting with the value of the first characteristic information as the horizontal coordinate and the value of the corresponding second characteristic information as the vertical coordinate. For another example, the above-mentioned fitting mapping model may be a mapping table, and the above-mentioned mapping table may include the second characteristic information corresponding to different values ​​of the first characteristic information.

[0106] After the computer device obtains the first feature information of the road object in the current scene, it can search for the second feature information corresponding to the first feature information according to the above-mentioned fitting mapping model, and determine it as the second feature information of the road object.

[0107] In the above information collection method, the computer device performs spatiotemporal synchronization processing on the first historical data and the second historical data, and establishes a fitting mapping model based on the established spatiotemporal mapping relationship between the first historical data and the second historical data, so that the fitting mapping model has the ability to associate the first feature information with the second feature information, so that the second feature information of the road object can be obtained according to the first sensor data, thereby improving the stability and reliability of the system.

[0108] Figure 4 : is a flow chart of an information collection method in another embodiment. This embodiment relates to a method for establishing a fitting mapping model. On the basis of the above embodiment, the first historical data includes multiple first data frames, each of which includes multiple first data, and the second historical data includes multiple second data frames, each of which includes multiple second data. Figure 4 As shown, the above S202 includes:

[0109] S301 , time-synchronize first historical data and second historical data to obtain a plurality of time-synchronized data frame pairs; each data frame pair includes a first data frame and a second data frame synchronized at sampling time.

[0110] When the computer device establishes the fitting mapping model based on the first historical data and the second historical data, the first historical data and the second historical data may be time synchronized so as to correspond the first historical data and the second historical data collected at the same time.

[0111] The first historical data may include multiple first data frames, and the second historical data may include multiple second data frames. The computer device may determine whether the first data frame and the second data frame are time synchronized according to the acquisition time of the first data frame and the second data frame, and then determine the first data frame and the second data frame synchronized in sampling time as a data frame pair.

[0112] Specifically, when the computer device performs time synchronization according to the sampling time, it can obtain the sampling time of each first data frame and each second data frame, and then compare the obtained sampling times to determine the synchronized data frames; in addition, the computer device can also first determine the second data frame synchronized with the first first data frame, and then calculate the positions of other second data frames synchronized with the first data frames in turn according to the sampling frequency of the first historical data and the second historical data; the above-mentioned time synchronization method is not limited here.

[0113] Optionally, the sampling frequencies of the first historical data and the second historical data may be the same. After the computer device determines the data pair synchronized at the first sampling time, the data frame after the first data frame in the data pair in the first historical data and the data frame after the second data frame in the data pair in the second historical data may be determined as the second data frame pair, and the time synchronization of each data frame may be completed in sequence, thereby improving the time synchronization efficiency of the first historical data and the second historical data.

[0114] S302 , performing coordinate system conversion on the first data frame and the second data frame in each data frame pair to obtain spatially synchronized data pairs, each data pair including first data in the spatially synchronized first data frame and second data in the second data frame.

[0115] Each first data frame may include multiple first data, and the first data may be data of a certain sampling point in the first data frame, or data of sampling points in a certain area in the first data frame; in addition, the first data may also be first historical data corresponding to a certain road object in the first data frame, which is not limited here. Each second data frame may include multiple second data, and the second data may be data of a certain sampling point in the second data frame, or data of sampling points in a certain area in the second data frame; in addition, the second data may also be second historical data corresponding to a certain road object in the second data frame, which is not limited here.

[0116] After the computer device obtains multiple data frame pairs synchronized with the sampling time, it can correspond the first data and the second data in each data frame pair to obtain the first data and the second data obtained by different sensors when collecting the same target.

[0117] Specifically, the computer device can perform spatial calibration by means of coordinate conversion, so that the data collected by the first sensor and the second sensor can be marked in the same coordinate system, thereby obtaining the corresponding relationship between the first data and the second data. The computer device can convert the first data to the coordinates where the second data is located, and can also convert the second data to the coordinates where the first data is located. In addition, the computer device can also convert both the first data and the second data to other coordinate systems, such as the earth coordinate system, etc.; the above-mentioned spatial calibration method is not limited here.

[0118] Accordingly, the computer device may establish a fitting mapping model based on the correspondence between the first feature information corresponding to the first data and the second feature information corresponding to the second data in each data pair.

[0119] Specifically, the computer device may perform feature extraction on the first data to obtain first feature information corresponding to the first data; then, perform feature extraction on the second data to obtain second feature information corresponding to the second data; and based on the correspondence between the first data and the second data, establish a fitting mapping model between the first feature information associated with the first data and the second feature information associated with the second data. The computer device may perform corresponding association between the first feature information corresponding to each of the first data and the second feature information corresponding to the second data, thereby obtaining values ​​of the second feature information corresponding to multiple values ​​of the first feature information, i.e., a fitting mapping model between the first feature information and the second feature information.

[0120] In the above information collection method, the computer device first synchronizes the time of the first historical data and the second historical data, and then performs spatial correspondence, so that the spatiotemporal mapping relationship between the first historical data and the second historical data can be quickly established, thereby improving the acquisition efficiency of the fitting mapping model.

[0121] Figure 5 is a flow chart of an information collection method in another embodiment. This embodiment relates to a method for a computer device to obtain a fitting mapping model. Based on the above embodiment, the first sensor is an image sensor, the second sensor is a laser radar, the first sampling point data is a two-dimensional image of the current road scene, and the second sampling point data is point cloud data of the road object; the first feature information is the pixel coordinates of the road object in the image coordinate system of the two-dimensional image, and the second feature information is the depth information of the road object, and the depth information is used to characterize the distance between the road object and the laser radar; Figure 5 As shown, the above S203 includes:

[0122] S401 . Map depth information to corresponding pixel coordinates in a two-dimensional image according to coordinates of second data to obtain a depth image, so that some pixel coordinates on the depth image have depth information.

[0123] Among them, the first data is a two-dimensional image of the current road scene, and the second data is the point cloud data of the road object; the first feature information is the pixel coordinates of the road object in the image coordinate system of the two-dimensional image, and the second feature information is the depth information of the road object. The above pixel coordinates represent the position of each pixel point in the two-dimensional image in the two-dimensional coordinate system where the two-dimensional image is located, that is, the pixel coordinate system. The above depth information is used to characterize the distance between the road object and the laser radar, and can be represented by the relative position coordinates of the road object and the laser radar. The above relative position coordinates can be Cartesian coordinate values ​​or polar coordinate values; in addition, the above depth information can also be represented by the distance value between the road object and the laser radar, which is not limited here.

[0124] For example, the second sensor coordinate system is the world coordinate system used by the laser radar, and the conversion relationship between the above point cloud data and the two-dimensional image data is:

[0125]

[0126] Among them, in this conversion relationship, is the internal parameter matrix of the first sensor, is the external parameter matrix of the first sensor; where r 11 ,r 12 ,…r 33 represents the rotation angle between the first sensor and the second sensor, and the above t1-t3 represents the translation amount between the first sensor and the second sensor. is the coordinate of the second data in the world coordinate system, is the coordinate of the second data in the pixel coordinate system.

[0127] On the basis of the correspondence between the first data and the second data obtained by the computer device, it can be determined which point cloud data each pixel coordinate in the two-dimensional image corresponds to, thereby determining the depth information corresponding to each pixel coordinate. The computer device can set the corresponding pixel coordinates for each depth information, such as marking them on the two-dimensional image, to obtain a depth image. The above-mentioned depth image can include pixel values ​​corresponding to each pixel coordinate, and each pixel value can also be binarized, and each pixel value can also be set to a preset value, such as setting them all to white, which is not limited here. Figure 6 The depth image corresponding to a scene is shown as mapping the depth information onto a two-dimensional image. Figure 6 The points in represent pixel coordinates with depth information. For a road object in the figure, such as point M on a vehicle, the depth information mapped at point M in the above depth image can represent the distance between the vehicle and the lidar.

[0128] S402 , performing curve fitting connection on some pixel coordinates in the depth image to obtain a plurality of depth fitting curves, and determining the plurality of depth fitting curves as a fitting mapping model.

[0129] The density of the point cloud data obtained by the laser radar can be determined according to the number of laser radar lines. For example, the point cloud data obtained by a 64-line laser radar is denser than the point cloud data obtained by a 16-line laser radar. However, since the density of the point cloud data may not match the number of pixels in the two-dimensional image, and there may be data loss when the laser radar is scanning, the depth image obtained by the computer device may only have depth information on some pixel coordinates.

[0130] Based on the above depth image, the computer device can perform curve fitting connection on the partial pixel coordinates with depth information to obtain multiple depth fitting curves, such as Figure 7 As shown, some pixel coordinates in the depth image that originally did not have depth information obtain corresponding depth information, thereby enhancing the richness of the depth information on the depth image, so that the computer device can more accurately obtain the second feature information of the road object according to the fitting mapping model.

[0131] In addition, after obtaining the point cloud data, the computer device can also perform densification processing on the point cloud data, so that more depth fitting curves can be obtained. For example, a 16-line laser radar can obtain 16 depth fitting curves. After performing densification processing on the point cloud data, the computer device can obtain 32 or more depth fitting curves, further improving the accuracy of the fitting mapping model.

[0132] When the computer device performs curve fitting connection on the pixel coordinates with depth information, the depth information can be filtered according to the smoothness of the curve, and the pixel coordinates that are not successfully connected can be filtered out, so that the obtained depth fitting curve is smoother. The above-mentioned depth fitting curve can be a fitting circle or other types of curves, which are not limited here.

[0133] In the above information collection method, the computer device obtains a depth image and performs curve fitting connection on some pixel coordinates in the depth image, and determines the obtained multiple depth fitting curves as a fitting mapping model. After the computer device obtains a two-dimensional image, it can determine the depth information corresponding to the pixel coordinates in the above depth fitting curve according to the pixel coordinates of the road object in the two-dimensional image, so that the depth information of the road object can be directly determined, thereby avoiding system operation abnormalities caused by the inability of the laser radar to obtain depth information, thereby improving the reliability of the system.

[0134] Figure 8: is a flowchart of an information collection method in another embodiment. This embodiment relates to a method in which a computer device obtains second feature information of a road object according to first feature information and a fitting mapping model. On the basis of the above embodiment, the above S102 includes:

[0135] S501: In a depth image, determine whether pixel coordinates of a road object fall on a plurality of depth fitting curves.

[0136] When establishing the fitting mapping model, the computer device may not necessarily cover all pixel coordinates, so after obtaining the pixel coordinates of the road object on the two-dimensional image, it can first determine whether the pixel coordinates have completely corresponding depth information in the fitting mapping model. Specifically, the computer device can determine whether the pixel coordinates of the road object fall on the depth fitting curve in the depth image.

[0137] S502: If yes, determine the depth information corresponding to the pixel coordinates of the road object as the depth information of the road object.

[0138] The pixel coordinates of the road object may be pixel coordinates corresponding to the center point of the road object, or may be pixel coordinates corresponding to any point in the contour of the road object, which is not limited here.

[0139] If the pixel coordinates of the road object fall on the depth fitting curve, that is, the pixel coordinates of the road object have completely corresponding depth information, then the computer device can determine the depth information corresponding to the pixel coordinates as the depth information of the road object.

[0140] Of course, the pixel coordinates of the above road objects may not fall on multiple depth fitting curves, such as Fig. 9 The pixel coordinate A shown in the figure, then the computer can obtain multiple target pixel coordinates around the pixel coordinate of the road object; then, a preset interpolation algorithm is used to interpolate the depth information corresponding to each target pixel coordinate to obtain the depth information of the road object; wherein the above target pixel coordinates are pixel coordinates with depth information that fall on multiple depth fitting curves.

[0141] The number of the target pixel coordinates may be two or more. This embodiment is mainly described by taking two target pixel coordinates as an example.

[0142] When acquiring multiple target pixel coordinates, the computer device calculates the distance between the pixel coordinates of the road object and the pixel coordinates on the depth fitting curve, and then determines a preset number of target pixel coordinates that are closer to the pixel coordinates of the road object based on the above distance; or the computer device can use the pixel coordinates of the road object as the center of the circle and determine a circular area of ​​the target pixel coordinates based on a preset radius, and all pixel coordinates with depth information in the circular area can be determined as the target pixel coordinates; the method for determining the above target pixel coordinates is not limited here.

[0143] Optionally, the computer device can determine two target depth fitting curves that are closest to the pixel coordinates of the road object in the depth image; determine a calibration line passing through the pixel coordinates of the road object; intersect the calibration line with the two target depth fitting curves; and determine the pixel coordinates of the intersection of the calibration line and the two target depth fitting curves as the target pixel coordinates.

[0144] The calibration line is a line intersecting the two target depth fitting curves. The calibration line may be a line parallel to the coordinate axis or a normal to the depth fitting curve. The method for determining the calibration line is not limited herein.

[0145] Continue with Fig. 9 For example, the pixel coordinates of the road object in the figure are pixel coordinates A(x0,y0), and the two target depth fitting curves closest to pixel coordinate A are curve 1 and curve 2. The calibration line in the figure is a line perpendicular to the x-axis in the pixel coordinates of the depth image. The intersection points of the above calibration line with curve 1 and curve 2 are pixel coordinates B(x0,y1) and pixel coordinates C(x0,y2), respectively. The computer device can determine the above pixel coordinates B(x0,y1) and pixel coordinates C(x0,y2) as the target pixel coordinates.

[0146] Furthermore, when the computer device performs interpolation processing on the depth information corresponding to each target pixel coordinate, the distance value between the pixel coordinate of the road object and each target pixel coordinate can be calculated; then, according to each distance value, the depth information corresponding to each target pixel coordinate can be interpolated to obtain the depth information of the road object. Fig. 9 For example, in the above target pixel coordinates, the depth information corresponding to the pixel coordinate B (x0, y1) is (X1, Y1, Z1), and the depth information corresponding to the pixel coordinate C (x0, y2) is (X2, Y2, Z2). The computer device can interpolate X1 and X2, Y1 and Y2, and Z1 and Z2 according to the ratio of y1-y0 and y0-y2 to obtain the depth information corresponding to the pixel coordinate A (x0, y0).

[0147] In the above information collection method, after obtaining the pixel coordinate system of the road object, the computer device determines whether the pixel coordinates of the road object fall on multiple depth fitting curves. When the pixel coordinates fall on the depth fitting curve, the depth information corresponding to the pixel coordinates can be directly determined as the depth information of the road object, so that the computer device can quickly determine the depth information of the road object according to the pixel coordinates; further, when the pixel coordinates do not fall on the depth fitting curve, the computer device interpolates the depth information corresponding to each target pixel coordinate to make the obtained depth information of the road object more accurate.

[0148] Fig.10 FIG. 1 is a flow chart of an information collection method in another embodiment. This embodiment relates to a method in which a computer device performs spatial calibration on each data pair. Based on the above embodiment, Figure 8 As shown, the above S302 includes:

[0149] S601 . According to a preset conversion matrix, convert the second data in the data frame pair into a first sensor coordinate system to obtain mapping coordinates of each second data in the first sensor coordinate system.

[0150] The above-mentioned conversion matrix is ​​a matrix that converts the data collected by the second sensor into the coordinates of the first sensor, which can be determined based on the relative posture between the first sensor and the second sensor. The above-mentioned relative posture can include the translation and rotation angle between the first sensor and the second sensor.

[0151] The above conversion matrix can be obtained by manual measurement by a staff member and then input into the computer device; it can also be obtained by automatic calibration of the computer device according to the current relative posture relationship between the first sensor and the second sensor, which is not limited here. The computer device converts the second data in the data pair into the first sensor coordinate system according to the preset conversion matrix, and the mapping coordinates of each second data in the first sensor coordinate system can be obtained.

[0152] The above-mentioned second data can be two-dimensional data or three-dimensional data in the second sensor coordinate system, which is not limited here. When the above-mentioned second data is two-dimensional data, the computer device can first map the second data to the second sensor coordinate system to obtain the second data corresponding to each three-dimensional space point. For example, if the second data is a two-dimensional image, the computer device can map the pixel value of the target object in the two-dimensional image to the corresponding position of the target object in the three-dimensional space according to a preset pixel conversion algorithm. Furthermore, the computer device can convert the second data corresponding to each three-dimensional space point in the second sensor coordinate system to the first sensor coordinate system through a conversion matrix to obtain the mapping coordinates of each second data in the first sensor coordinate system.

[0153] S602: In the data frame pair, obtain first data corresponding to the mapping coordinates.

[0154] The first data may be two-dimensional data or three-dimensional data in the first sensor coordinate system, which is not limited here. When the first data is two-dimensional data, the computer device may first map the first data to the first sensor coordinate system to obtain the first data corresponding to each three-dimensional space point; or the computer device may correspond the coordinates to the coordinates of the first data in the two-dimensional coordinate system to determine the two-dimensional coordinates corresponding to the coordinates and the first data corresponding to the two-dimensional coordinates.

[0155] Furthermore, the computer device may determine the first data corresponding to each mapping coordinate according to each mapping coordinate. Specifically, the computer device may search for the first data corresponding to the mapping coordinate in the data pair; if the data pair does not contain the data corresponding to the mapping coordinate, the computer device may determine the first data corresponding to the mapping coordinate according to the distance between the mapping coordinate and the coordinate value corresponding to the first data. For example, when the distance between the mapping coordinate and the coordinate value corresponding to the first data is less than a preset distance threshold, the computer device may determine the first data as the first data corresponding to the mapping coordinate.

[0156] S603: Determine the second data corresponding to each mapping coordinate and the first data corresponding to the mapping coordinate as a data pair.

[0157] Based on the above steps, the computer device determines the second data corresponding to each mapping coordinate and the first data corresponding to each mapping coordinate, and can associate the second data with the first data through the mapping coordinates, thereby establishing a corresponding relationship between the first data and the second data.

[0158] In the above information collection method, the computer device completes the spatial calibration of the data pair through the transformation matrix, so that the computer device can obtain the second data corresponding to each mapping coordinate and the first data corresponding to each mapping coordinate, so that the corresponding relationship between the above first data and the second data can be established according to the mapping coordinates, which is conducive to quickly and accurately establishing a fitting mapping model.

[0159] Fig.11 FIG. 1 is a flow chart of an information collection method in another embodiment. This embodiment relates to a method for a computer device to synchronize time between first historical data and second historical data. Based on the above embodiment, Fig.11 As shown, the above S301 includes:

[0160] S701: Convert the first historical data and the second historical data to the same time axis.

[0161] The first sensor and the second sensor are two independent devices, and the time axes of the first historical data and the second historical data obtained may be different. For example, the time axis of the first sensor is the Global Positioning System (GPS) time axis, while the time axis of the second sensor is determined by the device itself, and there is a certain time axis difference. The computer device can convert the first historical data and the second historical data to the same time axis, so that the system obtains the first data frame and the second data frame synchronized with the sampling time.

[0162] Specifically, the computer device can convert the first historical data to the time axis of the second historical data, or convert the second historical data to the time axis of the first historical data, or convert both the first historical data and the second historical data to another time axis, for example, to the GPS time axis. The above conversion method is not limited here.

[0163] S702 . On the time axis, obtain a first sampling time of each first data frame in the first historical data and a second sampling time of each second data frame in the second historical data.

[0164] Furthermore, the computer device may obtain the first sampling time of each first data frame and the second sampling time of the second data frame. The first sampling time may be a timestamp marked on the first data frame when the first sensor collects the first historical data, or may be a first sampling time obtained according to the order of each first data frame and the start sampling time. The method for obtaining the first sampling time is not limited here. The method for obtaining the second sampling time is similar to the method for obtaining the first sampling time, and will not be described in detail here.

[0165] S703: Calculate the difference between the first sampling time and the second sampling time.

[0166] S704: If the difference is less than a preset threshold, determine that the first data frame corresponding to the first sampling moment and the second data frame corresponding to the second sampling moment are a data frame pair.

[0167] The computer device may determine that the first data frame and the second data frame are a data pair when the difference between the first sampling moment and the second sampling moment is within a certain range. The computer device may calculate the absolute value of the difference between the first sampling moment and the second sampling moment; if the absolute value of the difference is less than a preset threshold, the first data frame corresponding to the first sampling moment and the second data frame corresponding to the second sampling moment are determined to be a data frame pair.

[0168] In the above-mentioned information collection method, the computer device converts the first historical data and the second historical data to the same time axis, so that the computer device can accurately synchronize the first data frame and the second data frame; further, the computer device obtains the absolute value of the difference between the first sampling moment and the second sampling moment, and when the absolute value of the above-mentioned difference is less than a preset threshold value, it is determined that the first data frame and the second data frame are a data frame pair, thereby avoiding time synchronization failure caused by the first sampling moment and the second sampling moment not being completely the same due to differences in sampling frequency, etc., thereby improving the stability of the information collection process.

[0169] Fig.12 FIG. 1 is a flow chart of an information collection method in an embodiment. Fig.12 , information collection methods include:

[0170] S801: Convert the first historical data and the second historical data to the same time axis.

[0171] S802: On the time axis, obtain a first sampling time of each first data frame in the first historical data and a second sampling time of each second data frame in the second historical data.

[0172] S803, calculating the difference between the first sampling time and the second sampling time, and if the difference is less than a preset threshold, determining that the first data frame corresponding to the first sampling time and the second data frame corresponding to the second sampling time are a data frame pair.

[0173] S804 . According to a preset conversion matrix, convert the second data in the data pair into the first sensor coordinate system to obtain mapping coordinates of each second data in the first sensor coordinate system.

[0174] S805. Obtain first data corresponding to the mapping coordinates in the data frame pair.

[0175] S806: Determine the second data corresponding to each mapping coordinate and the first data corresponding to the mapping coordinate as a data pair.

[0176] S807: Extract features from the first data to obtain first feature information corresponding to the first data.

[0177] S808: Extract features from the second data to obtain second feature information corresponding to the second data

[0178] S809: Establish a fitting mapping model based on the correspondence between the first feature information corresponding to the first data and the second feature information corresponding to the second data in each data pair.

[0179] S810: Acquire first sensor data of a road object collected by a first sensor.

[0180] S811. Obtain first feature information of a road object based on first sensor data.

[0181] S812: Determine whether the second sensor is invalid, and if so, execute S813.

[0182] S813: Obtain second feature information of the road object in a second sensor viewing angle according to the first feature information of the road object and a fitting mapping model corresponding to the current road scene.

[0183] The implementation principle and technical effect of the above information collection method are similar to those of the above embodiment and will not be repeated here.

[0184] In one embodiment, a method for collecting information is provided. Fig.13 As shown, the above method includes:

[0185] S901. Determine whether a sensor switching condition is met according to a current sensor state.

[0186] The sensor switching condition may be sensor failure or sensor stoppage caused by computer equipment, which is not limited here. The sensor failure may be sensor equipment damage or abnormal data collected by the sensor, etc. The computer equipment may determine whether the sensor fails based on the information returned by the sensor, or determine whether the laser radar fails based on the data collected by the sensor, which is not limited here.

[0187] S902: If the sensor switching condition is met, the current sensor is used as the second sensor, and the steps of the information collection method corresponding to the above embodiment are executed to collect information.

[0188] The implementation principle and technical effect of the above information collection method are similar to those of the above embodiment and will not be repeated here.

[0189] It should be understood that although Figure 2-13 The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. Moreover, Figure 2-13 At least part of the steps may include multiple steps or multiple stages. These steps or stages are not necessarily performed at the same time, but can be performed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed in turn or alternately with other steps or at least part of the steps or stages in other steps.

[0190] In one embodiment, Fig.14As shown, an information collection device is provided, including: an acquisition module 10 and a fitting module 20, wherein:

[0191] An acquisition module 10 is used to acquire first sensor data of a road object collected by a first sensor, and obtain first feature information of the road object based on the first sensor data;

[0192] The fitting module 20 is used to obtain the second feature information of the road object under the perspective of the second sensor according to the first feature information of the road object and the fitting mapping model corresponding to the current road scene; the fitting mapping model includes the correspondence between the first feature information and the second feature information under the perspective of the second sensor, and the types of the first feature information and the second feature information are different.

[0193] In one embodiment, based on the above embodiment, the above device further includes an establishment module 30, such as Fig.15 As shown, the establishment module 30 includes:

[0194] An acquisition unit 301 is used to acquire first historical data collected by a first sensor and second historical data collected by a second sensor in the same time period and the same scene;

[0195] A synchronization unit 302, configured to perform spatiotemporal synchronization processing on first historical data collected by the first sensor and second historical data collected by the second sensor to obtain a spatiotemporal mapping relationship between the first historical data and the second historical data;

[0196] An extraction unit 303 is used to perform feature extraction on the first historical data to obtain first feature information, and perform feature extraction on the second historical data to obtain second feature information;

[0197] The establishing unit 304 is used to fit the first feature information and the second feature information based on the spatiotemporal mapping relationship between the first historical data and the second historical data to establish a fitting mapping model.

[0198] In one embodiment, based on the above embodiment, the first historical data includes multiple first data frames, each of which includes multiple first data, and the second historical data includes multiple second data frames, each of which includes multiple second data; Fig.16 As shown, the synchronization unit 302 includes:

[0199] The synchronization subunit 3021 is used to perform time synchronization on the first historical data and the second historical data to obtain a plurality of time-synchronized data frame pairs; each data frame pair includes a first data frame and a second data frame synchronized at a sampling time;

[0200] The conversion subunit 3022 is used to perform coordinate system conversion on the first data frame and the second data frame in each data frame pair to obtain spatially synchronized data pairs, each data pair including first data in the spatially synchronized first data frame and second data in the second data frame.

[0201] In one embodiment, based on the above embodiment, the first sensor is an image sensor, the second sensor is a laser radar, the relative positions of the first sensor and the second sensor are fixed, the first data is a two-dimensional image of the current road scene, and the second data is point cloud data of the road object; the first feature information is the pixel coordinates of the road object in the image coordinate system of the two-dimensional image, and the second feature information is the depth information of the road object, and the depth information is used to characterize the distance between the road object and the laser radar; the above-mentioned establishment unit 304 is specifically used to: map the depth information to the corresponding pixel coordinates in the two-dimensional image according to the coordinates of the second data to obtain a depth image so that some pixel coordinates on the depth image have depth information; perform curve fitting connection on some pixel coordinates in the depth image to obtain multiple depth fitting curves, and determine the multiple depth fitting curves as a fitting mapping model.

[0202] In one embodiment, based on the above embodiment, Fig.17 As shown, the fitting module 20 includes:

[0203] A first determining unit 201 is used to determine whether the pixel coordinates of the road object fall on a plurality of depth fitting curves in the depth image;

[0204] The second determining unit 202 is configured to determine the depth information corresponding to the pixel coordinates of the road object as the depth information of the road object when the pixel coordinates of the road object fall on a plurality of depth fitting curves.

[0205] In one embodiment, based on the above embodiment, the second determination unit 202 is further used to: if the pixel coordinates of the road object do not fall on multiple depth fitting curves, obtain multiple target pixel coordinates around the pixel coordinates of the road object; the target pixel coordinates are pixel coordinates with depth information; and use a preset interpolation algorithm to interpolate the depth information corresponding to each target pixel coordinate to obtain the depth information of the road object.

[0206] In one embodiment, based on the above embodiment, the above second determination unit 202 is specifically used to: calculate the distance value between the pixel coordinates of the road object and each target pixel coordinate; and interpolate the depth information corresponding to each target pixel coordinate according to the distance value to obtain the depth information of the road object.

[0207] In one embodiment, based on the above embodiment, the above second determination unit 202 is specifically used to: determine the two target depth fitting curves that are closest to the pixel coordinates of the road object in the depth image; determine the calibration line passing through the pixel coordinates of the road object; the calibration line intersects the two target depth fitting curves; and determine the pixel coordinates of the intersection of the calibration line and the two target depth fitting curves as the target pixel coordinates.

[0208] In one embodiment, based on the above embodiment, the above conversion subunit 3022 is specifically used to: convert the second data in the data frame pair to the first sensor coordinate system according to a preset conversion matrix, and obtain the mapping coordinates of each second data in the first sensor coordinate system; in the data frame pair, obtain the first data corresponding to the mapping coordinates; and determine the second data corresponding to each mapping coordinate and the first data corresponding to the mapping coordinate as a data pair.

[0209] In one embodiment, based on the above embodiment, the above-mentioned synchronization subunit 3021 is specifically used to: convert the first historical data and the second historical data to the same time axis; on the time axis, obtain the first sampling moment of each first data frame in the first historical data, and the second sampling moment of each second data frame in the second historical data; calculate the difference between the first sampling moment and the second sampling moment; if the difference is less than a preset threshold, determine that the first data frame corresponding to the first sampling moment and the second data frame corresponding to the second sampling moment are a data frame pair.

[0210] In one embodiment, based on the above embodiment, the sampling frequencies of the first historical data and the second historical data are in a multiple relationship.

[0211] The information collection devices provided in the above embodiments can execute the above information collection method embodiments, and their implementation principles and technical effects are similar and will not be repeated here.

[0212] In one embodiment, an information collection device is provided, such as Fig.18 As shown, the above device comprises:

[0213] A determination module 110, configured to determine whether a sensor switching condition is met according to a current sensor state;

[0214] The acquisition module 120 is used to use the current sensor as the second sensor and execute the steps of the above information acquisition method to collect information when the current sensor state meets the sensor switching condition.

[0215] The information collection devices provided in the above embodiments can execute the above information collection method embodiments, and their implementation principles and technical effects are similar and will not be repeated here.

[0216] For the specific definition of the information collection device, please refer to the definition of the information collection method above, which will not be repeated here. Each module in the above information collection device can be implemented in whole or in part by software, hardware and a combination thereof. The above modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0217] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Fig.19 As shown. The computer device includes a processor, a memory and a network interface connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store information collection data. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, an information collection method is implemented.

[0218] Those skilled in the art will understand that Fig.19 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0219] In one embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above-mentioned method embodiments when executing the computer program.

[0220] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0221] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory may include read-only memory (ROM), magnetic tape, floppy disk, flash memory or optical memory, etc. Volatile memory may include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM).

[0222] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0223] The above embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.

Claims

1. An information collection method, characterized in that: The method comprises: Acquire first sensor data of a road object collected by a first sensor, and obtain first feature information of the road object based on the first sensor data; According to the first feature information of the road object and the fitting mapping model corresponding to the current road scene, second feature information of the road object under the perspective of a second sensor is obtained; the fitting mapping model includes a correspondence between the first feature information and the second feature information under the perspective of the second sensor, and the first feature information and the second feature information are of different types; Before acquiring the first sensor data of the road object collected by the first sensor, the method further includes: Acquire first historical data collected by a first sensor and second historical data collected by a second sensor in the same time period and the same scene; Performing spatiotemporal synchronization processing on first historical data collected by the first sensor and second historical data collected by the second sensor to obtain a spatiotemporal mapping relationship between the first historical data and the second historical data; Extracting features from the first historical data to obtain first feature information, and extracting features from the second historical data to obtain second feature information; Based on the spatiotemporal mapping relationship between the first historical data and the second historical data, fitting the first feature information and the second feature information to establish the fitting mapping model; The first historical data includes a plurality of first data frames, the second historical data includes a plurality of second data frames, and the first historical data collected by the first sensor and the second historical data collected by the second sensor are subjected to spatiotemporal synchronization processing to obtain a spatiotemporal mapping relationship between the first historical data and the second historical data, including: Performing time synchronization on the first historical data and the second historical data to obtain a plurality of time-synchronized data frame pairs; each data frame pair includes a first data frame and a second data frame synchronized at a sampling time; Performing coordinate system conversion on the first data frame and the second data frame in each data frame pair to obtain spatially synchronized data pairs, each data pair including first data in the first spatially synchronized data frame and second data in the second spatially synchronized data frame; Correspondingly, the fitting of the first feature information and the second feature information based on the spatiotemporal mapping relationship between the first historical data and the second historical data to establish the fitting mapping model includes: The fitting mapping model is established based on the corresponding relationship between the first feature information corresponding to the first data and the second feature information corresponding to the second data in each of the data pairs.

2. The information collection method according to claim 1, characterized in that: The first sensor is an image sensor, the second sensor is a laser radar, the relative positions of the first sensor and the second sensor are fixed, the first data is a two-dimensional image of the current road scene, and the second data is point cloud data of a road object; the first feature information is the pixel coordinates of the road object in the image coordinate system of the two-dimensional image, and the second feature information is the depth information of the road object, and the depth information is used to characterize the distance between the road object and the laser radar; The establishing of the fitting mapping model based on the correspondence between the first feature information corresponding to the first data and the second feature information corresponding to the second data in each of the data pairs includes: According to the coordinates of the second data, the depth information is mapped to corresponding pixel coordinates in the two-dimensional image to obtain a depth image, so that some pixel coordinates on the depth image have depth information; Curve fitting is performed on the partial pixel coordinates in the depth image to obtain a plurality of depth fitting curves, and the plurality of depth fitting curves are determined as the fitting mapping model.

3. The information collection method according to claim 2, characterized in that: The obtaining, according to the first feature information of the road object and the fitting mapping model corresponding to the current road scene, second feature information of the road object under the perspective of the second sensor includes: In the depth image, determining whether pixel coordinates of the road object fall on the plurality of depth fitting curves; If so, the depth information corresponding to the pixel coordinates of the road object is determined as the depth information of the road object.

4. The information collection method according to claim 3, characterized in that: The method further comprises: If not, then obtaining a plurality of target pixel coordinates around the pixel coordinates of the road object; the target pixel coordinates are pixel coordinates having depth information; A preset interpolation algorithm is used to interpolate the depth information corresponding to each of the target pixel coordinates to obtain the depth information of the road object.

5. The information collection method according to claim 4, characterized in that: The method of using a preset interpolation algorithm to interpolate the depth information corresponding to each target pixel coordinate to obtain the depth information of the road object includes: Calculating the distance between the pixel coordinates of the road object and the pixel coordinates of each target; According to the distance value, the depth information corresponding to each of the target pixel coordinates is interpolated to obtain the depth information of the road object.

6. The information collection method according to claim 5, characterized in that: The step of acquiring a plurality of target pixel coordinates around the pixel coordinate of the road object comprises: In the depth image, determining two target depth fitting curves that are closest to pixel coordinates of the road object; Determine a calibration line passing through the pixel coordinates of the road object; the calibration line intersects the two target depth fitting curves; The pixel coordinates of the intersection of the calibration line and the two target depth fitting curves are determined as the target pixel coordinates.

7. The information collection method according to any one of claims 1 to 6, characterized in that: Performing coordinate system conversion on the first data frame and the second data frame in each data frame pair to obtain a spatially synchronized data pair includes: According to a preset conversion matrix, the second data in the data frame pair is converted into a first sensor coordinate system to obtain mapping coordinates of each second data in the first sensor coordinate system; In the data frame pair, obtaining first data corresponding to the mapping coordinates; The second data corresponding to each of the mapping coordinates and the first data corresponding to the mapping coordinates are determined as a data pair.

8. The information collection method according to any one of claims 1 to 6, characterized in that: The step of performing time synchronization on the first historical data and the second historical data to obtain a plurality of time-synchronized data frame pairs includes: Converting the first historical data and the second historical data to the same time axis; Under the time axis, obtaining a first sampling time of each first data frame in the first historical data, and a second sampling time of each second data frame in the second historical data; Calculating a difference between the first sampling time and the second sampling time; If the difference is smaller than a preset threshold, it is determined that the first data frame corresponding to the first sampling moment and the second data frame corresponding to the second sampling moment are a data frame pair.

9. The information collection method according to claim 8, characterized in that: The sampling frequencies of the first historical data and the second historical data are in a multiple relationship.

10. An information collection method, characterized in that: The method comprises: According to the current sensor state, determine whether the sensor switching condition is met; If the sensor switching condition is met, the current sensor is used as the second sensor, and the steps of the information collection method according to any one of claims 1 to 9 are executed to collect information.

11. An information collection device, characterized in that: The device comprises: an acquisition module, configured to acquire first sensor data of a road object collected by a first sensor, and obtain first feature information of the road object based on the first sensor data; a fitting module, configured to obtain second feature information of the road object under a second sensor perspective according to the first feature information of the road object and a fitting mapping model corresponding to a current road scene; the fitting mapping model includes a correspondence between the first feature information and the second feature information under a second sensor perspective, and the first feature information and the second feature information are of different types; The device also includes a building module, and the building module includes: An acquisition unit, used to acquire first historical data collected by the first sensor and second historical data collected by the second sensor in the same time period and the same scene; a synchronization unit, configured to perform spatiotemporal synchronization processing on first historical data collected by the first sensor and second historical data collected by the second sensor to obtain a spatiotemporal mapping relationship between the first historical data and the second historical data; an extraction unit, configured to perform feature extraction on the first historical data to obtain first feature information, and perform feature extraction on the second historical data to obtain second feature information; an establishing unit, configured to fit the first feature information and the second feature information based on a spatiotemporal mapping relationship between the first historical data and the second historical data, and establish the fitting mapping model; The first historical data includes a plurality of first data frames, the second historical data includes a plurality of second data frames, and the synchronization unit includes: A synchronization subunit, configured to perform time synchronization on the first historical data and the second historical data to obtain a plurality of time-synchronized data frame pairs; each data frame pair includes a first data frame and a second data frame synchronized at a sampling time; a conversion subunit, configured to perform coordinate system conversion on the first data frame and the second data frame in each data frame pair to obtain spatially synchronized data pairs, each data pair comprising first data in the first spatially synchronized data frame and second data in the second spatially synchronized data frame; Correspondingly, the establishing unit is specifically configured to establish the fitting mapping model based on the corresponding relationship between the first feature information corresponding to the first data and the second feature information corresponding to the second data in each of the data pairs.

12. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 10 are implemented.

13. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 10 are implemented.

Images