LiDAR, information collection system and roadside base station

Abstract

The present application relates to a laser radar, an information collection system and a roadside base station. The laser radar includes a transceiver, a memory and a processor, wherein the memory stores a mapping model, and the transceiver is used to obtain point cloud data within the perception range; the processor is used to call the mapping model in the memory to process the point cloud data to obtain color features; wherein the mapping model contains a mapping relationship between the point cloud features of the point cloud data and the color features of the two-dimensional image of the visual sensor, wherein the point cloud features include depth features and / or intensity features, and the visual sensor is used to collect two-dimensional images within the perception range. When the laser radar is used, when the visual sensor cannot provide the color features of the two-dimensional image, the color features corresponding to the point cloud data can also be obtained through the laser radar.

Description

Technical Field

[0001] The present application relates to the field of sensor technology, and in particular to a laser radar, a multi-sensor information collection system, a roadside base station and a radar information collection system. Background Art

[0002] Intelligent transportation system refers to the provision of real-time traffic information from various locations by traffic participants to the traffic information center through sensors and transmission equipment installed on roads and vehicles. After receiving and processing this information, the traffic information center can provide traffic participants with road traffic information and other travel-related information. Travelers can determine their own travel methods and choose routes based on this information, thereby ensuring traffic safety.

[0003] In related technologies, when providing real-time traffic information from various places to a traffic information center, image data in road scenes are usually collected through a monocular camera, and depth information such as distance in road scenes is collected through a radar. The camera and radar then transmit their collected information to the traffic information center for processing.

[0004] However, the above technology is difficult to provide image data to the traffic information center when the camera fails. Summary of the invention

[0005] Based on this, it is necessary to provide a laser radar, multi-sensor information collection system, roadside base station and radar information collection system that can still provide image data when the camera fails to solve the above technical problems.

[0006] In a first aspect, a laser radar is provided, comprising a transceiver, a memory and a processor, wherein the memory stores a mapping model,

[0007] The above transceiver is used to obtain point cloud data within the sensing range;

[0008] The above-mentioned processor is used to call the mapping model in the memory to process the point cloud data to obtain color features; wherein the above-mentioned mapping model contains the mapping relationship between the point cloud features of the point cloud data and the color features of the two-dimensional image of the visual sensor, the point cloud features include depth features and / or intensity features, and the visual sensor is used to collect two-dimensional images within the perception range.

[0009] In one embodiment, the processor is further configured to:

[0010] Acquire first historical data collected by a visual sensor and second historical data collected by a laser radar in the same time period and the same scene; the first historical data is a two-dimensional image of a road scene containing historical objects, and the second historical data is point cloud data of the road scene containing historical objects;

[0011] Performing spatiotemporal synchronization processing on the first historical data and the second historical data to obtain a spatiotemporal mapping relationship between the first historical data and the second historical data;

[0012] Performing feature extraction on the first historical data to obtain color features, and performing feature extraction on the second historical data to obtain depth features and / or intensity features;

[0013] Based on the spatiotemporal mapping relationship between the first historical data and the second historical data, the color feature is associated with the depth feature and / or the intensity feature to establish a mapping model.

[0014] In one embodiment, the processor is specifically configured to

[0015] Performing polar coordinate conversion on the acquired second historical data to obtain a two-dimensional point cloud;

[0016] Mapping the point cloud points in the two-dimensional point cloud to the coordinate system of the first historical data using the calibration parameters to obtain a mapping relationship between the point cloud points and the pixel points in the two-dimensional image;

[0017] The mapping relationship is used to mark the color features corresponding to the point cloud clusters in the second historical data.

[0018] In one embodiment, the mapping model is a deep learning model, and the processor is specifically used to

[0019] Processing the second historical data to obtain at least one point cloud cluster corresponding to the historical object in the second historical data; each point cloud cluster corresponds to a color feature;

[0020] According to at least one point cloud cluster, determining point cloud features corresponding to each point cloud cluster;

[0021] The point cloud features corresponding to each point cloud cluster in the second historical data and the color features corresponding to each point cloud cluster are labeled to construct training samples for the deep learning model.

[0022] In one embodiment, the processor is specifically configured to

[0023] The above training samples are input into the initial deep learning model to obtain output information; a regression operation is performed on the output information to obtain the corresponding color result matrix, and the loss between the color result matrix and the corresponding color features in the training sample is calculated using the loss function to obtain the loss value; the loss function is iteratively optimized using the gradient descent algorithm until the loss value is less than the preset threshold, thereby obtaining a deep learning model.

[0024] In one embodiment, the mapping model is a fitting mapping model, the fitting mapping model includes a correspondence table between point cloud features and color characteristics, and the processor is specifically used to

[0025] The input point cloud data is processed according to the corresponding table to obtain the corresponding color features.

[0026] In one embodiment, the point cloud feature further includes an echo area feature and / or a roughness feature, and the processor is specifically configured to:

[0027] According to the above-mentioned depth feature, intensity feature, echo area feature, roughness feature and the preset function correspondence, the corresponding color feature is determined.

[0028] In a second aspect, a multi-sensor information acquisition system is provided, comprising the above-mentioned laser radar and visual sensor.

[0029] In a third aspect, a roadside base station is provided, comprising the above-mentioned multi-sensor information collection system and a roadside unit, wherein the visual sensor of the multi-sensor information collection system and the laser radar of the multi-sensor information collection system are both communicatively connected to the roadside unit,

[0030] The visual sensor is used to obtain a two-dimensional image within the sensing range and transmit the two-dimensional image to the roadside unit;

[0031] The above-mentioned laser radar is used to obtain point cloud data within the sensing range and transmit the point cloud data to the roadside unit;

[0032] The above-mentioned roadside unit is used to process the received two-dimensional image and point cloud data to obtain the perception information of the roadside base station.

[0033] In a fourth aspect, a radar information collection system is provided, including a transceiver and an edge device.

[0034] The above transceiver is used to obtain point cloud data within the sensing range;

[0035] The above-mentioned edge device is used to process point cloud data using a preset mapping model to obtain color features; wherein the above-mentioned mapping model includes a mapping relationship between point cloud features of the point cloud data and color features of a two-dimensional image of a visual sensor, the point cloud features include depth features and / or intensity features, and the visual sensor is used to collect two-dimensional images within a perception range.

[0036] The above-mentioned laser radar, multi-sensor information collection system, roadside base station and radar information collection system, the laser radar includes a transceiver, a memory and a processor, the memory stores a mapping model, the above-mentioned transceiver is used to obtain point cloud data within the perception range, and the above-mentioned processor is used to call the mapping model in the memory to process the point cloud data to obtain color features; wherein, the mapping model includes the mapping relationship between the point cloud features and the color features of the two-dimensional image of the visual sensor, the point cloud features include depth features and / or intensity features, and the visual sensor is used to collect two-dimensional images within the perception range. Through the laser radar, a mapping model containing the mapping relationship between the point cloud features and the color features of the two-dimensional image can be established, and the point cloud features include depth features and / or intensity features. In this way, when the visual sensor fails and cannot provide the color features of the two-dimensional image, the laser radar itself can also obtain the color features corresponding to the point cloud data through the established mapping model, thereby providing the color features of the two-dimensional image to the intelligent traffic information center, so that the intelligent traffic information center can process the provided color features to ensure the safe driving of each object in the target scene. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 is a diagram of the internal structure of a laser radar in one embodiment;

[0038] Figure 2 A schematic diagram of a process for training a deep learning model in one embodiment;

[0039] Figure 3 is a structural block diagram of a multi-sensor information acquisition system in one embodiment;

[0040] Figure 4 is a structural block diagram of a roadside base station in an embodiment;

[0041] Figure 5 1 is a structural block diagram of a radar information acquisition system in one embodiment. DETAILED DESCRIPTION

[0042] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0043] In one embodiment, a laser radar is provided. Figure 1As shown, the laser radar includes a transceiver, a memory and a processor, the memory stores a mapping model, the transceiver is used to obtain point cloud data within the perception range; the processor is used to call the mapping model in the memory to process the point cloud data to obtain color features; wherein the mapping model contains a mapping relationship between the point cloud features of the point cloud data and the color features of the two-dimensional image of the visual sensor, the point cloud features include depth features and / or intensity features, and the visual sensor is used to collect two-dimensional images within the perception range.

[0044] The processor and transceiver may be an integrated device or a separate device. When the processor and transceiver are separate devices, the processor may be an edge device or a cloud server.

[0045] The sensing range refers to the range in which data can be detected. The sensing ranges of the radar sensor and the visual sensor can be different or the same. In this embodiment, when color feature mapping is performed, color feature mapping is mainly performed on point cloud data within the sensing range of the radar sensor.

[0046] The image within the above-mentioned perception range may be an image of a target scene, which may be a road scene or other scenes. The road scene may be an outdoor road scene or an indoor amusement road scene, etc. The target scene may include a target object or may not include a target object. When a target object is included, the target object may be a vehicle, a pedestrian, etc. in the road scene.

[0047] Here, the laser radar may include 8-line, 16-line, 24-line, 128-line laser radars, etc. The visual sensor may be a monocular camera. Usually, the laser radar and the visual sensor may be set on the side poles in the road (which may be vertical poles, horizontal poles, etc.), so that the laser radar can collect point cloud data of vehicles or pedestrians in the road scene, and the visual sensor can collect two-dimensional images of vehicles or pedestrians in the road scene. Of course, the laser radar and the visual sensor may also be set on the target object in the road scene, for example, they may be set on the vehicle in the road to capture the point cloud data and two-dimensional images of other vehicles or pedestrians in the road scene. It should be noted that the laser radar and the visual sensor are usually set in a relatively close position, which can facilitate the calibration of the laser radar and the visual sensor, thereby making the collected data more accurate.

[0048] Before the processor calls the mapping model in the memory to process the point cloud data and obtains the color features corresponding to the point cloud data, the processor can also determine whether the roadside only includes a laser radar. If so, the processor directly calls the mapping model to process the point cloud data. If not, that is, the roadside includes visual sensors and laser radars, etc., then it is possible to first detect whether the visual sensor fails. If the visual sensor fails, the processor can be used to call the mapping model to process the point cloud data. The visual sensor failure here refers to the damage of the visual sensor, the damage of part of the data collected by the visual sensor, the distortion of the data collected by the visual sensor due to objective reasons such as weather environment, data loss, data unavailability, etc.

[0049] In addition, the mapping model here can be a fitting mapping model or a deep learning model. Therefore, before using the point cloud features on the point cloud data to obtain the color features on the two-dimensional image, a fitting mapping model or a deep learning model between the point cloud features on the point cloud data and the color features on the two-dimensional image can be obtained first.

[0050] When obtaining the fitting mapping model and the deep learning model, the historical image data and historical point cloud data at various times in the same scene can be collected, wherein the historical point cloud data includes the echo intensity and / or depth information of the historical object, and the depth information can represent the distance between the historical object and the collection device, as well as the x, y coordinates and related angles in the physical coordinate system, and the collection device here refers to the radar; the historical image data includes the position of the historical object on the image and the color information at the position, and the historical image data can be measured by a visual sensor, and the historical object can be a vehicle, a pedestrian, etc. in the scene; then the color information at the same position in the historical image data at the same time and the depth information and / or echo intensity in the point cloud data are associated to obtain the mapping relationship between the two, that is, to obtain the fitting mapping model. Similarly, the historical point cloud data at the same time can be used as the input of the initial deep learning model, and the color information at the same position in the historical image data at the time can be used as a label to train the initial deep learning model to obtain the deep learning model.

[0051] It should be noted that, as far as the roadside is concerned, visual sensors and lidars are usually installed at the same position on the roadside. Therefore, the depth features in the historical point cloud data mentioned above can represent the distance between the historical object and the lidar, which is actually the distance between the visual sensor.

[0052] After the fitting mapping model or the deep learning model is established, the point cloud data can be input into the fitting mapping model or the deep learning model to obtain the color features of each point cloud point in the point cloud data.

[0053] Through the laser radar in this embodiment, both the point cloud features on the point cloud data and the color features on each point cloud point in the point cloud data can be obtained. It can be seen that this embodiment can realize the acquisition of point cloud features and color features through a laser radar. In this way, in actual application, there is no need to install too many sensors on the roadside or on the vehicle. Only one laser radar of this embodiment needs to be installed to realize the functions of both the camera and the radar, thereby reducing the overall cost of the system. At the same time, maintenance only needs to be maintained by the laser radar, which is relatively easy and reduces maintenance costs. Furthermore, system space can be saved and the number of sensors in the system can be reduced.

[0054] It should be noted that Figure 1 The structure shown in is only a block diagram of a partial structure related to the scheme of the present application, and does not constitute a limitation on the visual sensor to which the scheme of the present application is applied. The specific visual sensor may include more or fewer components than shown in the figure, or combine certain components, or have different component arrangements. In addition, any reference to memory, storage, database or other media used in the embodiments provided by the present application may include at least one of non-volatile and volatile memory. Non-volatile memory may include read-only memory (ROM), magnetic tape, floppy disk, flash memory or optical memory, etc. Volatile memory may include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM may be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc.

[0055] The above-mentioned laser radar includes a transceiver, a memory and a processor, the memory stores a mapping model, the above-mentioned transceiver is used to obtain point cloud data within the perception range, and the above-mentioned processor is used to call the mapping model in the memory to process the point cloud data to obtain color features; wherein, the mapping model includes a mapping relationship between point cloud features and color features of a two-dimensional image of a visual sensor, the point cloud features include depth features and / or intensity features, and the visual sensor is used to collect a two-dimensional image within the perception range. Through the laser radar, a mapping model including a mapping relationship between point cloud features and color features of a two-dimensional image can be established, and the point cloud features include depth features and / or intensity features, so that when the visual sensor fails and cannot provide the color features of the two-dimensional image, the laser radar itself can also obtain the color features corresponding to the point cloud data through the established mapping model, thereby providing the color features of the two-dimensional image to the intelligent traffic information center, so that the intelligent traffic information center can process the provided color features to ensure the safe driving of each object in the target scene.

[0056] In another embodiment, another laser radar is provided, and the processor is specifically used to obtain first historical data collected by a visual sensor and second historical data collected by a laser radar in the same time period and the same scene; the first historical data is a two-dimensional image containing historical objects in a road scene, and the second historical data is point cloud data containing historical objects in a road scene; the first historical data and the second historical data are processed in time and space synchronization to obtain a time and space mapping relationship between the first historical data and the second historical data; feature extraction is performed on the first historical data to obtain color features, and feature extraction is performed on the second historical data to obtain depth features and / or intensity features; based on the time and space mapping relationship between the first historical data and the second historical data, the color features are associated with the depth features and / or intensity features to establish a mapping model.

[0057] In this embodiment, the first historical data and the second historical data are data collected from historical objects in the same road scene in the same time period at a historical moment.

[0058] Before establishing a mapping model based on the first historical data and the second historical data, the laser radar can synchronize the first historical data and the second historical data in time and space so as to correspond the first historical data and the second historical data collected in the same time period and the same scene.

[0059] Optionally, the processor is specifically used to perform time synchronization on the first historical data and the second historical data to obtain multiple time-synchronized data frame pairs; each data frame pair includes a first data frame and a second data frame synchronized at a sampling moment; coordinate system conversion is performed on the first data frame and the second data frame in each data frame pair to obtain spatially synchronized data pairs, each data pair includes the first data in the first data frame and the second data in the second data frame that are spatially synchronized.

[0060] The first historical data may include multiple first data frames, and the second historical data may include multiple second data frames. The processor may determine whether the first data frame and the second data frame are time synchronized according to the acquisition time of the first data frame and the second data frame, and then determine the first data frame and the second data frame synchronized in sampling time as a data pair.

[0061] When the processor performs time synchronization according to the sampling time, it can obtain the sampling time of each first data frame and each second data frame, and then compare the obtained sampling times to determine the synchronized data frames; in addition, the processor can also first determine the second data frame synchronized with the first first data frame, and then calculate the positions of other second data frames synchronized with the first data frames in turn according to the sampling frequencies of the first historical data and the second historical data; the above time synchronization method is not limited here.

[0062] The sampling frequencies of the first historical data and the second historical data may be the same or different. When they are different, the sampling frequency of the second historical data may be a multiple of the sampling frequency of the first historical data, and the multiple may be 1, 2, 3, etc. For example, the sampling frequency of the second historical data is 20KHz, and the sampling frequency of the first historical data is 10KHz. After the processor determines the data pair synchronized at the first sampling moment, the data frame after the first data frame in the data pair in the first historical data and the data frame after the second data frame in the data pair in the second historical data may be determined as the second data pair, and the time synchronization of each data frame may be completed in sequence, thereby improving the time synchronization efficiency of the first historical data and the second historical data.

[0063] When the processor performs spatial synchronization on the data frame pair after time synchronization, it can optionally perform polar coordinate conversion on the acquired second historical data to obtain a two-dimensional point cloud; use calibration parameters to map the point cloud points in the two-dimensional point cloud to the coordinate system of the first historical data, and obtain the mapping relationship between the point cloud points and the pixel points in the two-dimensional image; use the mapping relationship to mark the color features corresponding to the point cloud clusters in the second historical data.

[0064] That is to say, after obtaining the second historical data, usually the second historical data is collected in the world coordinate system or the physical coordinate system, then the coordinates of the point cloud data in the second historical data can be converted into polar coordinates, usually two-dimensional coordinates (including polar diameter and polar angle). Then, using the preset conversion relationship between the world coordinate system and the camera coordinate system (the camera coordinate system here refers to the coordinate system where the visual sensor is located, and the conversion relationship can be obtained when calibrating the laser radar and the visual sensor), the polar coordinates of the point cloud data are converted from the world coordinate system to the camera coordinate system to obtain the coordinates of the point cloud data in the camera coordinate system, that is, correspond to the position of the pixel points on the image in the camera coordinate system. Afterwards, after the coordinates are matched, the point cloud features on the point cloud data at the same position can be bound to the color features at that position to obtain a data pair consisting of the point cloud features and the color features.

[0065] After obtaining the data pairs consisting of point cloud features and color features, the data pairs can be used to train the mapping model. Optionally, the mapping model can be a deep learning model or a fitting mapping model. The following describes these two implementation methods:

[0066] In one implementation, the mapping model is described as a deep learning model:

[0067] When establishing a deep learning model, optionally, the above-mentioned processor is specifically used to process the second historical data to obtain at least one point cloud cluster corresponding to the historical object in the second historical data; each point cloud cluster corresponds to a color feature; based on at least one point cloud cluster, the point cloud features corresponding to each point cloud cluster are determined; the point cloud features corresponding to each point cloud cluster in the second historical data and the color features corresponding to each point cloud cluster are labeled to construct a training sample for the deep learning model.

[0068] Here, the second historical data may include one or more point cloud clusters of a historical object. When multiple point cloud clusters are included, each point cloud cluster may correspond to a different color feature. For example, a person wears a white top and black pants, and the top of the person object corresponds to one point cloud cluster, and the pants correspond to another point cloud cluster.

[0069] Then, the point cloud data corresponding to the historical object can be divided into multiple point cloud clusters by using the color features of each point cloud cluster known in advance, and then the intensity features and / or depth features corresponding to each point cloud cluster can be obtained by using the point cloud data corresponding to each point cloud cluster. For example, the intensity features of each point cloud point in a point cloud cluster can be summed and averaged, and the average intensity feature obtained can be used as the intensity feature corresponding to the point cloud cluster; the depth features of each point cloud cluster can be obtained in the same way.

[0070] After obtaining the point cloud features of each point cloud cluster, the point cloud cluster can be labeled with the color features corresponding to the point cloud cluster, that is, the point cloud features of each point cloud cluster and its corresponding color features are bound together to form a data pair as a training sample for the deep learning model.

[0071] After obtaining the training samples, the training samples can be used to train the initial deep learning model. During training, the above training samples can be optionally input into the initial deep learning model to obtain output information; a regression operation is performed on the output information to obtain a corresponding color result matrix, and a loss function is used to calculate the loss between the color result matrix and the corresponding color features in the training samples to obtain a loss value; the loss function is iteratively optimized using a gradient descent algorithm until the loss value is less than a preset threshold, thereby obtaining a deep learning model.

[0072] Here, you can see Figure 2 As shown, when the second historical data is processed as above to obtain each point cloud cluster, usually the point cloud cluster of the object far away from the laser radar has fewer point cloud points, and the point cloud cluster of the object close to the laser radar has more point cloud points. In order to balance the number of sample point cloud points, the size of each point cloud cluster can be calculated, that is, the same point cloud points are selected from the number of point cloud points of each point cloud cluster to participate in model training. For example, one point cloud cluster includes 50 point cloud points and the other point cloud cluster includes 40 point cloud points. Then, 40 point cloud points or 35 point cloud points, or another number of point cloud points, can be selected from each of the two point cloud clusters to participate in model training.

[0073] After selecting the point cloud points of each point cloud group, referring to the above method, the point cloud features on each point cloud group can be obtained, and then the point cloud features of each point cloud group can be arranged in a certain order to establish a data matrix. After that, the established data matrix can be preprocessed, including normalization, flipping, shuffling and other processing processes to obtain the preprocessed data matrix. Preprocessing the data matrix here can make the data more standardized, more random, and the model easier to train.

[0074] After obtaining the preprocessed data matrix, the preprocessed data matrix can be input into the initial deep learning model for feature extraction, that is, the initial deep learning model is used to perform downsampling, upsampling and multi-scale feature splicing on the data matrix to extract the point cloud feature information of each point cloud group to form a point cloud feature map. The point cloud feature map here can be recorded as output information; the size of the point cloud feature map can be n*n*c, and the sizes of n and c can be set according to the internal structure of the deep learning model.

[0075] After obtaining the point cloud feature map, the point cloud features can be input into the softmax layer of the initial deep learning model for regression to obtain a color result matrix corresponding to the input data matrix. The size of the color result matrix is ​​usually the same as the size of the input data matrix, but it can also be different. This embodiment is mainly described by taking the same size as an example.

[0076] Then, after obtaining the color result matrix, the data at each position in the color result matrix corresponds to the data at each position in the input data matrix, that is, for the data at any position in the color result matrix, the color feature marked at that position in the input data matrix can be obtained. After that, the loss function can be used to calculate the loss between the data at each position in the color result matrix and the color features at each position in the corresponding data matrix, and the obtained losses are summed to obtain a loss sum value, which is recorded as the value of the loss function. Then, the loss function is iterated using gradient descent and Adam optimizer, and during the iteration process, it is determined whether the loss value of the loss function is less than the preset threshold. When the loss value is less than the preset threshold or reaches the preset number of iterations, the iteration stops, and the parameters of the deep learning model are fixed at this time, and the trained deep learning model is obtained.

[0077] Here, the loss function can be set according to the actual situation, and no specific limitation is made here. The preset threshold and the preset number of iterations can also be set according to the actual situation, and no specific limitation is made here. For example, the preset threshold can be 0.01, the preset number of iterations can be 1000, and so on.

[0078] After the deep learning model is trained as described above, the point cloud data can be input into the trained deep learning model to obtain the color features corresponding to the point cloud features in the point cloud data.

[0079] In another implementation, the mapping model is described as a fitting mapping model:

[0080] When establishing the fitting mapping model, optionally, the fitting mapping model includes a correspondence table between point cloud features and color features.

[0081] Optionally, a correspondence can be established between the point cloud features and color features in each data pair through the data pairs consisting of the point cloud features and color features established above, and a correspondence can be established between the point cloud features and color features in each data pair in turn. A correspondence table containing the correspondence between the point cloud features and the color features can be obtained, and then the point correspondence table can be directly used as a fitting mapping model.

[0082] Of course, the point cloud features and color features in the above correspondence relationship may be interpolated to obtain a correspondence table containing more correspondence relationships between point cloud features and color features as a fitting mapping model.

[0083] In short, a fitting mapping model can be established in the above manner. After the fitting mapping model is established, the established fitting mapping model can be used to process the point cloud data. Optionally, the input point cloud data can be processed according to the corresponding table to obtain the corresponding color features. In other words, the point cloud data can be input into the fitting mapping model, and the corresponding table of the fitting mapping model can be searched to obtain the color features corresponding to the point cloud features of the point cloud data.

[0084] Of course, it is also possible that the corresponding color feature cannot be found in the corresponding table. In this case, several point cloud features close to the point cloud features of the point cloud data can be obtained in the corresponding table, and the color features corresponding to these point cloud features can be interpolated. The interpolated color features can be used as the color features corresponding to the point cloud features of the point cloud data. In this way, it can be ensured that the point cloud features of all point cloud data can obtain corresponding color features.

[0085] In this embodiment, the first historical data collected by the visual sensor and the second historical data collected by the laser radar in the same time period and the same scene can be processed in time and space synchronization to obtain a time and space mapping relationship, and the color features of the first historical data and the point cloud features of the second historical data can be associated based on the time and space mapping relationship to establish a mapping model. By performing time and space synchronization on the first historical data and the second historical data, the data source used in establishing the mapping model can be more accurate, so that the mapping model finally obtained based on the more accurate data source will be more accurate.

[0086] In another embodiment, another laser radar is provided, wherein the point cloud features also include echo area features and / or roughness features, and the processor is specifically used to determine the corresponding color features based on the correspondence between the depth features, intensity features, echo area features, roughness features and preset functions.

[0087] Here, there is a certain information correlation between the color feature and the depth feature, intensity feature, echo area feature, roughness feature, etc., for example, it can be a first-order function relationship, or a second-order function relationship, etc. In short, there is a certain functional relationship between the color feature of the point cloud data and the point cloud feature.

[0088] This functional relationship can be obtained through training of a neural network model. According to this functional relationship, after obtaining the depth feature, roughness feature, echo area feature, and intensity feature of the point cloud data, the corresponding color feature can be obtained.

[0089] In this embodiment, the corresponding color features can be determined through the correspondence between depth features, intensity features, echo area features, roughness features and preset functions. This can comprehensively consider various aspects of the point cloud data and combine more point cloud feature information, thereby making the final color features more accurate.

[0090] In one embodiment, a multi-sensor information acquisition system is provided. Figure 3 As shown, it includes the above-mentioned laser radar and visual sensor. Of course, the multi-sensor information acquisition system can also include other components.

[0091] Through the new multi-sensor acquisition system of this embodiment, since the laser radar included therein can establish a mapping model that includes the mapping relationship between point cloud features and color features of two-dimensional images, and the point cloud features include depth features and / or intensity features, when the visual sensor fails and cannot provide the color features of the two-dimensional image, the laser radar itself can also obtain the color features corresponding to the point cloud data through the established mapping model, thereby providing the color features of the two-dimensional image to the intelligent traffic information center, so that the intelligent traffic information center can process the provided color features to ensure the safe driving of each object in the target scene.

[0092] In one embodiment, a roadside base station is provided. Figure 4 As shown, it includes the above-mentioned multi-sensor information collection system and the roadside unit. The visual sensor of the multi-sensor information collection system and the laser radar of the multi-sensor information collection system are both communicatively connected with the roadside unit. The above-mentioned visual sensor is used to obtain the two-dimensional image within the perception range and transmit the two-dimensional image to the roadside unit; the above-mentioned laser radar is used to obtain the point cloud data within the perception range and transmit the point cloud data to the roadside unit; the above-mentioned roadside unit is used to process the received two-dimensional image and point cloud data to obtain the perception information of the roadside base station.

[0093] The visual sensor and laser radar are both connected to the roadside unit, and the data collected by each can be transmitted to the roadside unit. The connection between the roadside unit and the visual sensor and laser radar can be a wired or wireless connection. The roadside unit can be a computer device, such as a terminal, a server, an edge device, etc. The perception information of the roadside base station can be the depth information of each pixel on the two-dimensional image, the speed, acceleration, trajectory, angular velocity, heading angle and other information of the target object on the image.

[0094] Through the roadside base station of this embodiment, since the laser radar included therein can establish a mapping model that includes the mapping relationship between point cloud features and color features of two-dimensional images, and the point cloud features include depth features and / or intensity features, when the visual sensor fails and cannot provide the color features of the two-dimensional image, the laser radar itself can also obtain the color features corresponding to the point cloud data through the established mapping model, thereby providing the color features of the two-dimensional image to the intelligent traffic information center, so that the intelligent traffic information center can process the provided color features to ensure the safe driving of each object in the target scene.

[0095] In one embodiment, a radar information collection system is provided. Figure 5 As shown, it includes a transceiver and an edge device, the transceiver is used to obtain point cloud data within the perception range; the edge device is used to process the point cloud data using a preset mapping model to obtain color features; wherein the mapping model includes a mapping relationship between the point cloud features of the point cloud data and the color features of the two-dimensional image of the visual sensor, the point cloud features include depth features and / or intensity features, and the visual sensor is used to collect two-dimensional images within the perception range.

[0096] The specific process of the edge device establishing the mapping model and using the established mapping model can be the same as the steps performed by the processor in the above-mentioned lidar, and will not be repeated here.

[0097] Through the radar information collection system of this embodiment, a mapping model including the mapping relationship between point cloud features and color features of two-dimensional images can be established, and the point cloud features include depth features and / or intensity features. In this way, when the visual sensor fails and cannot provide the color features of the two-dimensional image, the laser radar itself can also obtain the color features corresponding to the point cloud data through the established mapping model, thereby providing the color features of the two-dimensional image to the intelligent traffic information center, so that the intelligent traffic information center can process the provided color features to ensure the safe driving of each object in the target scene.

[0098] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0099] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.

Claims

1. A laser radar, comprising a transceiver, a memory and a processor, wherein the memory stores a mapping model, characterized in that: The transceiver is used to obtain point cloud data within the sensing range; The processor is used to call the mapping model in the memory to process the point cloud data to obtain color features; wherein the mapping model contains a mapping relationship between point cloud features of the point cloud data and color features of a two-dimensional image of a visual sensor, the point cloud features include depth features and / or intensity features, and the visual sensor is used to capture a two-dimensional image within the perception range.

2. The laser radar according to claim 1, characterized in that: The processor is also used to Acquire first historical data collected by the visual sensor and second historical data collected by the laser radar in the same time period and the same scene; the first historical data is a two-dimensional image containing historical objects in the road scene, and the second historical data is point cloud data containing historical objects in the road scene; Performing spatiotemporal synchronization processing on the first historical data and the second historical data to obtain a spatiotemporal mapping relationship between the first historical data and the second historical data; Performing feature extraction on the first historical data to obtain color features, and performing feature extraction on the second historical data to obtain depth features and / or intensity features; Based on the spatiotemporal mapping relationship between the first historical data and the second historical data, the color feature is associated with the depth feature and / or intensity feature to establish the mapping model.

3. The laser radar according to claim 2, characterized in that: The processor is specifically used for Performing polar coordinate conversion on the acquired second historical data to obtain a two-dimensional point cloud; Mapping the point cloud points in the two-dimensional point cloud to the coordinate system of the first historical data using calibration parameters to obtain a mapping relationship between the point cloud points and the pixel points in the two-dimensional image; The mapping relationship is used to mark the color features corresponding to the point cloud clusters in the second historical data.

4. The laser radar according to claim 3, characterized in that: The mapping model is a deep learning model, and the processor is specifically used to Processing the second historical data to obtain at least one point cloud cluster corresponding to the historical object in the second historical data; each point cloud cluster corresponds to a color feature; Determining point cloud features corresponding to each point cloud cluster according to the at least one point cloud cluster; The point cloud features corresponding to each point cloud cluster in the second historical data and the color features corresponding to each point cloud cluster are labeled to construct training samples for the deep learning model.

5. The laser radar according to claim 4, characterized in that: The processor is specifically used for The training samples are input into the initial deep learning model to obtain output information; a regression operation is performed on the output information to obtain a corresponding color result matrix, and a loss function is used to calculate the loss between the color result matrix and the corresponding color features in the training samples to obtain a loss value; the loss function is iteratively optimized using a gradient descent algorithm until the loss value is less than a preset threshold, thereby obtaining the deep learning model.

6. The laser radar according to claim 3, characterized in that: The mapping model is a fitting mapping model, and the fitting mapping model includes a correspondence table between point cloud features and color features. The processor is specifically used to The input point cloud data is processed according to the correspondence table to obtain corresponding color features.

7. The laser radar according to any one of claims 1 to 6, characterized in that: The point cloud features also include echo area features and / or roughness features, and the processor is specifically configured to The corresponding color feature is determined according to the depth feature, the intensity feature, the echo area feature, the roughness feature and a preset function correspondence.

8. A multi-sensor information acquisition system, characterized in that: It comprises a laser radar and a visual sensor as described in any one of claims 1 to 7.

9. A roadside base station, characterized in that: The method comprises the multi-sensor information collection system as claimed in claim 8 and a roadside unit, wherein the visual sensor of the multi-sensor information collection system and the laser radar of the multi-sensor information collection system are both connected to the roadside unit for communication, The visual sensor is used to obtain a two-dimensional image within a sensing range and transmit the two-dimensional image to a roadside unit; The laser radar is used to obtain point cloud data within the sensing range and transmit the point cloud data to the roadside unit; The roadside unit is used to process the received two-dimensional image and point cloud data to obtain perception information of the roadside base station.

10. A radar information collection system, comprising a transceiver and an edge device, characterized in that: The transceiver is used to obtain point cloud data within the sensing range; The edge device is used to process the point cloud data using a preset mapping model to obtain color features; wherein the mapping model includes a mapping relationship between point cloud features of the point cloud data and color features of a two-dimensional image of a visual sensor, the point cloud features include depth features and / or intensity features, and the visual sensor is used to capture two-dimensional images within the perception range.

Images