Vehicle sensing data processing method and device, vehicle, and storage medium

Abstract

The application relates to a sensing data processing method and device of a vehicle, the vehicle and a storage medium, wherein the method comprises the following steps: acquiring original sensing data collected by a plurality of vehicle-mounted sensors of a current vehicle; uniformly converting the collected original sensing data into a multi-channel two-dimensional representation, and composing a multi-mode original data layer to obtain sensing data based on the original data layer; inputting the sensing data into a pre-trained multi-modal deep convolution network, performing convolution processing to obtain at least one fusion feature, and identifying perception information of at least one target based on the at least one fusion feature, wherein a convolution kernel group of the pre-trained multi-modal deep convolution network is obtained based on the original data layer. According to the application, the collected original data is composed into a multi-mode original data layer, and variable convolution kernels are used to extract features from the original data layer to generate an identification result, so that the network maintenance cost is reduced, the overfitting of the network is avoided, and the generalization ability of the network is improved.

Description

Technical Field

[0001] This application relates to the field of sensor fusion technology, and in particular to a method, apparatus, vehicle, and storage medium for processing sensor data of a vehicle. Background Technology

[0002] Current deep network recognition technology has achieved remarkable results in the field of vehicle perception and recognition, leading the way for the practical application of autonomous driving. However, different sensors require different configurations, such as... Figure 1 As shown, this not only requires pre-configuration of sensor parameters and installation locations, but also necessitates extensive and complex adaptation work by engineers. Furthermore, the cost of different sensors varies significantly. Even within the same vehicle model, sensors are assembled in stages across different levels of assisted driving or autonomous driving, resulting in perception networks based on each sensor either operating independently or through fixed fusion. Figure 2 As shown, this results in different recognition networks depending on the sensor configuration, leading to a significant workload for network adaptation and maintenance. Furthermore, the raw signals and formats detected by different sensors vary. For example, due to differences in their photosensitive elements, cameras produce images of varying quality after processing by different ISP (Image Signal Processor) algorithms. Figure 3 As shown, different sensors will have different preprocessing algorithms and recognition network algorithms, which makes algorithm migration and reuse difficult.

[0003] Currently, related technologies can obtain spatial alignment points through measurement information from millimeter-wave radar and cameras, select regions of interest including the target vehicle, and thus obtain the vehicle's lower boundary shadow area corresponding to the gray-level region divided by the minimum threshold, thereby obtaining the vehicle's lower boundary shadow line and realizing vehicle lower boundary detection. In addition, related technologies can detect and perceive the target vehicle's driving environment based on multi-sensor fusion technology, determine environmental perception information, and thus realize automatic control of the target vehicle.

[0004] However, the relevant technologies can only fuse specific sensors, and the network models for different sensor configurations are also different, resulting in a large maintenance workload. It is impossible to process different sensor configurations through a unified network model, which urgently needs to be solved. Summary of the Invention

[0005] This application provides a vehicle sensor data processing method, device, vehicle, and storage medium to solve the problems that related technologies can only fuse specific sensors, and the network models for different sensor configurations are also different, resulting in a large maintenance workload and the inability to process different sensor configurations through a unified network model.

[0006] The first aspect of this application provides a method for processing vehicle sensor data, comprising the following steps: acquiring raw sensor data collected by multiple on-board sensors of the current vehicle; unifying the format of the acquired raw sensor data into a multi-channel two-dimensional representation and forming a multi-modal raw data layer, obtaining sensor data based on the raw data layer; and inputting the sensor data into a pre-trained multi-modal deep convolutional network, performing convolution processing to obtain at least one fusion feature, and identifying perceptual information of at least one target based on the at least one fusion feature, wherein the multiple sets of convolutional kernels of the pre-trained multi-modal deep convolutional network are obtained from the data of the multiple on-board sensors.

[0007] Based on the above technical means, the embodiments of this application form a multi-modal raw data layer by assembling the collected raw data and extracting features from the raw data layer according to the variable convolution kernel to generate recognition results. This enables the use of a unified recognition network to adaptively extract features from raw data under different sensor configurations for recognition and detection, greatly reducing network maintenance costs and improving the efficiency and performance of vehicle sensor data analysis and processing.

[0008] Optionally, in one embodiment of this application, obtaining sensor data based on the original data layer includes: linking the original sensor data using a preset high-dimensional array; or, processing the original sensor data based on a preset preprocessing network.

[0009] Based on the above technical means, the embodiments of this application can link the original sensing data through a high-dimensional array, or process the original sensing data based on a corresponding preprocessing network, so as to achieve a unified representation of sensor data in different formats, thereby providing reliable fusion data for subsequent feature extraction.

[0010] Optionally, in one embodiment of this application, before acquiring the raw sensing data collected by multiple on-board sensors of the current vehicle, the method further includes: performing spatiotemporal calibration on the multiple on-board sensors.

[0011] Based on the above technical means, this application embodiment determines the spatiotemporal relationship of different sensors by performing spatiotemporal calibration on multiple vehicle-mounted sensors, and can obtain information on the same obstacle collected by different sensors, which facilitates subsequent fusion processing.

[0012] The first aspect of this application provides a method for processing vehicle sensor data, which further includes the following steps: determining the design information of multiple sets of convolutional kernel groups based on data from multiple sensors, so as to generate corresponding fusion features when extracting multiple sensor combinations; training the network based on the design information of the multiple sets of convolutional kernel groups and a preset convolutional group selection mechanism to obtain a pre-trained multimodal deep convolutional network, so as to input the sensor data into the multimodal deep convolutional network to obtain at least one fusion feature.

[0013] Based on the above technical means, the embodiments of this application determine the design information of multiple sets of convolutional kernel groups based on data from multiple sensors, and train the multimodal deep convolutional network, thereby avoiding network overfitting and effectively improving the network's generalization ability.

[0014] Optionally, in one embodiment of this application, determining the design information of multiple sets of convolutional kernel groups based on the configuration of multiple sensors includes: matching the proportion of convolutional kernels in each layer based on the proportion of channels in the original data and / or feature maps.

[0015] Based on the above technical means, the embodiments of this application match the proportion of each convolutional kernel according to the proportion of the number of channels in the original data or feature map, thereby strengthening the feature extraction of the corresponding sensing data channels and effectively ensuring the performance of feature extraction.

[0016] Optionally, in one embodiment of this application, the step of training the network based on the design information of the multiple sets of convolutional kernel groups and a preset convolutional group selection mechanism to obtain a pre-trained multimodal deep convolutional network includes: obtaining the weights of each set of convolutional kernel groups according to the original data and / or the feature map channel ratio; and performing convolutional feature extraction training based on the weights of each set of convolutional kernel groups.

[0017] Based on the above technical means, the embodiments of this application obtain the weight of each convolution kernel group according to the proportion of the original data or feature map channels in order to perform convolution feature extraction, thereby increasing the weight of the convolution group and the corresponding sensor data channel and improving the ability of the convolution group to extract information.

[0018] A second aspect of this application provides a vehicle sensor data processing apparatus, comprising: an acquisition module for acquiring raw sensor data collected by multiple on-board sensors of the current vehicle; a preprocessing module for unifying the format of the acquired raw sensor data into a multi-channel two-dimensional representation and forming a multi-modal raw data layer, and obtaining sensor data based on the raw data layer; and a processing module for inputting the sensor data into a pre-trained multimodal deep convolutional network, performing convolution processing to obtain at least one fusion feature, and identifying perceptual information of at least one target based on the at least one fusion feature, wherein the multiple sets of convolutional kernels of the pre-trained multimodal deep convolutional network are obtained from the data of the multiple on-board sensors.

[0019] Optionally, in one embodiment of this application, the preprocessing module is specifically used to link the various raw sensor data using a preset high-dimensional array, or to process the various raw sensor data based on a preset preprocessing network.

[0020] Optionally, in one embodiment of this application, the acquisition module includes: a calibration unit, used to perform spatiotemporal calibration on the multiple vehicle sensors before acquiring the raw sensing data collected by the multiple vehicle sensors of the current vehicle.

[0021] A second aspect of this application provides a vehicle sensor data processing device, further comprising: a design module for determining design information of multiple sets of convolutional kernel groups based on the configuration of multiple sensors, so as to generate corresponding fusion features when extracting multiple sensor combinations; and a training module for training a network based on the design information of the multiple sets of convolutional kernel groups and a preset convolutional group selection mechanism to obtain a pre-trained multimodal deep convolutional network, so as to input sensor data into the multimodal deep convolutional network to obtain at least one fusion feature.

[0022] Optionally, in one embodiment of this application, the design module includes: a matching unit, used to match the proportion of convolutional kernels per layer according to the proportion of the original data and / or the proportion of the number of feature map channels.

[0023] Optionally, in one embodiment of this application, the training module includes: an acquisition unit, configured to acquire the weights of each group of convolutional kernels based on the original data and / or the feature map channel ratio; and a training unit, configured to perform convolutional feature extraction training based on the weights of each group of convolutional kernels.

[0024] A third aspect of this application provides a vehicle, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the vehicle's sensor data processing method as described in the above embodiments.

[0025] A fourth aspect of this application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the vehicle sensor data processing method described above.

[0026] Therefore, the embodiments of this application have the following beneficial effects:

[0027] (1) In this embodiment, the collected raw data is combined into a multi-mode raw data layer, and features are extracted from the raw data layer according to the variable convolution kernel to generate recognition results. Thus, a unified recognition network can adaptively extract features of raw data under different sensor configurations for recognition and detection, which greatly reduces network maintenance costs and improves the efficiency and performance of vehicle sensor data analysis and processing.

[0028] (2) The embodiments of this application can link the original sensing data through a high-dimensional array, or process the original sensing data based on a corresponding preprocessing network, so as to achieve a unified representation of sensor data in different formats, thereby providing reliable fusion data for subsequent feature extraction.

[0029] (3) In this embodiment of the application, the spatiotemporal relationship of different sensors is determined by performing spatiotemporal calibration on multiple vehicle sensors, and information on the same obstacle collected by different sensors can be obtained, which facilitates subsequent fusion processing.

[0030] (4) The embodiments of this application determine the design information of multiple sets of convolution kernel groups based on data from multiple sensors and train the multimodal deep convolutional network, thereby avoiding network overfitting and effectively improving the generalization ability of the network.

[0031] (5) In this embodiment, the proportion of each convolutional kernel is matched according to the proportion of the number of channels in the original data or feature map, thereby strengthening the feature extraction of the corresponding sensing data channels and effectively ensuring the performance of feature extraction.

[0032] (6) In this embodiment of the application, the weight of each convolution kernel group is obtained according to the proportion of the original data or feature map channels to perform convolution feature extraction, thereby increasing the weight of the convolution group and the corresponding sensor data channel and improving the ability of the convolution group to extract information.

[0033] Additional aspects and advantages of this application will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of this application. Attached Figure Description

[0034] The above and / or additional aspects and advantages of this application will become apparent and readily understood from the following description of the embodiments taken in conjunction with the accompanying drawings, wherein:

[0035] Figure 1 This is a schematic diagram illustrating the configuration of different sensors in related technologies;

[0036] Figure 2 This is a schematic diagram of a multi-sensor identification network in related technologies;

[0037] Figure 3 This is a schematic diagram illustrating the impact of ISP resolution on imaging.

[0038] Figure 4 This is a flowchart illustrating the online identification process of a vehicle sensor data processing method according to an embodiment of this application.

[0039] Figure 5 This is a schematic diagram of a multimodal convolutional morphology according to an embodiment of this application;

[0040] Figure 6 This is a schematic diagram illustrating a feature-based fitting method according to an embodiment of this application;

[0041] Figure 7 This is a schematic diagram of a sparse convolution kernel according to an embodiment of this application;

[0042] Figure 8 This is a schematic diagram of an input raw sensor data layer according to an embodiment of this application;

[0043] Figure 9 This is a flowchart illustrating the offline training process of a vehicle sensor data processing method according to an embodiment of this application.

[0044] Figure 10 This is a schematic diagram illustrating the execution logic of a vehicle sensor data processing method according to an embodiment of this application;

[0045] Figure 11 This is an example diagram of a vehicle sensor data processing device according to an embodiment of this application;

[0046] Figure 12 This is a schematic diagram of the vehicle structure provided in an embodiment of this application.

[0047] Among them, 10-vehicle sensor data processing device, 100-acquisition module, 200-preprocessing module, 300-processing module, 400-design module, 500-training module, 1201-memory, 1202-processor, 1203-communication interface. Detailed Implementation

[0048] The embodiments of this application are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain this application, and should not be construed as limiting this application.

[0049] The following description, with reference to the accompanying drawings, describes a vehicle sensor data processing method, apparatus, vehicle, and storage medium according to embodiments of this application. Addressing the problems mentioned in the background section, this application provides a vehicle sensor data processing method. In this method, raw sensor data collected from multiple onboard sensors of the current vehicle is acquired; the format of the acquired raw sensor data is unified into a multi-channel two-dimensional representation and formed into a multi-modal raw data layer; sensor data is obtained based on the raw data layer; the sensor data is input into a pre-trained multi-modal deep convolutional network, and convolution processing is performed to obtain at least one fusion feature, so as to identify the perception information of at least one target based on at least one fusion feature. The convolution kernel group of the pre-trained multi-modal deep convolutional network is obtained from the raw data layer. This application reduces network maintenance costs, avoids network overfitting, and improves the network's generalization ability by forming the acquired raw data into a multi-modal raw data layer and extracting features from the raw data layer according to variable convolution kernels to generate recognition results. This solves the problems that related technologies can only fuse specific sensors, and the network models for different sensor configurations are also different, resulting in a large maintenance workload and the inability to process different sensor configurations through a unified network model.

[0050] Specifically, Figure 4 This is a flowchart of a vehicle sensor data processing method provided in an embodiment of this application. The process mainly involves the online identification process of the vehicle sensor data processing method.

[0051] like Figure 4 As shown, the sensor data processing method for this vehicle includes the following steps:

[0052] In step S401, raw sensing data collected by multiple on-board sensors of the current vehicle are acquired.

[0053] It is understood that current vehicles are equipped with a variety of sensors, such as cameras, lidar, and millimeter-wave radar. The embodiments of this application can integrate multiple sensors to obtain real-time information from multiple angles of the vehicle and fuse the acquired data information to improve the overall recognition performance of the vehicle sensors and assist the vehicle in driving safely and stably.

[0054] Optionally, in one embodiment of this application, before acquiring the raw sensing data collected by multiple on-board sensors of the current vehicle, the method further includes: performing spatiotemporal calibration on the multiple on-board sensors.

[0055] It should be noted that different sensors in a vehicle require temporal calibration due to their asynchronous timing and different sampling frequencies. Simultaneously, multiple sensors require spatial calibration due to their different installation positions and rotation angles. Therefore, the embodiments of this application first perform spatiotemporal calibration of the vehicle-mounted sensors in the following manner, the specific calibration process of which is described below.

[0056] 1. Time calibration

[0057] The embodiments of this application can provide the same reference time to each sensor through a single clock source, and each sensor calibrates its own clock time according to the provided reference time, thereby achieving time synchronization in hardware.

[0058] As one possible approach, technicians can use GPS time as a reference time and employ the PTP / gPTP clock synchronization protocol to achieve time synchronization between various sensors.

[0059] 2. Spatial calibration

[0060] 1) Intrinsic parameter calibration and extrinsic parameter calibration: Intrinsic parameters determine the mapping relationship inside the sensor, such as the focal length, eccentricity and pixel aspect ratio of the camera, while extrinsic parameters determine the transformation relationship between the sensor and an external coordinate system, such as attitude parameters (rotation and translation degrees of freedom).

[0061] 2) Coordinate system synchronization: Convert the measured values ​​from different sensor coordinate systems to the same coordinate system.

[0062] It should be noted that those skilled in the art can also perform spatiotemporal calibration of vehicle sensors in other ways depending on the actual situation, and no specific restrictions are imposed here.

[0063] Therefore, the embodiments of this application determine the spatiotemporal relationship of different sensors by performing spatiotemporal calibration on multiple vehicle-mounted sensors, and can obtain information on the same obstacle collected by different sensors, which facilitates subsequent fusion processing.

[0064] In step S402, the format of the acquired raw sensor data is unified into a multi-channel two-dimensional representation and formed into a multi-mode raw data layer. Sensor data is obtained based on the raw data layer.

[0065] After spatiotemporally calibrating vehicle sensors and acquiring raw sensing data from multiple onboard sensors of the current vehicle, the data types acquired and processed by each sensor are different. For example, camera images consist of RGB three-channel images; LiDAR consists of 3D point cloud information and the intensity of each point; 4D millimeter-wave radar consists of point cloud information and information such as the velocity of each point. The embodiments of this application can preprocess the raw sensing data acquired from multiple onboard sensors, unifying the format of the acquired raw sensing data into a multi-channel two-dimensional representation and forming a multi-mode raw data layer. Sensing data is obtained based on this raw data layer, thus obtaining unified sensing data. This achieves the fusion of multiple different types of sensor data, providing a large amount of reliable data for subsequent recognition network models.

[0066] Optionally, in one embodiment of this application, obtaining sensor data based on the original data layer includes: linking the original sensor data using a preset high-dimensional array; or, processing the original sensor data based on a preset preprocessing network.

[0067] It should be noted that the embodiments of this application can use the following two methods to unify the raw data format of various sensors into a multi-channel two-dimensional representation for unified data representation.

[0068] 1. Multi-source sensor data fusion based on raw data. For example... Figure 5 As shown, a high-dimensional array is used to link the data from various sensors: the first three channels are the three color gamut channels of the RGB image, the third to sixth channels are the three-dimensional spatial coordinates and echo intensity of the LiDAR, and the seventh to tenth channels are the echo point coordinates and frequency information channels of the millimeter-wave radar.

[0069] 2. Multi-source sensor data fusion based on preliminary features. For example... Figure 6 As shown, the camera data is processed into feature maps using appropriate preprocessing; the LiDAR data is processed into voxel data using a preprocessing network; and the millimeter-wave radar data is processed into sparse point features using a preprocessing network.

[0070] It should be noted that the embodiments of this application may use, but are not limited to, the two methods described above for unified data representation. Those skilled in the art may use other methods to achieve unified data format based on actual circumstances, and no specific restrictions are imposed here.

[0071] Therefore, embodiments of this application can link the original sensing data through a high-dimensional array, or process the original sensing data based on a corresponding preprocessing network, to achieve a unified representation of sensor data in different formats, thereby providing reliable fusion data for subsequent feature extraction.

[0072] In step S403, the sensing data is input into a pre-trained multimodal deep convolutional network, and the convolutional processing yields at least one fusion feature to identify the perception information of at least one target based on the at least one fusion feature. The multiple sets of convolutional kernels of the pre-trained multimodal deep convolutional network are obtained from data from multiple vehicle-mounted sensors.

[0073] After assembling the collected raw data into a multi-modal raw data layer, i.e., obtaining multi-channel multi-modal raw data or feature maps with uniform resolution, embodiments of this application can further design variable convolution kernels based on the raw data layer, such as... Figure 7 As shown.

[0074] It should be noted that the embodiments of this application can activate corresponding pre-trained convolutional kernels based on different combinations of input sensor data, such as the original data layer obtained from a combination of vehicle cameras and LiDAR, or the original data layer obtained from a combination of vehicle cameras and millimeter-wave radar, thereby extracting features from the original data layer through grouped convolution. Figure 8 As shown, a feature layer containing both single-sensor data convolutional feature layers and feature layers containing different sensor combinations is generated. This allows for deep convolution based on different sensor inputs, ultimately extracting perceptual features and generating extraction and recognition results.

[0075] It is understood that the embodiments of this application extract features from the original data layer through variable convolution kernels and perform deep convolution to extract and identify perceptual features. This allows vehicles with different sensor configurations to complete deep network identification of multi-source sensor data through a unified identification network. By maintaining this unified identification network, it can serve different sensor configuration vehicle models, greatly reducing network maintenance costs.

[0076] Secondly Figure 9 This is a flowchart of a vehicle sensor data processing method provided in an embodiment of this application. The process mainly involves the offline training process of the vehicle sensor data processing method.

[0077] like Figure 9 As shown, the sensor data processing method for this vehicle includes the following steps:

[0078] In step S901, design information for multiple sets of convolutional kernels is determined based on data from multiple sensors to generate corresponding fusion features when extracting combinations of multiple sensors.

[0079] The embodiments of this application perform spatiotemporal calibration on the vehicle-mounted sensors and unify the multi-mode sensor data format of the collected sensor data to combine the raw data layers of various sensors into a unified input data with a channel number of C. Figure 5As shown. Furthermore, embodiments of this application can design different convolution kernels to convolve and calculate different combined features of the input layer. The convolution kernels can be designed with different depths (sparse designs) to extract convolution features of each sensor data, convolution features fused from each sensor, etc., thereby obtaining a feature layer after one convolution, which contains feature maps of a single sensor as well as feature maps fused from different sensors.

[0080] Therefore, the embodiments of this application determine the design information of multiple sets of convolutional kernel groups based on data from multiple sensors, generate corresponding fusion features, thereby improving the adaptive ability of the network and ensuring the training performance of the recognition network model.

[0081] Optionally, in one embodiment of this application, design information for multiple sets of convolutional kernel groups is determined based on the configuration of multiple sensors, including: matching the proportion of convolutional kernels in each layer according to the proportion of channels in the original data and / or feature maps.

[0082] After obtaining multi-channel, multi-mode raw data or feature maps with uniform resolution, embodiments of this application can perform feature extraction through grouped convolution.

[0083] It should be noted that, in the embodiments of this application, the proportion of each convolutional kernel can be set according to the proportion of the original data or feature map channels, as shown in the following formula, so as to ensure that the corresponding proportion of convolutional kernels can enhance the feature extraction of the corresponding sensing data channels.

[0084]

[0085]

[0086]

[0087]

[0088] Where r is the random activation value, used to randomly determine the convolution kernel for computation; α m The weights of the corresponding convolutional kernel groups are m, where 1, 2, 3... represent cameras, LiDAR, millimeter-wave radar, etc.

[0089] Therefore, the embodiments of this application match the proportion of each convolutional kernel according to the proportion of the number of channels in the original data or feature map, thereby enhancing the feature extraction of the corresponding sensing data channels and effectively ensuring the performance of feature extraction.

[0090] In step S902, the network is trained based on the design information of multiple sets of convolutional kernels and the preset convolutional group selection mechanism to obtain a pre-trained multimodal deep convolutional network. The sensor data is then input into the multimodal deep convolutional network to obtain at least one fused feature.

[0091] After obtaining multiple sets of convolutional kernel design information, such as the ratio of the number of channels in the original data or feature map matching the ratio of the number of convolutional kernels in each layer, the embodiments of this application can use a convolutional group selection mechanism to adapt to the corresponding convolutional calculations when different sensor data are input, and perform network training, thereby obtaining a multimodal deep convolutional network.

[0092] Therefore, embodiments of this application can input a large amount of sensor data with different combinations into a multimodal deep convolutional network to train the convolutional kernel and other related parameters, thereby ensuring the network's recognition and detection performance.

[0093] Optionally, in one embodiment of this application, network training is performed based on the design information of multiple sets of convolutional kernel groups and a preset convolutional group selection mechanism to obtain a pre-trained multimodal deep convolutional network, including: obtaining the weights of each set of convolutional kernel groups according to the original data and / or the feature map channel ratio; and performing convolutional feature extraction training based on the weights of each set of convolutional kernel groups.

[0094] It should be noted that after setting the proportion of each convolution kernel, the embodiments of this application can assign different weights to different groups of convolution kernels and different original sensor data channels or feature map channels. Thus, according to the convolution group selection mechanism, the convolution kernel can dynamically and adaptively perform corresponding convolution calculations with different multi-channel multi-mode original data or feature maps.

[0095] It is understood that the embodiments of this application obtain the weights of each convolution kernel group based on the proportion of the original data or feature map channels in order to perform convolution feature extraction, thereby increasing the weights of the convolution group and the corresponding sensor data channels and improving the ability of the convolution group to extract information.

[0096] In the embodiments of this application, when training the recognition network, convolutional kernel parameters are learned for each of the full sensor configurations. During training, the embodiments of this application not only randomly clip the features of the input feature map, mask the weights, and perform dropout training to train adaptive convolutional kernels, thereby enhancing network generalization and reducing overfitting, but also enhance the corresponding sensor data response based on the different sensors.

[0097] It should be noted that when training the network with missing sensors, the embodiments of this application can perform zero-padding on the corresponding data channels. Therefore, during training, when corresponding sensor data is available, the corresponding convolutional kernel parameters are enhanced; if the corresponding sensor is absent, the corresponding convolutional kernel parameters are zero, but the object features in that area will be enhanced by other sensors to ensure the final response of the target.

[0098] It is understood that the embodiments of this application use a sparse training process to regularize the network, effectively avoiding overfitting and improving the generalization performance of the recognition network.

[0099] The vehicle sensor data processing method of this application will be described below with reference to the accompanying drawings and through a specific embodiment.

[0100] Figure 10 This is a schematic diagram illustrating the execution logic of a vehicle's sensor data processing method. For example... Figure 9 As shown, the main steps for processing vehicle sensor data in the embodiments of this application are as follows:

[0101] S1001: Perform spatiotemporal calibration on vehicle-related sensor equipment;

[0102] S1002: Collects raw data information acquired by various sensors, such as image information from cameras, 3D point cloud information from lidar, and intensity information of each point, as well as point cloud information from millimeter-wave radar and velocity information of each point.

[0103] S1003: Unify the raw data format of various sensors to a multi-channel two-dimensional representation to obtain multi-channel multi-mode raw data or feature maps with uniform resolution.

[0104] S1004: Train the recognition network by learning the convolutional kernel parameters for each sensor configuration. During training, not only are the features of the input feature map randomly clipped for dropout training, but the corresponding sensor data response is also enhanced according to the different sensors. When training the network with missing sensors, zero-filling can be performed on the corresponding data channels to train adaptive convolutional kernels.

[0105] S1005: Execute steps S1002-S1003, and after converting the real-time collected data into multi-channel multi-mode raw data or feature maps of uniform resolution, input them into the recognition network trained in step S1004 to activate each convolution kernel, perform corresponding convolution operations, and generate the final recognition result.

[0106] The vehicle sensor data processing method proposed in this application involves acquiring raw sensor data from multiple onboard sensors of the current vehicle; unifying the format of the acquired raw sensor data into a multi-channel two-dimensional representation and forming a multi-modal raw data layer; obtaining sensor data based on the raw data layer; inputting the sensor data into a pre-trained multi-modal deep convolutional network; performing convolution processing to obtain at least one fusion feature; and identifying the perceptual information of at least one target based on the at least one fusion feature. The convolution kernel group of the pre-trained multi-modal deep convolutional network is obtained from the raw data layer. This application reduces network maintenance costs, avoids network overfitting, and improves the network's generalization ability by forming the acquired raw data into a multi-modal raw data layer and extracting features from the raw data layer according to variable convolution kernels.

[0107] Next, with reference to the accompanying drawings, a vehicle sensor data processing device according to an embodiment of this application is described.

[0108] Figure 11 This is a block diagram of a vehicle sensor data processing device according to an embodiment of this application.

[0109] like Figure 11 As shown, the vehicle's sensor data processing device 10 includes: an acquisition module 100, a preprocessing module 200, and a processing module 300.

[0110] The acquisition module 100 is used to acquire raw sensing data collected by multiple on-board sensors of the current vehicle.

[0111] The preprocessing module 200 is used to unify the format of the acquired raw sensor data into a multi-channel two-dimensional representation and form a multi-mode raw data layer, based on which sensor data is obtained.

[0112] The processing module 300 is used to input sensing data into a pre-trained multimodal deep convolutional network, and the convolutional processing obtains at least one fusion feature to identify the perception information of at least one target based on the at least one fusion feature. The pre-trained multimodal deep convolutional network has multiple sets of convolutional kernels obtained from data from multiple vehicle sensors.

[0113] Optionally, in one embodiment of this application, the preprocessing module 200 is specifically used to link each original sensor data using a preset high-dimensional array, or to process each original sensor data based on a preset preprocessing network.

[0114] Optionally, in one embodiment of this application, the acquisition module 100 includes: a calibration unit, used to perform spatiotemporal calibration on the multiple vehicle sensors before acquiring the raw sensing data collected by the multiple vehicle sensors of the current vehicle.

[0115] The vehicle's sensor data processing device 10 also includes a design module 400 and a training module 500.

[0116] The design module 400 is used to determine the design information of multiple sets of convolutional kernel groups based on the configuration of multiple sensors, so as to generate corresponding fusion features when extracting multiple sensor combinations.

[0117] The training module 500 is used to train the network based on the design information of multiple sets of convolutional kernel groups and a preset convolutional group selection mechanism to obtain a pre-trained multimodal deep convolutional network. Sensing data is then input into the multimodal deep convolutional network to obtain at least one fused feature.

[0118] Optionally, in one embodiment of this application, the design module 400 includes: a matching unit, used to match the proportion of convolutional kernels per layer according to the proportion of the original data and / or the proportion of the number of feature map channels.

[0119] Optionally, in one embodiment of this application, the training module 500 includes: an acquisition unit and a training unit.

[0120] The acquisition unit is used to acquire the weights of each group of convolutional kernels based on the original data and / or the channel ratio of the feature map.

[0121] The training unit is used to train convolutional feature extraction based on the weights of each set of convolutional kernels.

[0122] It should be noted that the foregoing explanation of the vehicle sensor data processing method embodiment also applies to the vehicle sensor data processing device of this embodiment, and will not be repeated here.

[0123] The vehicle sensing data processing device proposed in this application acquires raw sensing data collected by multiple on-board sensors of the current vehicle; unifies the format of the acquired raw sensing data into a multi-channel two-dimensional representation and forms a multi-modal raw data layer, and obtains sensing data based on the raw data layer; inputs the sensing data into a pre-trained multi-modal deep convolutional network, and performs convolution processing to obtain at least one fusion feature, so as to identify the perception information of at least one target based on at least one fusion feature, wherein the convolution kernel group of the pre-trained multi-modal deep convolutional network is obtained from the raw data layer. This application reduces network maintenance costs, avoids network overfitting, and improves the network's generalization ability by forming the acquired raw data into a multi-modal raw data layer and extracting features from the raw data layer according to variable convolution kernels.

[0124] Figure 12 A schematic diagram of the structure of a vehicle provided in an embodiment of this application. The vehicle may include:

[0125] The memory 1201, the processor 1202, and the computer program stored on the memory 1201 and executable on the processor 1202.

[0126] When the processor 1202 executes the program, it implements the vehicle sensor data processing method provided in the above embodiments.

[0127] Furthermore, the vehicle also includes:

[0128] Communication interface 1203 is used for communication between memory 1201 and processor 1202.

[0129] The memory 1201 is used to store computer programs that can run on the processor 1202.

[0130] The memory 1201 may include high-speed RAM memory, and may also include non-volatile memory, such as at least one disk storage.

[0131] If the memory 1201, processor 1202, and communication interface 1203 are implemented independently, then the communication interface 1203, memory 1201, and processor 1202 can be interconnected via a bus to complete communication between them. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be divided into address buses, data buses, control buses, etc. For ease of representation, Figure 12 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0132] Optionally, in a specific implementation, if the memory 1201, processor 1202, and communication interface 1203 are integrated on a single chip, then the memory 1201, processor 1202, and communication interface 1203 can communicate with each other through an internal interface.

[0133] The processor 1202 may be a central processing unit (CPU), an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of this application.

[0134] This embodiment also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described vehicle sensor data processing method.

[0135] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0136] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this application, "N" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0137] Any process or method described in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or N executable instructions for implementing custom logic functions or processes, and the scope of the preferred embodiments of this application includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functions involved, as should be understood by those skilled in the art to which embodiments of this application pertain.

[0138] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-included system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of computer-readable media include: an electrical connection having one or more wires (electronic device), a portable computer disk drive (magnetic device), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Alternatively, the computer-readable medium may be paper or other suitable media on which the program can be printed, since the program can be obtained electronically by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in a computer memory.

[0139] It should be understood that the various parts of this application can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, the N steps or methods can be implemented using software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0140] Those skilled in the art will understand that all or part of the steps of the methods in the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, the program includes one or a combination of the steps of the method embodiments.

[0141] Furthermore, the functional units in the various embodiments of this application can be integrated into a processing module, or each unit can exist physically separately, or two or more units can be integrated into a module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium.

[0142] The storage medium mentioned above can be a read-only memory, a disk, or an optical disk, etc. Although embodiments of this application have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting this application. Those skilled in the art can make changes, modifications, substitutions, and variations to the above embodiments within the scope of this application.

Claims

1. A method for processing sensor data of a vehicle, characterized in that, Includes the following steps: Acquire raw sensor data collected by multiple onboard sensors of the current vehicle; The collected raw sensor data format is unified into a multi-channel two-dimensional representation and formed into a multi-mode raw data layer. Sensor data is obtained based on the raw data layer. as well as The sensing data is input into a pre-trained multimodal deep convolutional network, and the convolutional processing yields at least one fusion feature to identify the perception information of at least one target based on the at least one fusion feature. The multiple sets of convolutional kernels of the pre-trained multimodal deep convolutional network are obtained from the data of the multiple vehicle sensors. The raw sensor data includes point clouds and RGB images.

2. The method according to claim 1, characterized in that, The sensing data obtained based on the original data layer includes: The original sensor data is linked using a pre-defined high-dimensional array; Alternatively, the raw sensor data can be processed based on a preset preprocessing network.

3. The method according to claim 1, characterized in that, Before acquiring the raw sensing data collected by multiple onboard sensors of the current vehicle, the method further includes: Spatiotemporal calibration is performed on the multiple vehicle-mounted sensors.

4. A method for processing sensor data of a vehicle, characterized in that, Includes the following steps: The design information of multiple sets of convolutional kernels is determined based on data from multiple sensors to adapt to the generation of corresponding fusion features when extracting combinations of multiple sensors; Based on the design information of the multiple sets of convolutional kernels and the preset convolutional group selection mechanism, the network is trained to obtain a pre-trained multimodal deep convolutional network. Sensing data is then input into the multimodal deep convolutional network to obtain at least one fusion feature. The data from the various sensors include point clouds and RGB images.

5. The method according to claim 4, characterized in that, The process of determining the design information for multiple sets of convolutional kernels based on the configuration of various sensors includes: Match the proportion of convolutional kernels for each layer based on the proportion of channels in the original data and / or feature maps.

6. The method according to claim 5, characterized in that, The process of training the network based on the design information of the multiple sets of convolutional kernels and a preset convolutional group selection mechanism to obtain a pre-trained multimodal deep convolutional network includes: The weights of each group of convolutional kernels are obtained based on the original data and / or the feature map channel ratio. Convolutional feature extraction training is performed based on the weights of each set of convolutional kernels.

7. A vehicle sensor data processing device, characterized in that, include: The acquisition module is used to acquire raw sensor data collected by multiple on-board sensors of the current vehicle; The preprocessing module is used to unify the format of the acquired raw sensor data into a multi-channel two-dimensional representation and form a multi-mode raw data layer, and obtain sensor data based on the raw data layer. as well as The processing module is used to input the sensing data into a pre-trained multimodal deep convolutional network, and to obtain at least one fusion feature through convolution processing, so as to identify the perception information of at least one target based on the at least one fusion feature, wherein the multiple sets of convolutional kernels of the pre-trained multimodal deep convolutional network are obtained from the data of the multiple vehicle-mounted sensors; The raw sensor data includes point clouds and RGB images.

8. The apparatus according to claim 7, characterized in that, The preprocessing module is specifically used to link the various raw sensor data using a preset high-dimensional array, or to process the various raw sensor data based on a preset preprocessing network.

9. The apparatus according to claim 8, characterized in that, The acquisition module includes: The calibration unit is used to perform spatiotemporal calibration on the multiple on-board sensors before acquiring the raw sensing data collected by the multiple on-board sensors of the current vehicle.

10. A vehicle sensor data processing device, characterized in that, include: The design module is used to determine the design information of multiple sets of convolutional kernel groups based on the configuration of multiple sensors, so as to generate corresponding fusion features when extracting multiple sensor combinations; The training module is used to train the network based on the design information of the multiple sets of convolutional kernels and the preset convolutional group selection mechanism to obtain a pre-trained multimodal deep convolutional network, so as to input the sensor data into the multimodal deep convolutional network to obtain at least one fusion feature. The sensing data includes point clouds and RGB images.

11. The apparatus according to claim 10, characterized in that, The design module includes: The matching unit is used to match the proportion of convolutional kernels in each layer based on the proportion of channels in the original data and / or feature maps.

12. The apparatus according to claim 11, characterized in that, The training module includes: The acquisition unit is used to acquire the weights of each group of convolutional kernels based on the original data and / or the feature map channel ratio; The training unit is used to perform convolutional feature extraction training based on the weights of each group of convolutional kernels.

13. A vehicle, characterized in that, include: A memory, a processor, and a computer program stored in the memory and executable on the processor, the processor executing the program to implement the vehicle sensor data processing method as described in any one of claims 1-3 or 4-6.

14. A computer-readable storage medium having a computer program stored thereon, characterized in that, The program is executed by the processor to implement the vehicle sensor data processing method as described in any one of claims 1-3 or 4-6.

Images