Sensor data processing method and device, electronic equipment and storage medium

Abstract

The invention provides a sensor data processing method and device, electronic equipment and a storage medium, and relates to the technical field of data processing. According to the specific implementation scheme, n to-be-processed data sets of a target vehicle are acquired; determining m candidate data sets in the n to-be-processed data sets according to a type screening condition; and determining at least one target data set in the m candidate data sets based on the spatial density coefficient. According to the technical scheme, secondary cleaning of multiple types of sensor data can be realized based on the data type composition of the acquired sensor data and the spatial density distribution of the data acquisition position, so that the problem of large redundant data volume caused by sensor data cleaning only based on a single-type dimension is avoided; and the usability of the processed sensor data in an automatic driving simulation test scene is improved.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a sensor data processing method, device, electronic device, and storage medium. Background Art

[0002] Autonomous driving simulation testing is of great significance for the rapid update and iteration of autonomous driving technology. Autonomous driving simulation testing can use data from different types of sensors in real vehicles, such as lidar, cameras, millimeter-wave radar, to build simulation scenarios, thereby simulating various road conditions and traffic scenarios to verify and optimize autonomous driving algorithms.

[0003] Currently, most of the collection and processing of real-vehicle sensor data only involves data cleaning and screening from the perspective of a single sensor, resulting in difficulties in aligning the processed sensor data of various types. Many processed data are still difficult to effectively use for scene reconstruction in autonomous driving simulation tests, which not only takes up storage space but also affects the efficiency of autonomous driving simulation tests. Summary of the Invention

[0004] Embodiments of the present application provide a sensor data processing method, device, electronic device, and storage medium to improve the usability of processed vehicle sensor data in an autonomous driving simulation test scenario.

[0005] In a first aspect, an embodiment of the present application provides a sensor data processing method, comprising: obtaining n to-be-processed data sets of a target vehicle, wherein each to-be-processed data set includes at least one sensor data and a collection time and a collection location corresponding to the to-be-processed data set, and n is an integer greater than 1;

[0006] Determine m candidate data sets from n data sets to be processed according to a type screening condition, wherein the type screening condition includes at least one specified data type, m is an integer and 1≤m≤n;

[0007] At least one target data set is determined from the m candidate data sets based on a spatial density coefficient, wherein the spatial density coefficient is a maximum value of the number of target data sets in a preset unit area.

[0008] In one possible implementation, at least one target data set is determined from m candidate data sets based on a spatial density coefficient, including: determining at least one screening area based on the collection positions corresponding to the m candidate data sets, wherein any screening area is associated with at least one candidate data set and the screening area contains the collection positions corresponding to all candidate data sets associated with the screening area; for any screening area, determining the maximum number of target data sets corresponding to the screening area based on the spatial density coefficient; and determining at least one target data set from the candidate data sets associated with the screening area based on the maximum number.

[0009] In one possible implementation, at least one target data set is determined from the candidate data sets associated with the screening area according to the maximum number, including: in response to the number of candidate data sets associated with the screening area being not greater than the maximum number, each candidate data set associated with the screening area is determined as a target data set.

[0010] In one possible implementation, at least one target data set is determined from the candidate data sets associated with the screening area based on the maximum number of target data sets corresponding to the screening area, including: in response to the number of candidate data sets associated with the screening area being greater than the maximum number, screening out the maximum number of target data sets from the candidate data sets associated with the screening area based on the collection time corresponding to each candidate data set, wherein the time difference between the collection time corresponding to any target data set in the screened target data sets and the collection time corresponding to the other target data sets is greater than a preset duration.

[0011] In one possible implementation, m candidate data sets are determined from n data sets to be processed according to a type screening condition, including: for any data set to be processed, determining whether the data type of the sensor data contained in the data set to be processed covers the specified data type in the type screening condition; in response to the data type of the sensor data contained in the data set to be processed covering the specified data type in the type screening condition, generating a candidate data set corresponding to the data set to be processed.

[0012] In one possible implementation, in response to the data type of the sensor data contained in the data set to be processed covering the specified data type in the type screening condition, a candidate data set corresponding to the data set to be processed is generated, including: deleting the sensor data other than the specified data type in the data set to be processed, and generating the candidate data set corresponding to the data set to be processed.

[0013] In one possible implementation, at least one type of sensor data includes at least 2D image data, and n sets of data to be processed of a target vehicle are obtained, including: determining the 2D image data from the original sensor data of the target vehicle; detecting each frame of 2D image data using a category detection model to determine at least one frame of target 2D image data, each frame of target 2D image data containing image data of an object of a specified type; for any frame of target 2D image data, extracting other types of sensor data corresponding to the frame of target 2D image data from the original sensor data, the other types of sensor data having the same acquisition time and acquisition location as the frame of target 2D image data; and constructing the target 2D image data and the corresponding other types of sensor data into a set of data to be processed.

[0014] In a second aspect, an embodiment of the present application provides a sensor data processing device, comprising: a data acquisition module, configured to acquire n to-be-processed data sets of a target vehicle, wherein each to-be-processed data set includes at least one type of sensor data and a collection time and a collection location corresponding to the to-be-processed data set, and n is an integer greater than 1;

[0015] A first screening module is configured to determine m candidate data sets from n to-be-processed data sets according to a type screening condition, wherein the type screening condition includes at least one specified data type, m is an integer, and 1≤m≤n;

[0016] The second screening module is configured to determine at least one target data set from the m candidate data sets based on a spatial density coefficient, wherein the spatial density coefficient is a maximum value of the number of target data sets in a preset unit area.

[0017] In a third aspect, an embodiment of the present application provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory, wherein the processor implements any method of the embodiments of the present application when executing the computer program.

[0018] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, in which a computer program is stored. When the computer program is executed by a processor, the method of any one of the embodiments of the present application is implemented.

[0019] Based on the method of the embodiment of the present application, secondary cleaning of multiple types of sensor data can be achieved based on the data type composition of the acquired sensor data and the spatial density distribution of the data collection location, avoiding the problem of large amounts of redundant data caused by sensor data cleaning based only on a single type dimension, and improving the availability of processed sensor data in autonomous driving simulation test scenarios.

[0020] The above description is only an overview of the technical solution of this application. In order to more clearly understand the technical means of this application, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of this application more obvious and easy to understand, the specific implementation methods of this application are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] The accompanying drawings are provided to facilitate a better understanding of the present invention and do not constitute a limitation of the present application.

[0022] Figure 1a This is a schematic diagram of a data processing flow for 2D image data provided by an exemplary embodiment of the present application;

[0023] Figure 1b This is a schematic diagram of a data processing flow for 3D point cloud data provided by an exemplary embodiment of the present application;

[0024] Figure 1c This is a schematic diagram of a data processing flow for depth map data provided by an exemplary embodiment of the present application;

[0025] Figure 2 1 is a flowchart of a method for processing sensor data provided by an exemplary embodiment of the present application;

[0026] Figure 3 This is a flow chart of a method for processing sensor data provided by an exemplary embodiment of the present application. Figure 2 ;

[0027] Figure 4 This is a flow chart of a method for processing sensor data provided by an exemplary embodiment of the present application. Figure 3 ;

[0028] Figure 5 is a schematic diagram of a sensor data processing device provided by an exemplary embodiment of the present application;

[0029] Figure 6 is a block diagram of an electronic device used to implement the embodiments of the present application. DETAILED DESCRIPTION

[0030] Hereinafter, only certain exemplary embodiments are briefly described. As will be appreciated by those skilled in the art, the described embodiments may be modified in various ways without departing from the spirit or scope of the present application. Therefore, the drawings and description are to be regarded as illustrative in nature and not restrictive.

[0031] To facilitate understanding of the technical solutions of the embodiments of the present application, the following describes the related technologies of the embodiments of the present application. The following related technologies can be combined with the technical solutions of the embodiments of the present application as optional solutions, and all of them fall within the scope of protection of the embodiments of the present application.

[0032] Application Scenario

[0033] In autonomous driving simulation testing, 3D scene reconstruction is the core link in building a virtual test environment. Its purpose is to transform the real road environment into a digital model that can be perceived by the algorithm. Before 3D creation and reconstruction, it is necessary to pre-process the collected real vehicle sensor data, such as 2D image data, 3D point cloud data, and depth map data. Figure 1a The data processing flow for 2D image data provided by an exemplary embodiment of the present application is shown.

[0034] like Figure 1a As shown, for the 2D image data collected by the vehicle-mounted sensor, the processing flow may include:

[0035] 1) Image deduplication

[0036] For example, the ORB (Oriented FAST and Rotated BRIEF) algorithm is used for feature-based image deduplication. The FAST (Features from Accelerated Segment Test) keypoint detector first determines the grayscale difference between a pixel and its surrounding neighborhood pixels. Pixels whose grayscale values ​​differ from the center pixel by a set threshold for at least nine consecutive pixels in the neighborhood are identified as keypoints. Directional information is then assigned to each keypoint using the grayscale centroid method. Multiple pairs of points are randomly selected around the keypoints, and the grayscale differences between the pairs are compared to generate BRIEF (Binary Robust Independent Elementary Features) descriptors. The Hamming distance between the keypoint descriptors in different images is then calculated to accurately determine image similarity, achieving image deduplication.

[0037] 2) Image deblurring

[0038] For example, a fast Fourier transform operation can be performed on the input image to detect the distribution of high-frequency and low-frequency components in the image, and whether the image is blurred is determined based on the amount of high-frequency components. If the high-frequency components are less than a preset threshold, it is determined to be a blurred image; otherwise, it is a clear image.

[0039] 3) Formatting

[0040] Since different vehicles have different specifications of on-board sensors, possible image data formats include but are not limited to mp4, jpg, png, etc. Therefore, after image deduplication and deblurring, the image data can be uniformly converted into jpg format or png format by specifying a unified output format to facilitate subsequent processing and use of the image data.

[0041] Figure 1b and Figure 1c The data processing flow for 3D point cloud data and depth map data provided by an exemplary embodiment of the present application is shown respectively. Figure 1b As shown, for the 3D point cloud data collected by the vehicle-mounted 3D lidar, the processing flow may include:

[0042] 1) Outlier removal

[0043] In adverse weather conditions, such as rain and snow, raindrops or snowflakes can scatter and reflect the laser signal, interfering with it. This can cause the sensor to receive erroneous signals, resulting in outliers. Furthermore, in complex urban environments, multiple reflections from building surfaces can also lead to the generation of abnormal points, which differ significantly from the distribution of the surrounding normal point cloud and appear as outliers.

[0044] The average distance between each point and other points in the neighborhood can be calculated, and then the variance of the average distance of all points can be calculated. Points whose average distance exceeds a preset threshold can be identified as outliers.

[0045] 2) Ground segmentation

[0046] In autonomous driving scenarios, ground points are crucial for vehicle positioning and path planning and are typically retained. However, in some object detection tasks, ground points can interfere with the detection of the target object and therefore need to be removed. For example, when detecting pedestrians or vehicles on the road, ground points increase the data volume and computational complexity, potentially leading to false detections, and therefore need to be segmented from the point cloud.

[0047] The Random Sample Consensus algorithm can be used to randomly select some points in the point cloud, assume that these points form a plane model, and then calculate the distance of other points to the plane. If the number of points with a distance below a certain threshold exceeds a certain proportion, the plane model is considered valid, and the defined ground plane is removed from the 3D point cloud data.

[0048] 3) Dynamic object filtering

[0049] Moving vehicles and pedestrians appear as transient point clouds in the collected point cloud. Because their positions are constantly changing, their point cloud features differ significantly from those of surrounding stationary objects. The point clouds of these dynamic objects can interfere with point cloud-based object detection and tracking algorithms, leading to inaccurate detection results or tracking loss. For example, in traffic scenes, fast-moving vehicles and pedestrians generate a large number of dynamic point clouds, which require effective filtering to improve the algorithm's detection and tracking performance for static objects.

[0050] 4) Voxel Downsampling

[0051] The measurement principle of 3D radar determines that its point cloud density varies at different distances. Nearby objects have a higher point cloud density due to the favorable laser beam angle and reflection intensity. Distant objects, on the other hand, have a lower point cloud density due to laser signal attenuation and scattering. This uneven density poses challenges to subsequent point cloud processing algorithms. For example, during point cloud registration and target recognition, feature extraction and matching vary in difficulty across density regions, potentially leading to performance degradation.

[0052] The point cloud is divided into voxel grids, each of which can be thought of as a small cube. Within each voxel grid, a representative point is retained according to certain rules, such as the centroid of all points within the voxel grid. This reduces the number of points in the point cloud, balances the point cloud density, and reduces the computational effort of subsequent processing while preserving the overall shape and structure of the point cloud.

[0053] like Figure 1c As shown, for the depth map data collected by the vehicle-mounted depth camera, the processing flow may include:

[0054] 1) Invalid value replacement

[0055] When facing transparent objects, such as glass, the depth camera cannot obtain valid depth information because light can pass through it directly. Such areas are usually filled with 0 or NaN. Similarly, for highly reflective objects, such as mirrors, the reflected light signals received by the sensor are chaotic, making it difficult to accurately measure depth, resulting in invalid values ​​in these areas.

[0056] Invalid values ​​in the depth map are replaced by calculating the mean of the nearby valid values. A suitable neighborhood window, such as a 3×3 or 5×5 window, is set around the invalid value point. All valid values ​​within the window are traversed, their mean is calculated, and then the mean is assigned to the invalid value point.

[0057] 2) Bilateral filtering

[0058] Bilateral filtering preserves edge information while smoothing an image. It considers not only the spatial distance between pixels but also the difference between pixel values. For each pixel in the depth map, a weight is calculated based on the spatial distance and depth difference between its neighboring pixels. The new depth value for that pixel is then calculated by weighted averaging. This method requires careful adjustment based on the noise level and edge characteristics of the depth map to achieve optimal smoothing and edge preservation.

[0059] 3) Gaussian filtering

[0060] Data smoothing is achieved by taking a weighted average of each pixel in the depth map and its neighborhood. The weight is determined by a Gaussian function, and the closer the pixel is to the center pixel, the greater the weight is.

[0061] As can be seen from this, for the collected vehicle sensor data, during the data preprocessing process, corresponding data cleaning will be performed for different types of sensor data. During the data cleaning process, some unqualified data will inevitably be screened out. For example, during the image deduplication and image deblurring process of 2D image data, image data with high duplication and blurred image data will be deleted. In addition, the acquisition frequency of different sensors on the same vehicle may also be different. The three-dimensional scene reconstruction in autonomous driving simulation testing requires combining multiple types of sensor data for feature modeling. If a certain vehicle sensor data lacks other types of sensor data collected at the same time and location, it may be difficult to use this data for effective modeling. If these difficult-to-align vehicle sensor data are uploaded to the simulation database, they will need to be further screened and processed before they can be used for simulation testing, which will undoubtedly consume additional manpower and material resources for autonomous driving simulation testing.

[0062] Secondly, even if the sensor collection frequency on the same vehicle is set to be consistent, it is difficult to keep the frequency of data collection by sensors of different vehicle models consistent. In addition, the driving conditions of vehicles on different road conditions are also different. For sensor data collected when stationary or driving at low speed in traffic jams, too much data will be collected in a small area. For the three-dimensional scene reconstruction of autonomous driving simulation tests, it is aimed at using real sensor data in different scenarios to build a virtual environment. In the same area, it may not be necessary to collect too much sensor data with similar positions, resulting in redundant sensor data collected by some vehicles after a single type of data cleaning.

[0063] Based on this, the present application proposes a sensor data processing method, which can perform secondary cleaning of multiple types of sensor data according to the data type set by the user and the spatial density distribution of the data collection location after data cleaning of a single type of sensor data, thereby improving the usability of the processed sensor data in the autonomous driving simulation test scenario.

[0064] It should be noted that the processing flow and processing technology for a single type of sensor data mentioned in this section are intended to better illustrate the necessity of secondary cleaning of multi-type sensor data, and do not limit the processing of a single type of sensor data.

[0065] Exemplary Methods

[0066] Figure 2FIG is a flow chart of a method for processing sensor data provided by an exemplary embodiment of the present application. This embodiment can be applied to electronic devices such as Figure 2 As shown, the method includes the following steps:

[0067] Step S210 , obtaining n to-be-processed data sets of the target vehicle, wherein each to-be-processed data set includes at least one type of sensor data and a collection time and a collection location corresponding to the to-be-processed data set, and n is an integer greater than 1.

[0068] Here, the same set of data to be processed includes sensor data collected by various types of on-board sensors on the target vehicle at the same time and location, and may include one or more of 2D image data, 3D point cloud data, depth map data, and inertial measurement data.

[0069] For example, a high-precision clock module can be used on the target vehicle to provide a unified time reference for all sensors. Each sensor is connected to the clock module through a hardware interface. Each time data is collected, the sensor can embed timestamps and GPS information into the collected data. The collected sensor data is named in the format of "sensor type-timestamp-positioning point", so that various types of sensor data collected at the same time and location can be constructed into a set of data to be processed, which is convenient for subsequent rapid retrieval and call of data based on time points or locations.

[0070] Step S220 : determining m candidate data sets from the n data sets to be processed according to a type screening condition, wherein the type screening condition includes at least one specified data type, m is an integer, and 1≤m≤n.

[0071] It can be understood that the sensor data in the data set to be processed have undergone a data cleaning process based on a single type setting, and invalid data or highly repetitive data have been eliminated according to the different sensor data types. Therefore, the number of types of sensor data contained in different data sets to be processed may be different. For example, the data set to be processed with an acquisition time of 8:00am and an acquisition location of a degree north latitude and b degrees east longitude contains 2D image data, 3D point cloud data and depth map data, but the data set to be processed with an acquisition time of 8:05am and an acquisition location of a degree north latitude and c degrees east longitude may only contain 2D image data and 3D point cloud data.

[0072] Since 3D scene reconstruction in autonomous driving simulation testing requires feature modeling based on multiple types of sensor data, in order to align different types of sensor data in each processed data set used for autonomous driving simulation testing with sensor data in other data sets to ensure data consistency, in step S210, m candidate data sets are determined from the n data sets to be processed based on the type screening condition, including:

[0073] For any data set to be processed, determine whether the data type of the sensor data contained in the data set to be processed covers the specified data type in the type filtering condition; in response to the data type of the sensor data contained in the data set to be processed covering the specified data type in the type filtering condition, generate a candidate data set corresponding to the data set to be processed.

[0074] Here, the specified data type in the type filtering condition can be set by the user according to the data requirements of subsequent autonomous driving simulation tests. For example, the user can set the specified data type to include 2D image data, 3D point cloud data and depth map data at the same time. Only the data sets to be processed that contain these three types of sensor data can be filtered as candidate data sets. If any of these three types of sensor data does not exist in the data set to be processed, the corresponding data set to be processed needs to be removed.

[0075] Specifically, the set of data to be processed that covers the specified data type in the type screening condition may also include sensor data that does not belong to the specified data type. In order to reduce the storage pressure of the simulation database, for the set of data to be processed that meets the type screening condition, the process of generating the candidate data set may further include:

[0076] The sensor data other than the specified data type in the data set to be processed is deleted to generate a candidate data set corresponding to the data set to be processed.

[0077] The method of this embodiment can effectively ensure that the screened sensor data meets the data consistency requirements of the autonomous driving simulation test and significantly reduce the amount of redundant or useless data uploaded to the simulation database.

[0078] Step S230 : determining at least one target data set from the m candidate data sets based on a spatial density coefficient, wherein the spatial density coefficient is a maximum value of the number of target data sets in a preset unit area.

[0079] Here, since the target vehicle may be in a low-speed or stationary driving state during the data collection process, the on-board sensor may still collect data at the set collection frequency in such a driving state. However, the three-dimensional scene reconstruction process of the autonomous driving simulation test does not require high-similarity sensor data in the same area. In order to further improve the value of the data uploaded to the simulation database and reduce storage pressure, the maximum value of the target data set in the unit area can be specified by pre-setting the spatial density coefficient. For example, the spatial density coefficient is set to every 5m 2 A maximum of 5 target data sets are retained in the system, or every 10m 2 A maximum of 20 target data sets are retained, and candidate data sets are further screened.

[0080] like Figure 3 As shown, in step S230, at least one target data set is determined from the m candidate data sets based on the spatial density coefficient, including:

[0081] Step S231 : determining at least one screening area based on the collection positions corresponding to the m candidate data sets, wherein any screening area is associated with at least one candidate data set and the screening area includes the collection positions corresponding to all candidate data sets associated with the screening area.

[0082] It can be understood that since each candidate data set corresponds to a collection time and a collection location, the driving trajectory of the target vehicle within a period of time can be generated through these collection locations and collection times, thereby dividing at least one screening area according to the driving trajectory. The shape of each screening area can be square or circular, etc., and the area can be equal or unequal. The candidate data set whose collection location falls in the screening area is also the candidate data set associated with the screening area.

[0083] Step S232: For any screening area, determine the maximum number of target data sets corresponding to the screening area according to the spatial density coefficient.

[0084] Step S233: determining at least one target data set from the candidate data sets associated with the screening area according to the maximum number.

[0085] Here, for any screening area, the maximum number of target data sets corresponding to the screening area can be calculated based on the area and spatial density coefficient of the screening area. If the number of candidate data sets associated with the screening area is not greater than the maximum number of target data sets corresponding to the screening area, then at least one target data set is determined from the candidate data sets associated with the screening area based on the maximum number in step S233, including:

[0086] In response to the number of candidate data sets associated with the screening area being no greater than a maximum number, each candidate data set associated with the screening area is determined as a target data set.

[0087] If the number of candidate data sets associated with the screening area is greater than the maximum number of target data sets corresponding to the screening area, then at step S233, at least one target data set is determined from the candidate data sets associated with the screening area according to the maximum number, including:

[0088] In response to the number of candidate data sets associated with the screening area being greater than a maximum number, a maximum number of target data sets are screened out from the candidate data sets associated with the screening area based on the collection moments corresponding to the respective candidate data sets, wherein a time difference between the collection moments corresponding to any target data set in the screened target data sets and the collection moments corresponding to the other target data sets is greater than a preset duration.

[0089] Here, since the number of associated candidate data sets in the screening area is greater than the maximum number of target data sets corresponding to the screening area calculated based on the spatial density coefficient, it is necessary to further screen out target data sets in the associated candidate area sets. In order to ensure that the data between each data set in the screened target data sets have a certain degree of distinction, further screening can be performed based on the collection time. For example, the preset time length is set to 10s, so that after further screening, the time difference between the collection time of the target data set screened in a single screening area and other target data sets is greater than 10s, thereby ensuring that there are obvious differences between each target data set.

[0090] In one possible implementation, the target data sets may be screened by setting the distance between the collection positions of the target data sets and other target data sets in the same screening area. For example, the preset distance is set to 10 m, so that the distance between the collection position corresponding to any target data set in the screened target data sets and the collection positions corresponding to other target data sets is greater than 10 m, thereby determining the target data sets in a single screening area.

[0091] By adopting the method of this embodiment, after filtering the collected sensor data based on the type filtering conditions, the sensor data can be secondary filtered based on the spatial density distribution of the data collection location, ensuring that the sensor data in the target data set has a certain degree of distinction compared with the sensor data in other target data sets, and further improving the usability of the processed sensor data in the autonomous driving simulation test scenario.

[0092] In one possible implementation, if certain specific simulation tests need to be performed, there may be specific requirements for the three-dimensional reconstruction scene, such as the scene must include lane lines or pedestrians. In this case, pre-screening can be performed based on the 2D image data in the sensor data during the construction of the data set to be processed.

[0093] like Figure 4 As shown, in step S210, n sets of data to be processed of the target vehicle are obtained, including:

[0094] Step S211 : Determine 2D image data from the original sensor data of the target vehicle.

[0095] In step S212 , each frame of 2D image data is detected using a category detection model to determine at least one frame of target 2D image data, wherein each frame of target 2D image data contains image data of an object of a specified type.

[0096] Step S213 : for any frame target 2D image data, extract other types of sensor data corresponding to the frame target 2D image data from the original sensor data, wherein the other types of sensor data have the same acquisition time and acquisition position as the frame target 2D image data.

[0097] Step S214: construct the target 2D image data and other types of sensor data corresponding thereto into a data set to be processed.

[0098] Here, 2D image data can be first filtered out from the original sensor data of the target vehicle, and the category detection model can be used to detect the 2D image data to filter out target 2D image data containing specified types of objects (such as lane lines, traffic lights, or pedestrians, etc.). The specified types of objects can be one or more.

[0099] Furthermore, for each frame of target 2D image data, other types of sensor data corresponding to it are searched based on its corresponding acquisition time and acquisition position, and the target 2D image data and other types of sensor data with the same acquisition time and acquisition position as the target 2D image data are formed into a data set to be processed.

[0100] By adopting the method of this embodiment, it is possible to perform preliminary screening based on the detection of target objects in 2D image data before performing linkage processing of various types of sensor data based on the sensor data set, so that the constructed data set to be processed at least includes the detection data of the target objects by the on-board sensors, thereby meeting the data requirements of the autonomous driving simulation test for specific simulation scenarios and improving data processing efficiency.

[0101] The above describes the specific settings and implementation methods of the embodiments of the present application from different perspectives. Using the methods provided in the above embodiments, secondary cleaning of multiple types of sensor data can be achieved based on the data type composition of the acquired sensor data and the spatial density distribution of the data acquisition location, avoiding the problem of large amounts of redundant data caused by sensor data cleaning based only on a single type dimension, and improving the availability of processed sensor data in autonomous driving simulation test scenarios.

[0102] Exemplary devices

[0103] As the implementation of the above methods, Figure 5 As shown, an embodiment of the present application further provides a sensor data processing device, which may include:

[0104] The data acquisition module 501 is configured to acquire n to-be-processed data sets of the target vehicle, wherein each to-be-processed data set includes at least one type of sensor data and a collection time and collection location corresponding to the to-be-processed data set, and n is an integer greater than 1;

[0105] A first screening module 502 is configured to determine m candidate data sets from n to-be-processed data sets according to a type screening condition, wherein the type screening condition includes at least one specified data type, m is an integer, and 1≤m≤n;

[0106] The second screening module 503 is configured to determine at least one target data set from the m candidate data sets based on a spatial density coefficient, wherein the spatial density coefficient is the maximum value of the number of target data sets in a preset unit area.

[0107] Exemplarily, the second screening module 503 is used to: determine at least one screening area based on the collection positions corresponding to m candidate data sets, wherein any screening area is associated with at least one candidate data set and the screening area contains the collection positions corresponding to all candidate data sets associated with the screening area; for any screening area, determine the maximum number of target data sets corresponding to the screening area based on the spatial density coefficient; and determine at least one target data set from the candidate data sets associated with the screening area based on the maximum number.

[0108] Exemplarily, the second screening module 503 is further configured to: in response to the number of candidate data sets associated with the screening area being not greater than the maximum number, determine each candidate data set associated with the screening area as a target data set.

[0109] Exemplarily, the second screening module 503 is further used to: in response to the number of candidate data sets associated with the screening area being greater than the maximum number, screen out a maximum number of target data sets from the candidate data sets associated with the screening area based on the collection time corresponding to each candidate data set, wherein the time difference between the collection time corresponding to any target data set in the screened target data sets and the collection time corresponding to other target data sets is greater than a preset duration.

[0110] Exemplarily, the first screening module 502 is used to: for any data set to be processed, determine whether the data type of the sensor data contained in the data set to be processed covers the specified data type in the type screening condition; in response to the data type of the sensor data contained in the data set to be processed covering the specified data type in the type screening condition, generate a candidate data set corresponding to the data set to be processed.

[0111] Exemplarily, the first screening module 502 is further configured to delete sensor data other than a specified data type in the data set to be processed, and generate a candidate data set corresponding to the data set to be processed.

[0112] Exemplarily, at least one type of sensor data includes at least 2D image data, and the data acquisition module 501 is used to: determine the 2D image data in the original sensor data of the target vehicle; use the category detection model to detect each frame of 2D image data to determine at least one frame of target 2D image data, and each frame of target 2D image data contains image data of a specified type of object; for any frame of target 2D image data, extract other types of sensor data corresponding to the frame of target 2D image data from the original sensor data, and the other types of sensor data have the same acquisition time and acquisition position as the frame of target 2D image data; and construct the target 2D image data and the other types of sensor data corresponding to it into a data set to be processed.

[0113] The functions of each unit, module or sub-module in each device in the embodiments of the present application can be referred to the corresponding description in the above method embodiments, and have corresponding beneficial effects, which will not be repeated here.

[0114] Exemplary electronic devices and computer-readable storage media

[0115] Figure 6 FIG. 1 is a block diagram of an electronic device for implementing an embodiment of the present application. Figure 6 As shown, the electronic device includes: a memory 601 and a processor 602. The memory 601 stores a computer program that can be run on the processor 602. When the processor 602 executes the computer program, the method in the above embodiment is implemented. The number of the memory 601 and the processor 602 can be one or more.

[0116] The electronic device also includes:

[0117] The communication interface 603 is used to communicate with external devices and perform data exchange transmission.

[0118] If the memory 601, processor 602, and communication interface 603 are implemented independently, the memory 601, processor 602, and communication interface 603 can be connected to each other via a bus and communicate with each other. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 6 Only one thick line is used in the diagram, but this does not mean that there is only one bus or one type of bus.

[0119] Optionally, in a specific implementation, if the memory 601, the processor 602 and the communication interface 603 are integrated on a chip, the memory 601, the processor 602 and the communication interface 603 can communicate with each other through an internal interface.

[0120] An embodiment of the present application also provides a chip, which includes a processor for calling and executing instructions stored in the memory from the memory, so that a communication device equipped with the chip executes the method provided in the embodiment of the present application.

[0121] An embodiment of the present application also provides a chip, including: an input interface, an output interface, a processor and a memory. The input interface, the output interface, the processor and the memory are connected through an internal connection path. The processor is used to execute the code in the memory. When the code is executed, the processor is used to execute the method provided in the embodiment of the present application.

[0122] It should be understood that the processor may be a CPU, or other general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), an FPGA or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component, etc. The general-purpose processor may be a microprocessor or any conventional processor, etc. It is worth noting that the processor may be a processor that supports the Advanced RISC Machines (ARM) architecture.

[0123] Furthermore, optionally, the above-mentioned memory may include a read-only memory and a random access memory. The memory may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. Among them, the non-volatile memory may include a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may include a random access memory (RAM), which is used as an external cache. By way of example but not limitation, many forms of RAM are available. For example, static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link DRAM (SLDRAM) and direct memory bus random access memory (DR RAM).

[0124] In the above embodiments, all or part of the embodiments may be implemented using software, hardware, firmware, or any combination thereof. When implemented using software, all or part of the embodiments may be implemented in the form of a computer program product. A computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the processes or functions according to the present application are generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transferred from one computer-readable storage medium to another.

[0125] In addition to the above-mentioned methods and devices, embodiments of the present disclosure may also provide a computer program product, including computer program instructions, which, when executed by a processor, enable the processor to execute the steps of the image processing method of various embodiments of the present disclosure described in the above-mentioned "Exemplary Method" section.

[0126] The computer program product may be written in any combination of one or more programming languages ​​to implement the operations of the disclosed embodiments, including object-oriented programming languages ​​such as Java, C++, and conventional procedural programming languages ​​such as C or similar programming languages. The program code may be executed entirely on the user's computing device, partially on the user's computing device, as a stand-alone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0127] In the description of this specification, the reference terms "one embodiment," "some embodiments," "example," "specific example," or "some examples" mean that the specific features, structures, materials, or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. Moreover, the specific features, structures, materials, or characteristics described may be combined in any appropriate manner in any one or more embodiments or examples. In addition, those skilled in the art may combine and integrate different embodiments or examples described in this specification, as well as features of different embodiments or examples, unless they are mutually inconsistent.

[0128] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features being referred to. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one such feature. Throughout the description of this application, "plurality" means two or more, unless otherwise specifically defined.

[0129] Any process or method described in the flowchart or otherwise described herein can be understood to represent a module, segment or portion of code comprising one or more executable instructions for implementing the steps of a specific logical function or process. The scope of the preferred embodiments of the present application includes other implementations in which the functions may be performed in a different order than shown or discussed, including performing the functions substantially simultaneously or in reverse order depending on the functions involved.

[0130] The logic and / or steps described in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, apparatus or device (such as a computer-based system, a system including a processor or other system that can fetch instructions from an instruction execution system, apparatus or device and execute instructions), or used in combination with such instruction execution systems, apparatuses or devices.

[0131] It should be understood that various parts of the present application can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented using software or firmware stored in a memory and executed by a suitable instruction execution system. All or part of the steps of the above embodiment method can be completed by instructing the relevant hardware through a program, which can be stored in a computer-readable storage medium. When the program is executed, it includes one or a combination of the steps of the method embodiment.

[0132] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing module, or each unit may exist physically separately, or two or more units may be integrated into a single module. The aforementioned integrated modules may be implemented in the form of hardware or in the form of software functional modules. If the aforementioned integrated modules are implemented in the form of software functional modules and sold or used as independent products, they may also be stored in a computer-readable storage medium. The storage medium may be a read-only memory, a magnetic disk, or an optical disk, etc.

[0133] The above is merely an exemplary embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any person skilled in the art can easily conceive of various modifications or substitutions within the technical scope described in this application, and such modifications or substitutions should be included in the scope of protection of the present application. Therefore, the scope of protection of the present application should be based on the scope of protection of the claims.

Claims

1. A method for processing sensor data, characterized in that: include: Obtaining n to-be-processed data sets of the target vehicle, wherein each of the to-be-processed data sets includes at least one type of sensor data and a collection time and a collection location corresponding to the to-be-processed data set, and n is an integer greater than 1; Determining m candidate data sets from the n data sets to be processed according to a type screening condition, wherein the type screening condition includes at least one specified data type, m is an integer and 1≤m≤n; At least one target data set is determined from the m candidate data sets based on a spatial density coefficient, wherein the spatial density coefficient is a maximum value of the number of target data sets in a preset unit area.

2. The method according to claim 1, characterized in that Determining at least one target data set from the m candidate data sets based on the spatial density coefficient includes: Determining at least one screening area based on the collection positions corresponding to the m candidate data sets, wherein any of the screening areas is associated with at least one candidate data set and the screening area includes the collection positions corresponding to all candidate data sets associated with the screening area; For any of the screening areas, determining the maximum number of target data sets corresponding to the screening area according to the spatial density coefficient; The at least one target data set is determined from the candidate data sets associated with the screening area according to the maximum number.

3. The method according to claim 2, characterized in that Determining the at least one target data set from the candidate data sets associated with the screening area according to the maximum number includes: In response to the number of the candidate data sets associated with the screening area being not greater than the maximum number, each of the candidate data sets associated with the screening area is determined as the target data set.

4. The method according to claim 2, characterized in that Determining at least one target data set from candidate data sets associated with the screening area according to the maximum number of the target data sets corresponding to the screening area includes: In response to the number of the candidate data sets associated with the screening area being greater than the maximum number, a maximum number of target data sets are screened out from the candidate data sets associated with the screening area based on the collection moments corresponding to the respective candidate data sets, wherein a time difference between the collection moment corresponding to any target data set in the screened target data sets and the collection moments corresponding to the other target data sets is greater than a preset duration.

5. The method according to claim 1, wherein Determine m candidate data sets from the n data sets to be processed according to the type screening condition, including: For any of the data sets to be processed, determining whether the data type of the sensor data included in the data set to be processed covers the specified data type in the type screening condition; In response to the data type of the sensor data included in the data set to be processed covering the specified data type in the type screening condition, a candidate data set corresponding to the data set to be processed is generated.

6. The method according to claim 5, characterized in that In response to the data type of the sensor data included in the data set to be processed covering the specified data type in the type screening condition, generating a candidate data set corresponding to the data set to be processed, including: The sensor data other than the specified data type in the data set to be processed is deleted to generate a candidate data set corresponding to the data set to be processed.

7. The method according to any one of claims 1 to 6, characterized in that The at least one sensor data includes at least 2D image data, and obtaining n sets of to-be-processed data of the target vehicle includes: Determining 2D image data from raw sensor data of the target vehicle; Detecting each frame of 2D image data using a category detection model to determine at least one frame of target 2D image data, each frame of the target 2D image data containing image data of an object of a specified type; For any frame of the target 2D image data, extracting other types of sensor data corresponding to the target 2D image data of the frame from the original sensor data, wherein the other types of sensor data have the same acquisition time and acquisition position as the target 2D image data of the frame; The target 2D image data and other types of sensor data corresponding thereto are constructed into a data set to be processed.

8. A sensor data processing device, characterized in that: include: a data acquisition module, configured to acquire n to-be-processed data sets of the target vehicle, wherein each of the to-be-processed data sets comprises at least one type of sensor data and a collection time and a collection location corresponding to the to-be-processed data set, and n is an integer greater than 1; a first screening module, configured to determine m candidate data sets from the n data sets to be processed according to a type screening condition, wherein the type screening condition includes at least one specified data type, m is an integer and 1≤m≤n; The second screening module is configured to determine at least one target data set from the m candidate data sets based on a spatial density coefficient, wherein the spatial density coefficient is a maximum value of the number of target data sets in a preset unit area.

9. An electronic device, characterized in that: The method comprises a memory, a processor and a computer program stored in the memory, wherein the processor implements the method according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images