Data synthesis device and method

Abstract

The invention provides data synthesis equipment which comprises at least one first sensor used for providing at least one first type of real detection data; and a data simulation engine configured to process the first type of real detection data by using a trained data simulation model to generate at least a second type of virtual detection data different from the first type.

Description

Technical Field

[0001] The present invention relates to data processing, and in particular to the generation of virtual data. Background Art

[0002] At present, with the development of automobiles, especially autonomous driving technology, in order to achieve safe and reliable autonomous driving, in the development and improvement process of the ADAS (Advanced Driver Assistance System) functions equipped with the automobile autonomous driving system, data from different types of sensors (for example, images, radar and lidar point clouds) are needed for system training and functional testing. However, in practice, it takes a lot of time and cost to install different sensors and collect sensor data accordingly. Summary of the invention

[0003] The present invention proposes a method for generating other types of virtual sensor data from at least one type of sensor detection data based on deep learning, wherein the virtual sensor data can replace real data that should have been detected by an actual sensor.

[0004] According to one aspect of the present invention, there is provided a method for training a data simulation model, comprising acquiring first real detection data collected by a first type of sensor and second real detection data collected by a second type of sensor, wherein the second real detection data and the first real detection data have different data types; creating a training data set for training the data simulation model, comprising a first training data subset generated based on the first real detection data and a second training data subset generated based on the second real detection data; using the training data set to jointly train a first neural network model and a second neural network model, comprising: training the first neural network model with the first training data subset to generate virtual detection data, wherein the virtual detection data and the second real detection data have the same data type; training the second neural network model with the second training data subset and the virtual detection data to identify the loss between the second real detection data and the virtual detection data; wherein when the loss meets a predetermined requirement, the trained first neural network model is output as the data simulation model.

[0005] According to one aspect of the present invention, there is provided a data synthesis device, comprising at least one first sensor for providing at least a first type of real detection data; and a data simulation engine, configured to process the first type of real detection data using a trained data simulation model to generate at least a second type of virtual detection data different from the first type, wherein the data simulation model is trained using a method as claimed in any one of claims 1-7.

[0006] According to one aspect of the present invention, there is provided a data synthesis method, comprising collecting at least a first type of real detection data using at least one first sensor; and processing the first type of real detection data using a trained data simulation model to generate at least a second type of virtual detection data different from the first type. BRIEF DESCRIPTION OF THE DRAWINGS

[0007] Figure 1 Schematically illustrates a data processing device according to an embodiment of the present invention;

[0008] Figure 2 The flowchart of the training data simulation model is shown as an example. DETAILED DESCRIPTION

[0009] The exemplary embodiments of the present invention will be described in more detail below with reference to the accompanying drawings. Although exemplary embodiments are shown in the accompanying drawings, it should be understood that the present invention can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided to enable a more thorough understanding of the present invention.

[0010] Figure 1 A data processing device according to an example of the present invention is shown, which is used to synthesize other types of virtual sensor data by processing at least one type of sensor detection data. As shown in the figure, the data processing device 100 includes one or more sensors 100 and a data simulation engine 200. The sensor 100 can be installed on a vehicle, for example, to detect the environment around the vehicle, thereby providing at least one type of detection data. In this application, the vehicle environment data actually detected by the sensor is referred to as real detection data PyD. As an example, the sensor 100 can be a front-mounted camera or camera on a vehicle, which is used to capture the original environment image or photo around the vehicle, represented here by FRM, or to capture the video stream around the vehicle. Obviously, based on the video stream, one or more frames of the original environment image FRM sequence can be obtained. In the present invention, in order to meet the subsequent processing requirements, the image frame FRM has a predetermined size and format for subsequent processing. The data format here can be a common raw data format such as RAW, YUV, or it can be data after algorithm compression, such as H.264 and H.265. The size of the image frame FRM needs to be greater than or equal to a specific pixel size to ensure that the minimum size requirement can be met after cropping and scaling.

[0011] The data simulation engine 200 is configured to generate at least another type of detection data, such as radar data, based on the real detection data provided by the sensor 100, such as the image frame FRM. In the present invention, such data synthesized based on data conversion is also referred to as virtual detection data VrD. In this example, such virtual radar data VrD has the characteristics of generated data of the radar sensor, that is, it can be regarded as detection data generated by the radar detecting the vehicle environment. Here, the radar can be a millimeter wave radar or a laser radar, etc. In the following description of the embodiment of the present invention, the image frame FRM is used as the source, and the virtual point cloud data VrD is used as the target synthetic data.

[0012] According to an example, as shown in the figure, the data simulation engine 200 includes a preprocessing module 201, which is configured to perform image preprocessing on the original image FRM. The preprocessing here may include (1) cropping the original image FRM to a standard image with a predetermined ratio L*W, where the length L and width W of the image can be predetermined according to actual needs; (2) then performing image optimization processing on the standard image FRM, including one of the following processing: image denoising; brightness adjustment; contrast adjustment. Various noise reduction, brightness and contrast processing technologies known in the prior art can be used here, all of which can be used for image optimization processing; (3) Finally, the optimized image FRM is identified to determine whether the image FRM has abnormal conditions such as camera overexposure, occlusion, insufficient image clarity, blur, etc. For images with these abnormal conditions, they are eliminated, thereby obtaining a normal standard image for data synthesis, which is represented by sFRM here for ease of description.

[0013] For training, the preprocessing module 201 is further configured to extract image features p of each image sFRM, including but not limited to:

[0014] The overall features of the image, which take into account the global information of the entire image, including the image size, aspect ratio, brightness distribution, average brightness, average contrast, etc.

[0015] Semantic segmentation features of the entire image area, including semantic segmentation area, area type (e.g. road surface, building, vehicle, etc.), area shape, texture features, depth information, etc.

[0016] Features of each target object in the image, including: classification features of the target (people, cars, trees, railings, etc.); geometric features of the target, such as position, three-dimensional size, orientation, posture, bounding box or outline, distance (such as Euclidean distance, cosine distance, etc.); shape features of the target, such as rectangularity, eccentricity, curve description, etc.;

[0017] Pixel-level features in an image include color information such as RGB channel values, brightness information, and texture information.

[0018] Based on these image features, the multi-dimensional features of the image sFRM are established. 1 ,p 2 ,……,pM} means, here p i represents each image feature extracted from the image sFRM, and M represents the dimension of P, that is, the number of each image feature extracted.

[0019] The data simulation engine 200 calls a trained data simulation model 202 to process the multi-dimensional features P of the image sFRM. In the present invention, the data simulation model 202 is trained to perform the synthesis of virtual target data. For example, in this example, it can be trained to generate virtual point cloud data X={x} based on the image data sFRM. For example, for a virtual radar point cloud, each target point x in the point cloud can contain the following multi-dimensional feature information: such as radial distance, radial relative velocity, azimuth, pitch angle, reflection intensity, and classification, etc.; and for a virtual lidar point cloud, each target point x in the point cloud can contain the following multi-dimensional feature information: three-dimensional coordinates, reflection intensity, color, and classification, etc. Here, the multi-dimensional features of the target point cloud data can be obtained by training the corresponding data simulation model 202 according to actual needs. In the example of the present invention, the data simulation model 202 can be implemented using various currently known neural network architectures, such as a convolutional neural network CNN or a variational autoencoder VAE.

[0020] Therefore, taking the generation of a virtual target laser point cloud as an example, the data simulation engine 200 calls the trained data simulation model 202 to calculate the multi-dimensional features P of the image sFRM = {p 1 ,,p 2 ,……,p M} is processed to obtain the estimated virtual target laser point cloud X, where each target point x contains N-dimensional features {x 1 ,x 2 ,…,x N}, each dimensional feature corresponds to an attribute feature among three-dimensional coordinates, reflection intensity, color, and classification.

[0021] In the above example, the generation of point cloud data is described based on a single-frame image example. In another example of the present invention, a continuous point cloud can also be generated based on a video stream. To this end, the video stream can be converted into a single-frame image sFRM sequence at a constant or variable time interval, and then the data simulation model 202 is called to process the image frame at each time point in the sequence to generate the corresponding point cloud. Then, the point clouds at different time points are connected in series according to the timeline, so as to obtain a radar / lidar point cloud sequence under continuous image frames.

[0022] Figure 2 A flowchart of an example of training a data simulation model 202 according to the present invention is shown. The data simulation model 202 can be pre-trained on a computing platform of an automobile manufacturer and installed in a vehicle by pre-installation or downloading, such as by OTA, and called by a computing system equipped in the vehicle, such as an ECU (electronic control unit). In order to train the data simulation model 202, actual environmental data detected by sensors during actual driving of the vehicle are collected on the computing platform.

[0023] As shown in the figure, in step 301, real detection data collected by a first type of sensor and real detection data collected by a second type of sensor are obtained, wherein the two types of sensors are of different types, and thus the collected real detection data also have different data types or formats. For ease of description, in the following description, the first type of sensor is an image sensor such as a vehicle-mounted camera, and the second type of sensor is a radar / lidar as an example.

[0024] According to an embodiment of the present invention, a vehicle-mounted camera and a radar / lidar are installed on the acquisition vehicle at the same time, so that these sensors can "see" the same environmental scene at the same time. To this end, the raw data collected by each sensor contains timestamp information for time synchronization of sensor information. The collected image can be in the raw data format commonly used in video and graphics processing pipelines (pipeline), such as YUV, RAW, etc., or it can be data after algorithm compression (such as H.264, H.265); the data size must be greater than or equal to a specific pixel size so that it can meet the minimum size requirement after cropping and scaling.

[0025] In addition, with the development of machine intelligence. Visual perception algorithms can also be installed in vehicle cameras, so that instances of target objects in captured images can be identified. Radar / LiDAR can also be configured to provide point cloud information, such as the distance, motion state, and reflection intensity of target points. The real detection data including image and point cloud data obtained here can be collected by image sensors and radar / LiDAR on different types of vehicles, or by image sensors and radar / LiDAR on the same vehicle at different times, so that the trained model can adapt to different sensor configurations and vehicle environments. For ease of explanation, multiple image samples collected by the camera are marked as {FRM}, and point clouds collected by radar / LiDAR are marked as {X}.

[0026] In step 303, a training data set for training a data simulation model is created based on the real detection data {FRM}{X} collected in step 301. In this example, a first training data subset DataSet1 is generated based on the image sample {FRM} and a second training data subset DataSet1 is generated based on the point cloud data {X}.

[0027] According to an example of the present invention, when creating the first training data subset DataSet1, preprocessing is performed on each image frame in {FRM}, including cropping each original image FRM therein to a specific pixel size to obtain a standard image with a predetermined ratio, thereby obtaining an image of uniform size; image optimization processing is performed on the standard image, including one of the following processing: image denoising; brightness adjustment; contrast adjustment, thereby ensuring image quality; finally, abnormal images are eliminated from the standard image that has undergone image optimization processing, such as camera overexposure, occlusion, blur, etc., thereby eliminating abnormal images from the training data set, thereby eliminating abnormal images from the training data set, thereby, the first training data subset DataSet1 contains preprocessed image frames sFRM.

[0028] According to the present invention, the point cloud data in the second training data subset DataSet2 is used as the point cloud true value, i.e., the label Label, during the model training process. In order to achieve a good training effect, according to an embodiment of the present invention, when the second training data subset DataSet1 is generated based on the point cloud {X}, the point cloud data {X} also needs to be optimized. Specifically, the currently known multi-sensor fusion algorithm can be used to fuse the outputs of different sensors about the same environmental information. The fusion benchmark used here is to compare whether the points or target information detected by each sensor can match or associate with each other. As an example, first, a visual algorithm is used to determine the target bounding box (BoundingBox) in the image frame sFRM, or semantic segmentation is performed on the identified object instance to identify pedestrians, vehicles, lane lines, road edges, trees, etc., which are hereinafter referred to as Object1. Subsequently, based on the time correlation, the corresponding point cloud collected by the radar / lidar sensor is obtained and the object Object2 contained in the point cloud is identified using the target recognition algorithm. Then, the object fusion processing of the objects Object1 and Object2 is performed, that is, the margin of mutual matching between Object1 and Object2 is determined. Among them, if the matching margin meets the minimum requirement, such as 70%, Object2 and Object1 are basically fused, so the point cloud X corresponding to the current Object2 is put into the training data subset DataSet2; if the minimum margin requirement is not met, the point cloud corresponding to Object2 as interference is eliminated, and the corresponding image frame sFRM is also eliminated from the training data subset DataSet1.

[0029] In step 305, a deep machine learning model DNN is trained using the training data sets DataSet1 and DataSet2 obtained by the above process to generate a data simulation model 202. Here, the deep machine learning model DNN can be implemented using a variety of existing architectures in the prior art, including but not limited to generative adversarial networks (GANs), variational autoencoders VAEs, and CNNs. Here, GAN is used as an example to illustrate the training of the data simulation model 202. Usually GAN includes two parts, namely a generator and a discriminator. The generator accepts the image frame sFRM in the input training data set DataSet1 and synthesizes virtual radar data X′. Here, the generator can be implemented using architectures such as convolutional neural networks (CNNs) or variational autoencoders (VAEs).

[0030] The synthesized virtual radar data X' and the real point cloud data X in the training data set DataSet 2 are provided to the discriminator, wherein the discriminator is designed to distinguish the difference between the virtual radar data X' and the real point cloud data X in the training data set DataSet 2. Here, the discriminator can be implemented using CNN or a fully connected neural network. Thus, by jointly training the generator and discriminator in the GAN using the training data sets DataSet1 and DataSet2, the discriminator determines whether the difference between the virtual radar data X' and the real point cloud data X in the training data set DataSet 2 meets the minimization requirement.

[0031] In step 307, if the discriminator determines in step 305 that the difference between the virtual radar data X′ and the real point cloud data X in the training data set DataSet 2 meets the minimization requirement, it is confirmed that the generator at this time can accurately estimate the point cloud data X based on the image frame sFRM, and thus the generator under the current parameters is output as the data simulation model 202. In one example of the present invention, the discriminator can use the currently known adversarial loss function (Adversarial Loss) to characterize the difference between the virtual radar data X′ and the real point cloud data X, and by minimizing the adversarial loss between the generator and the discriminator, the radar data X′ generated by the generator is as close to the real radar data X as possible, while making it difficult for the discriminator to distinguish between the generated data and the real data.

[0032] In another example of the present invention, the discriminator may also use a feature matching network to measure the difference between the virtual radar data X′ and the real point cloud data X in the training data set DataSet 2. Specifically, the synthetic radar data X′ and the real radar data X output by the generator are input into the pre-trained feature extraction network, and the feature matching network extracts the features of the virtual radar data X′ and the real point cloud data X respectively and calculates the feature difference or loss between the two. When the feature matching loss is the smallest, it indicates that the generator has generated synthetic data that is closer to the real data, so the training is completed, and the generator at this time is output as a data simulation model.

[0033] According to another example of the present invention, the trained data simulation model 202 is verified. To this end, a new vehicle surrounding environment image FRM and real point cloud data can be collected, wherein after preprocessing, the new image sFRM can be used as input and provided to the data simulation model 202 to obtain synthesized radar data, and the similarity and consistency between the synthesized data and the real point cloud data can be evaluated, where other metrics such as mean square error, structural similarity index, etc. can be used for evaluation. Thus, using the trained data simulation model 202, other types of data, such as point cloud data, can be synthesized based on the image, thereby providing additional useful auxiliary information for target detection, tracking, environmental perception, etc.

[0034] Although the general embodiment of the present invention is described here with images as source data and radar / lidar data as target synthetic data, it is obvious that the present invention is not limited thereto, but can be applied to the synthesis of any other suitable type of data, such as synthesizing images from real radar / lidar data. Figure 2 In the example shown, the preprocessing module 201 is integrated in the data simulation engine 200, but in other examples, the preprocessing module 201 may also exist independently of the simulation engine 200 and communicate with the simulation engine 200. In addition, the simulation engine 200 of the present invention can be implemented by a dedicated computing device on the vehicle, or by a general ECU (electronic control unit) executing the data synthesis method described in the present disclosure, and the data synthesis method can be implemented by the ECU executing a computer program, wherein the computer program can be stored in the ECU or in a storage medium accessible to other ECUs.

[0035] Although the multi-node management circuit and server of the present invention are described above in conjunction with specific examples, the present invention is not limited to these disclosed embodiments. Those skilled in the art will appreciate that more embodiments of the present invention can be obtained by combining the disclosed embodiments, and these embodiments are also within the scope of protection of the present invention.

Claims

1. A method for training a data simulation model, comprising: Acquire first real detection data collected by a first type of sensor and second real detection data collected by a second type of sensor, wherein the second real detection data has a different data type from the first real detection data; Creating a training data set for a training data simulation model, including a first training data subset generated based on the first real detection data and a second training data subset generated based on the second real detection data; Using the training data set to jointly train the first neural network model and the second neural network model includes: Training a first neural network model using the first training data subset to generate virtual detection data, wherein the virtual detection data has the same data type as the second real detection data; training the second neural network model using the second training data subset and the virtual detection data to identify the loss between the second real detection data and the virtual detection data; When the loss meets the predetermined requirement, the trained first neural network model is output as the data simulation model.

2. The method of claim 1, wherein the first real detection data is a plurality of frames of original real images collected by an image sensor or a video stream containing the plurality of frames of original real images; The second real detection data is point cloud data collected by a radar, and the radar includes at least one of a millimeter wave radar and a laser radar; The real image and the point cloud data have time stamps for time synchronization.

3. The method of claim 2, wherein said creating a training data set for training said data simulation model comprises: Preprocessing the original real image to generate the first training data subset includes cropping the original image to a standard image with a predetermined ratio; The image optimization processing is performed on the standard image, including at least one of the following processing: image denoising; brightness adjustment; contrast adjustment; and eliminating abnormal images from the standard image that has undergone image optimization processing.

4. The method of claim 3, wherein said creating a training data set for training said data simulation model comprises: Preprocessing the point cloud data includes: Identifying a target object instance in the point cloud data; Based on the timestamp, the identified object instance is merged with the object instance identified based on the original real image to determine the matching degree between the two; The point cloud data of the object instance where the matching degree satisfies a predetermined margin value is used as a second training data subset.

5. The device as claimed in claims 1-4, wherein the first neural network model is implemented using CNN or VAE neural network; and the second neural network model is implemented using CNN or fully connected neural network.

6. The method of claim 5, wherein the first neural network model and the second neural network model constitute a Generative Adversarial Network (GAN) model, and the identifying the loss between the second real detection data and the virtual detection data comprises: An adversarial loss between the second real detection data and the virtual detection data is minimized.

7. The method of claim 5, wherein said identifying a loss between said second real detection data and said virtual detection data comprises: A feature matching network is used to extract at least a first feature of the virtual detection data and at least a second feature of the second real detection data, and a feature difference between the first feature and the second feature is calculated as the loss.

8. A data synthesis device, comprising: at least one first sensor for providing at least a first type of real detection data; A data simulation engine is configured to process the first type of real detection data using a trained data simulation model to generate at least a second type of virtual detection data different from the first type.

9. The device of claim 8, wherein the at least one first sensor comprises an image sensor for providing at least one frame of original real image or an original video stream containing at least one frame of image; The at least second type of virtual detection data represents virtual radar data, and the radar includes millimeter wave radar and / or laser radar.

10. The apparatus of claim 9, wherein the data simulation engine further comprises: The preprocessing module is configured to perform image preprocessing on the original image, including: Cropping the original image to a standard image with a predetermined ratio; Performing image optimization processing on the standard image, including one of the following processing: image denoising; brightness adjustment; contrast adjustment; Abnormal images are eliminated from standard images that have undergone image optimization processing.

11. The device as claimed in any one of claims 8-10, wherein the data simulation model is trained by using the method as claimed in any one of claims 1-7.

12. A data synthesis method, comprising: collecting at least a first type of real detection data using at least a first sensor; The first type of real detection data is processed using a trained data simulation model to generate at least a second type of virtual detection data different from the first type.

13. A computer-readable storage medium having a readable program stored thereon, wherein when the program is executed by a computing device, the computing device is caused to implement the method of claim 12.

Images