Target detection method, device, electronic device and medium based on data synthesis

Abstract

The present application provides a method, device, electronic device and medium for target detection based on data synthesis. The method includes: obtaining multi-view image data, original point cloud data, initial detection frame and pre-created virtual model; rendering the virtual model into the multi-view image data and generating new multi-view image data; updating the original point cloud data based on the relationship between the virtual model and the original point cloud data; updating the initial detection frame according to the spatial position of the virtual model; training the perception model using the new multi-view image data, the updated point cloud data and the updated detection frame to obtain the trained perception model; using the multi-view image data and point cloud data actually collected around the vehicle as the input of the trained perception model, using the trained perception model to perform target detection, and outputting the target detection result. The present application can reduce the cost of perception data annotation and improve the precision and accuracy of model detection.

Description

Technical Field

[0001] The present application relates to the field of autonomous driving technology, and in particular to a target detection method, device, electronic device and medium based on data synthesis. Background Art

[0002] Perception, as a key part of the autonomous driving system, models the surrounding environment through sensor data, including identifying the location, size, orientation and speed of obstacles, as well as the location, type and length of lane lines. The main algorithm of the perception module relies on neural networks dominated by artificial intelligence (AI). The core elements of these neural networks include data, algorithms and computing power. Among these three elements, the quality of the data directly determines the performance of the perception module. However, in the industrial production environment, data annotation costs are extremely high, usually requiring millions to billions of dollars in investment.

[0003] Even with a large investment in labeling, it is still difficult for perception systems to fully cover long-tail problems (i.e., special situations that are rarely encountered during autonomous driving but need to be accurately identified). The cost of solving these long-tail problems is often comparable to the cost of solving the first 99% of the problems. Therefore, reducing the cost of long-tail perception problems has become a technical need that needs to be urgently addressed.

[0004] At present, the BEV (Bird's Eye View) architecture widely adopted by the autonomous driving industry has become the most mainstream perception architecture since its release at Tesla AI-Day 2021. Compared with the traditional post-fusion architecture, the BEV architecture has stronger scalability and extensibility and is completely data-driven. However, as a data-driven perception architecture, the BEV architecture requires a large amount of high-quality data for training, which makes the long-tail problem of data more serious and further increases the cost of data annotation.

[0005] Therefore, in summary, the existing technologies have the following problems in autonomous driving perception systems: data labeling is costly and difficult to cover all scenarios; long-tail problem processing is expensive and the solution efficiency is low; the existing data-driven architecture (such as BEV) has a large demand for high-quality data, which further increases the burden of data labeling, resulting in reduced detection accuracy and poor accuracy of the model. Summary of the invention

[0006] In view of this, the embodiments of the present application provide a target detection method, device, electronic device and medium based on data synthesis to solve the problems existing in the prior art, such as high cost of perception data annotation, serious data long-tail problem, and reduced model detection precision and accuracy.

[0007] According to a first aspect of an embodiment of the present application, a target detection method based on data synthesis is provided, comprising: acquiring multi-view image data, original point cloud data, an initial detection frame, and a pre-created virtual model; rendering the virtual model into the multi-view image data, and generating new multi-view image data; updating the original point cloud data based on the relationship between the virtual model and the original point cloud data, so that the updated point cloud data contains point cloud information of the virtual model; updating the initial detection frame according to the spatial position of the virtual model to generate a detection frame containing the virtual model; training a pre-constructed perception model using the new multi-view image data, the updated point cloud data, and the updated detection frame to obtain a trained perception model; using the multi-view image data and point cloud data actually collected around the vehicle as input to the trained perception model, performing target detection using the trained perception model, and outputting the target detection result.

[0008] According to a second aspect of an embodiment of the present application, a target detection device based on data synthesis is provided, comprising: an acquisition module, configured to acquire multi-view image data, original point cloud data, an initial detection frame and a pre-created virtual model; a rendering module, configured to render the virtual model into the multi-view image data and generate new multi-view image data; an update module, configured to update the original point cloud data based on the relationship between the virtual model and the original point cloud data, so that the updated point cloud data contains the point cloud information of the virtual model; a generation module, configured to update the initial detection frame according to the spatial position of the virtual model to generate a detection frame containing the virtual model; a training module, configured to train a pre-constructed perception model using the new multi-view image data, the updated point cloud data and the updated detection frame to obtain a trained perception model; a detection module, configured to use the multi-view image data and point cloud data actually collected around the vehicle as input to the trained perception model, use the trained perception model to perform target detection, and output the target detection result.

[0009] According to a third aspect of an embodiment of the present application, an electronic device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the steps of the above method are implemented when the processor executes the computer program.

[0010] According to a fourth aspect of an embodiment of the present application, a computer-readable storage medium is provided, which stores a computer program, and when the computer program is executed by a processor, the steps of the above method are implemented.

[0011] At least one of the above technical solutions adopted in the embodiments of the present application can achieve the following beneficial effects:

[0012] By acquiring multi-view image data, original point cloud data, initial detection frame and pre-created virtual model; rendering the virtual model into the multi-view image data and generating new multi-view image data; based on the relationship between the virtual model and the original point cloud data, updating the original point cloud data so that the updated point cloud data contains the point cloud information of the virtual model; updating the initial detection frame according to the spatial position of the virtual model to generate a detection frame containing the virtual model; using the new multi-view image data, updated point cloud data and updated detection frame to train the pre-built perception model to obtain the trained perception model; using the multi-view image data and point cloud data actually collected around the vehicle as the input of the trained perception model, using the trained perception model to perform target detection, and outputting the target detection result. This application can reduce the cost of perception data annotation and improve the precision and accuracy of model detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0014] Figure 1 It is a flowchart of a target detection method based on data synthesis provided in an embodiment of the present application;

[0015] Figure 2 is a schematic diagram of the structure of a target detection device based on data synthesis provided in an embodiment of the present application;

[0016] Figure 3 It is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0017] In the following description, specific details such as specific system structures, technologies, etc. are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present application. However, it should be clear to those skilled in the art that the present application may also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to prevent unnecessary details from obstructing the description of the present application.

[0018] Perception is an important part of the autonomous driving system. It aims to complete the surrounding environment modeling based on sensor data, such as giving the location / size / direction / speed of obstacles, and the location / type / length of lane lines. Most of the algorithms in the perception module are implemented by AI-dominated neural networks. The three most core elements for neural networks are: data, algorithms, and computing power. Data occupies the most core position in the research and development of perception modules. Its quality directly determines the actual performance of the perception module. In the production environment of the industrial sector, data usually costs millions to billions of dollars for labeling. Even if a huge amount of labeling funds is invested, it is difficult to cover the long-tail problem of perception. Solving the last 1% of the long-tail problems may require the same cost as the other 99% of the problems. Therefore, how to reduce the cost of the long-tail problem of perception is an urgent problem to be solved.

[0019] The current mainstream perception architecture is the BEV (Bird's Eye View) architecture. Since Tesla AI-Day 2021, BEV has become the most mainstream perception architecture in the autonomous driving industry. Compared with the previous post-fusion architecture, the BEV architecture has stronger scalability and extensibility and is a truly data-driven architecture. As a completely data-driven perception architecture, the BEV architecture requires more data, so the architecture faces a more serious data long-tail problem.

[0020] The BEV architecture uses camera images from multiple perspectives, lidar point cloud data, and millimeter-wave radar data to model the environment. This data-driven architecture requires a large amount of data for training, but the labeling cost of this data is high and it is difficult to cover all autonomous driving scenarios, resulting in poor performance of the model in some scenarios.

[0021] Therefore, the main problem faced by existing perception systems is the high cost of perception data annotation, especially when dealing with long-tail problems. The cost is even greater. This application aims to synthesize long-tail training samples of perception 3D detection models through computer graphics to reduce the cost of data annotation. In order to describe the technical solution of this application in more detail, the following embodiment will demonstrate the data synthesis ideas provided by this application in detail by synthesizing a training sample containing a bus and a pickup truck.

[0022] The contents of the technical solution of the present application are described in detail below with reference to the accompanying drawings and specific embodiments.

[0023] Figure 1 It is a flowchart of a target detection method based on data synthesis provided in an embodiment of the present application. Figure 1 The data synthesis-based object detection method can be performed by the autonomous driving system. Figure 1 As shown, the target detection method based on data synthesis may specifically include:

[0024] S101, acquiring multi-view image data, original point cloud data, initial detection frame, and pre-created virtual model;

[0025] S102, rendering the virtual model into the multi-view image data, and generating new multi-view image data;

[0026] S103, based on the relationship between the virtual model and the original point cloud data, updating the original point cloud data so that the updated point cloud data contains point cloud information of the virtual model;

[0027] S104, updating the initial detection frame according to the spatial position of the virtual model to generate a detection frame including the virtual model;

[0028] S105, training a pre-built perception model using the new multi-view image data, the updated point cloud data, and the updated detection frame to obtain a trained perception model;

[0029] S106, using the multi-view image data and point cloud data actually collected around the vehicle as input to the trained perception model, using the trained perception model to perform target detection, and outputting the target detection result.

[0030] First, the embodiment of the present application gives an image {I1, I2, · · · , In} of a surround view with n perspectives, a Lidar point cloud P (i.e., original point cloud data), and a corresponding detection box B (i.e., an initial detection box). For a model M (i.e., a virtual model) of art modeling, the present application uses rendering to place the model M into the scene and updates the corresponding image {I1, I2, · · · , In}, the point cloud P, and the 3D detection box B, so that the added model material M, the imaging points of the model M in the Lidar point cloud P, and the detection box B containing the model M can be seen in the updated surround view image {I1, I2, · · · , In}.

[0031] In some embodiments, rendering the virtual model into multi-view image data and generating new multi-view image data includes:

[0032] Using computer graphics methods, the virtual model is rendered into the viewing space corresponding to each camera;

[0033] Superimposing the virtual model with the original multi-view image data to generate multi-view image data including the virtual model;

[0034] The multi-view image data including the virtual model corresponding to all view angles are corrected and fused to obtain new multi-view image data.

[0035] Specifically, computer graphics methods are used to render the virtual model into the viewing space corresponding to each camera. This process includes creating a virtual scene for each camera view and inserting the pre-created virtual model into the scene. For example, a virtual car model is created using 3D modeling software and placed at the appropriate position in the camera view.

[0036] Furthermore, the virtual model is superimposed on the original multi-view image data to generate multi-view image data containing the virtual model. Specifically, based on computer graphics technology, the model M (virtual model) is rendered into the space associated with each camera, and the image of the virtual model is superimposed on the original image of the corresponding camera. In this way, the existence of the virtual model can be accurately displayed on the original image.

[0037] For example, in one example, a virtual car model is rendered into the forward-view camera so that it appears in the forward-view image data, and the same virtual car model can also be imaged in all 6 surround-view cameras to ensure the consistency and continuity of the virtual model from all perspectives.

[0038] Furthermore, the multi-view image data containing the virtual model corresponding to all viewpoints are corrected and fused to obtain new multi-view image data. The correction step includes adjusting the geometric shape and position of the virtual model at each viewpoint to ensure that the display effect of the virtual model at different viewpoints is consistent. The fusion step includes integrating the corrected multi-view image data to generate a unified, multi-view image data set. This process can use image processing techniques, such as multi-view stereo matching and image stitching technology, to ensure that the multi-view image data finally generated is visually seamless and conforms to the physical laws of the real world.

[0039] For example, in one example, a virtual car model may be rendered into a front-view camera to generate front-view image data, and the virtual car model may be imaged in all six surround-view cameras to generate corresponding surround-view image data.

[0040] Through the method of the above-mentioned embodiment, multi-perspective image data containing a virtual model can be obtained. The 3D car model synthesized based on the method proposed in this application has a high degree of authenticity and consistency, can effectively simulate vehicle scenes in the real world, and improve the training effect of the perception model.

[0041] In some embodiments, based on the relationship between the virtual model and the original point cloud data, updating the original point cloud data includes:

[0042] After inserting the virtual model into the scene, points in the original point cloud data that are blocked by the virtual model are determined, and the points in the original point cloud data that are blocked by the virtual model are deleted;

[0043] The intersection point between each laser radar beam and the virtual model is calculated, the nearest intersection point is used as the newly added point cloud data, and the newly added point cloud data is fused with the original point cloud data to generate point cloud data containing the virtual model.

[0044] Specifically, a pre-created 3D virtual model is inserted into the scene. For example, a virtual car model and a bus model are inserted into the scene of the original point cloud data. These virtual models should be consistent with the actual environment in the original point cloud data to ensure the authenticity and accuracy of the updated point cloud data.

[0045] Furthermore, after inserting the virtual model, the points in the original point cloud data that are blocked by the virtual model are determined. Specifically, the intersection of the ray from the origin of the laser radar device to each point in the original point cloud and each facet of the virtual model is determined. If the ray intersects with a facet of the virtual model, the point is considered blocked and needs to be deleted from the original point cloud data. Through this process, it can be ensured that the updated point cloud data does not contain invalid points blocked by the virtual model.

[0046] Furthermore, the intersection of each laser radar beam and the virtual model is calculated, and the nearest intersection is used as the newly added point cloud data. For example, for each laser radar beam, the intersection point with the virtual model is solved and these intersection points are added to the point cloud data. The newly added point cloud data should match the surface features of the virtual model to ensure that the updated point cloud data can accurately reflect the shape and position of the virtual model.

[0047] Furthermore, the newly added point cloud data is fused with the original point cloud data to generate updated point cloud data containing the virtual model. During the fusion process, it is necessary to ensure the spatial consistency and continuity of the newly added point cloud data and the original point cloud data, so as to obtain complete and accurate point cloud data. For example, new point cloud data after adding cars and buses to the original point cloud data.

[0048] For example, in one example, after a 3D virtual model is inserted into the scene, the occluded points in the original point cloud should be deleted, and the points hit by the Lidar on the newly added object should be added to the point cloud. In order to delete the points in the original point cloud, the intersection of the origin of the Lidar device and the ray of each point in the original point cloud with each face of the virtual model can be determined. If the ray intersects with a face of the virtual model, the point is occluded and needs to be deleted; otherwise, the point is retained. At the same time, in order to make the data more realistic, new point cloud data introduced by the virtual object needs to be added. To achieve this goal, the intersection point of each Lidar line beam with the virtual model can be solved, and the nearest intersection point can be updated to the point cloud data.

[0049] Through the method of the present embodiment described above, the virtual model can be effectively integrated into the original point cloud data to generate updated point cloud data containing the virtual model. This method can improve the recognition accuracy of the perception system for virtual targets and enhance the adaptability of the perception model in complex scenarios.

[0050] In some embodiments, determining points in the original point cloud data that are blocked by the virtual model includes:

[0051] Determine the ray between the origin of the laser radar device and each point in the original point cloud data, and determine whether the point in the original point cloud data is blocked by the virtual model based on the intersection between the ray and each facet of the virtual model;

[0052] When the ray intersects with the face in the virtual model, it is determined that the point in the original point cloud data is a point blocked by the virtual model; otherwise, it is determined that the point in the original point cloud data is not a point blocked by the virtual model.

[0053] Specifically, first, rays are emitted from the origin of the laser radar device, and the paths between these rays and each point in the original point cloud data are determined. Assume that the origin of the laser radar device is O and each point in the original point cloud data is P. i , then each ray can be represented as a line from O to P i straight line.

[0054] Furthermore, the intersection of the ray and each facet of the virtual model is calculated: for each ray from the origin O of the LiDAR device to point P i The ray of the virtual model M is used to determine whether it intersects with the face of the virtual model M. The virtual model M is usually composed of multiple facets (such as triangles or quadrilaterals). The intersection of the ray and these facets can be calculated to determine whether the point P i Whether it is blocked by the virtual model.

[0055] For example, a geometric algorithm may be used to solve the intersection problem between a ray and a polygonal patch. For example, a ray-triangle intersection test algorithm may be used to determine whether a ray intersects with a triangle patch of the virtual model M.

[0056] Furthermore, when the ray intersects with the face in the virtual model M, it is determined that the point P in the original point cloud data i Points that are blocked by the virtual model need to be deleted from the original point cloud data. Otherwise, the point P i The points that are not blocked by the virtual model are retained.

[0057] Through the above judgment, the original point cloud data can be updated and the points blocked by the virtual model can be deleted. Specifically, in order to delete the points in the original point cloud, the following steps can be performed:

[0058] Send out rays from the origin of the lidar device and determine the paths of these rays to each point in the raw point cloud data.

[0059] Calculate the intersection of each ray with each face of the virtual model M. If the ray intersects with a face of M, the point in the point cloud data is considered to be blocked and needs to be deleted; otherwise, the point is retained.

[0060] For example, in a specific example, assume that the origin of the LiDAR device is O, and the points in the original point cloud data are P1, P2, ..., P n , the virtual model is M, which is composed of multiple triangular patches. For each point P i Calculate from O to P i The ray path is used to determine whether the ray intersects with any triangle patch of M. If so, delete point P. i ; Otherwise, keep point P i .

[0061] Through the method of the present embodiment described above, points in the original point cloud data that are blocked by the virtual model can be effectively determined and deleted, thereby updating the point cloud data to make it more realistic and accurate.

[0062] In some embodiments, the initial detection frame is updated according to the spatial position of the virtual model, including:

[0063] Calculate the circumscribed cuboid of the virtual model, convert the calculated circumscribed cuboid into a unified coordinate space, and generate a detection frame of the virtual model;

[0064] The detection frame of the virtual model is added to the initial detection frame so as to update the initial detection frame and generate a detection frame including the virtual model.

[0065] Specifically, given a 3D virtual model M to be inserted, the circumscribed cuboid of the model is first calculated. The circumscribed cuboid is the smallest cuboid that completely surrounds the virtual model M, and each of its faces is tangent to the boundary of the virtual model. In order to calculate the circumscribed cuboid, the coordinates of all vertices of the virtual model M can be used to determine the minimum and maximum values ​​on each coordinate axis, thereby defining the six faces of the circumscribed cuboid.

[0066] Furthermore, the calculated circumscribed cuboid is transformed into a unified coordinate space. The unified coordinate space is the reference coordinate system used by the entire perception system, which ensures that data from different data sources (such as cameras and lidar) are processed and fused in the same coordinate system. Specifically, the circumscribed cuboid of the virtual model M can be transformed from its local coordinate system to the global coordinate system through a coordinate transformation matrix.

[0067] Furthermore, a detection frame of the virtual model is generated in the unified coordinate space. The detection frame is a rectangular frame used to identify the position and size of the virtual model in space. This step ensures that the virtual model M has a clear spatial representation in the global coordinate system and can be recognized and processed by the subsequent perception processing module.

[0068] Furthermore, the detection frame of the generated virtual model is added to the initial detection frame B. The initial detection frame B contains the detection frames of all known objects in the original point cloud data and the multi-view image data. By adding the detection frame of the virtual model to the initial detection frame B, a detection frame set containing the virtual model is generated. This process ensures that all detection frames are updated and processed in the same coordinate system.

[0069] For example, in one example, assuming a given 3D virtual model M, the update of the initial detection box includes the following steps:

[0070] The circumscribed cuboid of the virtual model M is calculated, for example, by obtaining the minimum and maximum coordinate values ​​of the model M on the three coordinate axes of x, y, and z, thereby defining six faces of the circumscribed cuboid.

[0071] The coordinate transformation matrix is ​​used to transform the circumscribed cuboid from the local coordinate system of the virtual model M to the global coordinate system of the perception system. This step may involve transformation operations such as translation, rotation, and scaling to ensure the correct position and orientation of the detection box of the virtual model in the global coordinate system.

[0072] A detection frame of the virtual model M is generated in the global coordinate system, and the detection frame is used to identify the position and size of the virtual model in space.

[0073] The generated virtual model detection frame is added to the initial detection frame B, and the initial detection frame B is updated to obtain a detection frame set containing the virtual model.

[0074] Through the method of the above embodiment, the detection frame of the virtual model can be effectively fused with the original detection frame to generate an updated detection frame containing the virtual model. This method can improve the detection capability of the perception system for virtual targets and enhance the adaptability of the system in complex scenarios.

[0075] In some embodiments, the pre-built perception model is trained using the new multi-view image data, the updated point cloud data, and the updated detection box, including:

[0076] Determine the model structure and model parameters of the pre-built perception model, and construct a training data set using the new multi-view image data, the updated point cloud data, and the updated detection box;

[0077] Input the training data set into the perception model, optimize the model parameters using the gradient descent algorithm, and adjust the model parameters by minimizing the loss function;

[0078] During the model training process, the gradient descent algorithm is repeatedly used to optimize the model parameters until the loss function converges to obtain the trained perception model.

[0079] Specifically, the model structure G and initial model parameters θ of the pre-built perception model are first determined. The model structure G includes the neural network architecture of the perception model, such as convolutional layers, pooling layers, fully connected layers, etc. The initial model parameters θ include weights and biases in these layers.

[0080] Furthermore, the training dataset is constructed using the new multi-view image data, the updated point cloud data, and the updated detection box. Specifically, the new multi-view image data is used as the input data of the model, and the updated point cloud data and detection box are used as the label data of the model. The training dataset consists of multiple sample pairs (xi, yi), where xi is the input data of the model (such as multi-view image and point cloud data), and yi is the corresponding label data (such as detection box).

[0081] Furthermore, the training data set is input into the perception model, and the model parameters are optimized using the gradient descent algorithm. Specifically, the stochastic gradient descent algorithm (SGD) is used to minimize the loss function, thereby adjusting the model parameter θ. The loss function is used to measure the difference between the model prediction results and the actual label data. Commonly used loss functions include cross entropy loss and mean square error.

[0082] Furthermore, during the model training process, the gradient descent algorithm is repeatedly used to perform optimization operations on the model parameters. The specific steps are as follows:

[0083] Calculate the model's predicted output and compare it with the labeled data to get the loss value.

[0084] Calculate the gradient of the loss function with respect to the model parameters and use the back-propagation algorithm to update the model parameters.

[0085] Repeat the above steps and iteratively update the model parameters until the loss function converges or reaches the preset number of training rounds.

[0086] When the loss function converges, the trained perception model is obtained. At this point, the model parameters θ have been optimized and can accurately detect objects in new multi-view image data and point cloud data.

[0087] Specifically, given the model structure G and model parameters θ, the main task of model training is to optimize the value of the parameter θ. To achieve this goal, some labeled training data {(x1, y1), (x2, y2), ···, (x m ,y m )}, and use the gradient descent algorithm SGD (Stochastic Gradient Descent) to optimize the model weight θ, where xi is the input of the model and yi is the manually annotated label. The training process includes the following steps:

[0088] Input data: The training data set {(x1, y1), (x2, y2), ···, (x m , y m )} is input into the perception model.

[0089] Calculate loss: Compare the model prediction output with the actual label data yi and calculate the loss function value.

[0090] Calculate the gradient: Through the back-propagation algorithm, calculate the gradient of the loss function with respect to the model parameters θ.

[0091] Update parameters: Use the gradient descent algorithm SGD to update the model parameters θ according to the gradient.

[0092] Iterative training: Repeat the above steps until the loss function converges and the optimized model parameters θ are obtained.

[0093] Through the method of the present embodiment described above, the pre-built perception model can be effectively trained using new multi-view image data, updated point cloud data, and updated detection frames, thereby improving the model's target detection accuracy and robustness.

[0094] In some embodiments, the multi-view image data and point cloud data actually collected around the vehicle are used as inputs of the trained perception model, the trained perception model is used to perform target detection, and the target detection results are output, including:

[0095] Input the multi-view image data and point cloud data actually collected around the vehicle into the trained perception model;

[0096] In the trained perception model, the feature extraction module is used to extract features from the data;

[0097] The extracted features are converted into a unified bird's-eye view space, and the features in the bird's-eye view space are temporally fused to obtain time-related multimodal features;

[0098] The multimodal features are input into the decoder related to the preset task, and the decoder is used to identify the target objects in the multi-view image data and output the target detection results.

[0099] Specifically, the multi-view image data and point cloud data actually collected around the vehicle are input into the trained perception model. The multi-view image data usually includes surround view images from multiple cameras, and the point cloud data comes from multiple Lidar sensors.

[0100] Furthermore, in the trained perception model, the feature extraction module is used to extract features from the input data. The feature extraction module includes multiple convolutional neural network (CNN) layers to extract high-dimensional feature representations from multi-view images and point cloud data. These feature representations contain the spatial and semantic information of the input data.

[0101] Furthermore, the extracted features are converted to a unified bird's-eye view space (BEV). The BEV model architecture maps data from different sensors to a unified two-dimensional plane through geometric transformation, thereby achieving a unified representation of the data. Specifically, the internal and external parameters of the camera can be used to project image features, and the spatial coordinates of the point cloud data can be used to map point cloud features.

[0102] Furthermore, the features in the bird's-eye view space are temporally fused to obtain time-related multimodal features. The temporal fusion module captures dynamic change information through joint processing of multiple frames of data, improving the timeliness and stability of feature representation. This process can be achieved through a recurrent neural network (RNN) or a long short-term memory network (LSTM).

[0103] Furthermore, the multimodal features are input into the decoder related to the preset task, and the decoder is used to identify the target object in the multi-view image data. The decoder uses different neural network structures, such as convolutional layers, fully connected layers, etc., according to the requirements of the specific perception task, and outputs the target detection results. Common perception tasks include 3D object detection, lane line recognition, and occupied grid recognition.

[0104] In one example, the perception model of the embodiment of the present application adopts the BEV model architecture, and its input generally includes a surround multi-view image, multiple laser radar point cloud data, and millimeter wave / ultrasonic radar data. The model extracts the features of the corresponding sensor data through various feature extraction modules and converts these features into a unified BEV space (i.e., a bird's-eye view space). For example, multi-view image data and point cloud data are mapped to the BEV space through geometric transformation and projection.

[0105] In the BEV space, the perception model performs temporal fusion of features and uses multi-frame data to enhance the timeliness of features. Subsequently, the temporally related BEV multimodal features are input into the task-related decoder. For the 3D object detection task, the decoder identifies and outputs the three-dimensional position, size, and type information of the target object; for the lane line recognition task, the decoder outputs the position and type information of the lane line; for the occupancy grid recognition task, the decoder outputs the occupancy around the vehicle.

[0106] Through the method of the present embodiment described above, the trained perception model can use multi-view image data and point cloud data to accurately detect targets and output high-precision detection results. This method improves the performance and reliability of the perception system in complex driving environments.

[0107] According to the technical solution provided in the embodiments of the present application, the technical solution of the present application has at least the following advantages:

[0108] The method proposed in this application can effectively synthesize a large number of long-tail training samples. Compared with the traditional 2D copy-paste synthesis method, the method proposed in this application is more in line with real physical laws. Specifically, the synthesis process of this application fully follows objective physical logic, and by rendering the virtual model into multi-view image data and point cloud data, a high degree of consistency with the real scene is achieved.

[0109] First, the virtual model is accurately rendered into multi-view image data through computer graphics methods, and the original point cloud data is updated according to the position and shape of the virtual model to generate multimodal data containing the virtual model. This method ensures the consistency and continuity of the virtual model under different perspectives, making the synthetic data more realistic.

[0110] Secondly, compared with the method of complete virtual rendering synthesis, the synthetic training samples of this application are based on data collected by real sensors, except for the newly added targets. Since the background and environmental data in the entire scene are real sensor data, the characteristics of the synthetic training samples of this application are closer to the real data, thereby improving the training effect and practicality of the perception model.

[0111] In addition, the method proposed in this application can effectively solve the problem of long-tail samples in autonomous driving training data and significantly reduce the labeling cost of autonomous driving training samples. By synthesizing high-quality long-tail training samples, the workload and cost of manual labeling can be reduced, and the data coverage and generalization ability of the model can be improved. This efficient data synthesis method not only improves the performance of the perception system, but also accelerates the practical application and implementation of autonomous driving technology.

[0112] In summary, the method of the present application not only enhances the capabilities of the autonomous driving perception system by synthesizing high-quality training samples that conform to the laws of physics, but also reduces the labeling cost of training data, providing important support for the promotion and application of autonomous driving technology.

[0113] The following is an embodiment of the device of the present application at the bottom of the window, which can be used to execute the embodiment of the method of the present application. For details not disclosed in the embodiment of the device of the present application, please refer to the embodiment of the method of the present application. Figure 2 Schematic diagram of the structure of the target detection device based on data synthesis provided in the embodiment of the present application. Figure 2 As shown, the target detection device based on data synthesis includes:

[0114] An acquisition module 201 is configured to acquire multi-view image data, original point cloud data, an initial detection frame, and a pre-created virtual model;

[0115] A rendering module 202 is configured to render the virtual model into the multi-view image data and generate new multi-view image data;

[0116] An updating module 203 is configured to update the original point cloud data based on the relationship between the virtual model and the original point cloud data, so that the updated point cloud data contains the point cloud information of the virtual model;

[0117] A generating module 204 is configured to update the initial detection frame according to the spatial position of the virtual model to generate a detection frame including the virtual model;

[0118] The training module 205 is configured to train the pre-built perception model using the new multi-view image data, the updated point cloud data, and the updated detection frame to obtain a trained perception model;

[0119] The detection module 206 is configured to use the multi-view image data and point cloud data actually collected around the vehicle as input to the trained perception model, perform target detection using the trained perception model, and output the target detection result.

[0120] In some embodiments, Figure 2 The rendering module 202 uses computer graphics methods to render the virtual model into the viewing space corresponding to each camera; superimposes the virtual model with the original multi-view image data to generate multi-view image data containing the virtual model; and corrects and fuses the multi-view image data containing the virtual model corresponding to all viewing angles to obtain new multi-view image data.

[0121] In some embodiments, Figure 2After the update module 203 inserts the virtual model into the scene, it determines the points in the original point cloud data that are blocked by the virtual model, and deletes the points in the original point cloud data that are blocked by the virtual model; calculates the intersection of each laser radar beam and the virtual model, takes the nearest intersection as the newly added point cloud data, merges the newly added point cloud data with the original point cloud data, and generates point cloud data containing the virtual model.

[0122] In some embodiments, Figure 2 The update module 203 determines the ray between the origin of the laser radar device and each point in the original point cloud data, and judges whether it is a point in the original point cloud data blocked by the virtual model according to the intersection between the ray and each facet of the virtual model; wherein, when the ray intersects with the facet in the virtual model, it is judged that the point in the original point cloud data is a point blocked by the virtual model, otherwise, it is judged that the point in the original point cloud data is not a point blocked by the virtual model.

[0123] In some embodiments, Figure 2 The generation module 204 calculates the circumscribed cuboid of the virtual model, converts the calculated circumscribed cuboid into a unified coordinate space, and generates a detection frame of the virtual model; the detection frame of the virtual model is added to the initial detection frame so as to update the initial detection frame and generate a detection frame including the virtual model.

[0124] In some embodiments, Figure 2 The training module 205 determines the model structure and model parameters of the pre-built perception model, and uses the new multi-view image data, the updated point cloud data and the updated detection frame to build a training data set; inputs the training data set into the perception model, optimizes the model parameters using the gradient descent algorithm, and adjusts the model parameters by minimizing the loss function; during the model training process, repeatedly uses the gradient descent algorithm to perform the optimization operation of the model parameters until the loss function converges to obtain the trained perception model.

[0125] In some embodiments, Figure 2 The detection module 206 inputs the multi-view image data and point cloud data actually collected around the vehicle into the trained perception model; in the trained perception model, the feature extraction module is used to extract features from the data; the extracted features are converted into a unified bird's-eye view space, and the features in the bird's-eye view space are time-series fused to obtain time-series-related multimodal features; the multimodal features are input into a decoder related to a preset task, and the decoder is used to identify the target object in the multi-view image data, and the target detection result is output.

[0126] It should be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0127] Figure 3 Schematic diagram of the structure of the electronic device 3 provided in the embodiment of the present application. Figure 3 As shown, the electronic device 3 of this embodiment includes: a processor 301, a memory 302, and a computer program 303 stored in the memory 302 and executable on the processor 301. When the processor 301 executes the computer program 303, the steps in the above-mentioned method embodiments are implemented. Alternatively, when the processor 301 executes the computer program 303, the functions of the modules / units in the above-mentioned device embodiments are implemented.

[0128] Exemplarily, the computer program 303 may be divided into one or more modules / units, which are stored in the memory 302 and executed by the processor 301 to complete the present application. The one or more modules / units may be a series of computer program instruction segments capable of completing specific functions, which are used to describe the execution process of the computer program 303 in the electronic device 3.

[0129] The electronic device 3 may be a desktop computer, a notebook, a PDA, a cloud server, or other electronic device. The electronic device 3 may include, but is not limited to, a processor 301 and a memory 302. Those skilled in the art will appreciate that Figure 3 It is only an example of the electronic device 3 and does not constitute a limitation of the electronic device 3. It may include more or fewer components than shown in the figure, or a combination of certain components, or different components. For example, the electronic device may also include input and output devices, network access devices, buses, etc.

[0130] The processor 301 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or the processor may be any conventional processor, etc.

[0131] The memory 302 may be an internal storage unit of the electronic device 3, for example, a hard disk or memory of the electronic device 3. The memory 302 may also be an external storage device of the electronic device 3, for example, a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the electronic device 3. Further, the memory 302 may also include both an internal storage unit and an external storage device of the electronic device 3. The memory 302 is used to store computer programs and other programs and data required by the electronic device. The memory 302 may also be used to temporarily store data that has been output or is to be output.

[0132] The technicians in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In practical applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated in a processing unit, or each unit can exist physically separately, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here.

[0133] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0134] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0135] In the embodiments provided in the present application, it should be understood that the disclosed devices / computer equipment and methods can be implemented in other ways. For example, the device / computer equipment embodiments described above are only schematic. For example, the division of modules or units is only a logical function division. There may be other division methods in actual implementation. Multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0136] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0137] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0138] If the integrated module / unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. The computer program may include computer program code, which may be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may include: any entity or device capable of carrying computer program code, recording medium, U disk, mobile hard disk, disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signal, telecommunication signal and software distribution medium, etc.

[0139] The above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.

Claims

1. A target detection method based on data synthesis, characterized in that: include: Obtain multi-view image data, original point cloud data, initial detection frame, and pre-created virtual model; Rendering the virtual model into the viewing angle space corresponding to each camera using a computer graphics method; Superimposing the virtual model with the original multi-view image data to generate multi-view image data containing the virtual model; Correcting and fusing the multi-view image data including the virtual model corresponding to all view angles to obtain new multi-view image data; After inserting the virtual model into the scene, determining points in the original point cloud data that are blocked by the virtual model, and deleting the points in the original point cloud data that are blocked by the virtual model; Calculate the intersection of each laser radar line beam and the virtual model, use the nearest intersection as newly added point cloud data, and fuse the newly added point cloud data with the original point cloud data to generate point cloud data containing the virtual model; updating the initial detection frame according to the spatial position of the virtual model to generate a detection frame including the virtual model; Using the new multi-view image data, the updated point cloud data, and the updated detection frame, a pre-built perception model is trained to obtain a trained perception model; The multi-view image data and point cloud data actually collected around the vehicle are used as input to the trained perception model, the trained perception model is used to perform target detection, and the target detection result is output.

2. The method according to claim 1, characterized in that The determining the points in the original point cloud data that are blocked by the virtual model includes: Determine a ray between the origin of the laser radar device and each point in the original point cloud data, and determine whether the point in the original point cloud data is blocked by the virtual model according to the intersection between the ray and each facet of the virtual model; When the ray intersects with a face in the virtual model, it is determined that the point in the original point cloud data is a point blocked by the virtual model; otherwise, it is determined that the point in the original point cloud data is not a point blocked by the virtual model.

3. The method according to claim 1, characterized in that The updating of the initial detection frame according to the spatial position of the virtual model includes: Calculating a circumscribed cuboid of the virtual model, converting the calculated circumscribed cuboid into a unified coordinate space, and generating a detection frame of the virtual model; The detection frame of the virtual model is added to the initial detection frame so as to update the initial detection frame and generate a detection frame including the virtual model.

4. The method according to claim 1, characterized in that: The using the new multi-view image data, the updated point cloud data and the updated detection frame to train the pre-built perception model includes: Determine the model structure and model parameters of the pre-built perception model, and construct a training data set using the new multi-view image data, the updated point cloud data, and the updated detection frame; Inputting the training data set into the perception model, optimizing the model parameters using a gradient descent algorithm, and adjusting the model parameters by minimizing a loss function; During the model training process, the gradient descent algorithm is repeatedly used to optimize the model parameters until the loss function converges to obtain the trained perception model.

5. The method according to claim 1, characterized in that The method uses the multi-view image data and point cloud data actually collected around the vehicle as the input of the trained perception model, uses the trained perception model to perform target detection, and outputs the target detection result, including: Inputting the multi-view image data and point cloud data actually collected around the vehicle into the trained perception model; In the trained perception model, extracting features from data using a feature extraction module; The extracted features are converted into a unified bird's-eye view space, and the features in the bird's-eye view space are temporally fused to obtain time-related multimodal features; The multimodal features are input into a decoder associated with a preset task, the decoder is used to identify the target object in the multi-view image data, and the target detection result is output.

6. A target detection device based on data synthesis, characterized in that: include: An acquisition module, configured to acquire multi-view image data, original point cloud data, an initial detection frame, and a pre-created virtual model; A rendering module is configured to render the virtual model into a viewing angle space corresponding to each camera using a computer graphics method; The virtual model is superimposed on the original multi-view image data to generate multi-view image data containing the virtual model; the multi-view image data containing the virtual model corresponding to all view angles is corrected and merged to obtain new multi-view image data; The updating module is configured to, after inserting the virtual model into the scene, determine the points in the original point cloud data that are blocked by the virtual model, and delete the points in the original point cloud data that are blocked by the virtual model; calculate the intersection of each laser radar line beam and the virtual model, use the nearest intersection as the newly added point cloud data, fuse the newly added point cloud data with the original point cloud data, and generate point cloud data containing the virtual model; A generating module, configured to update the initial detection frame according to the spatial position of the virtual model to generate a detection frame including the virtual model; A training module is configured to train a pre-built perception model using the new multi-view image data, the updated point cloud data, and the updated detection frame to obtain a trained perception model; The detection module is configured to use the multi-view image data and point cloud data actually collected around the vehicle as the input of the trained perception model, use the trained perception model to perform target detection, and output the target detection result.

7. An electronic device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 5 are implemented.

8. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 5 are implemented.

Images