Virtual obstacle detection method and device based on data enhancement and storage medium
Data containing virtual obstacle noise is generated through data augmentation technology, and virtual obstacle detection model is trained, which solves the problem of low detection accuracy of virtual obstacles in autonomous driving systems, improving detection accuracy and system safety.
Patent Information
- Application Number
- CN202411140041.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-20
- Publication Date
- 2024-11-08
Smart Images

Figure CN118918566A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of autonomous driving technology, and in particular to a virtual obstacle detection method, device and storage medium based on data enhancement. Background Art
[0002] Perception is a crucial module in the autonomous driving system. It models the surrounding environment through sensor data and identifies the location, size, direction and speed of obstacles, as well as the location, type and length of lane lines. In the perception module, most algorithms rely on AI-led neural networks. For neural networks, data, algorithms and computing power are their core elements.
[0003] In the development of perception modules, data quality directly determines their performance. However, in the production environment of the industry, data annotation usually costs millions to billions of dollars, and even so, it is still difficult to cover the long-tail problems in perception. Solving the last 1% of the long-tail problems may cost the same as solving the first 99% of the problems. Therefore, how to reduce the cost of the long-tail problems in perception is a difficult problem that needs to be solved urgently.
[0004] As one of the core sensors in the perception system, Lidar has excellent ranging capabilities and mature obstacle detection algorithms. However, noise in real scenes can cause the Lidar detection model to identify virtual, non-existent obstacles, causing the autonomous vehicle to brake suddenly and cause serious traffic accidents. The industry has tried to introduce image semantic information to solve the virtual obstacle problem, but these methods have limited effectiveness because they require high time synchronization between sensors.
[0005] At present, the BEV (bird's eye view) multimodal fusion solution is believed to be able to completely solve the virtual obstacle problem. However, in actual scenarios, there is usually a lack of sufficient training samples containing Lidar noise, which makes the virtual obstacle detection model based on the BEV front-center fusion architecture have low detection accuracy and poor detection result accuracy, which reduces the model detection performance when dealing with virtual obstacle problems. Summary of the invention
[0006] In view of this, the embodiments of the present application provide a virtual obstacle detection method, device and storage medium based on data enhancement to solve the problem that the virtual obstacle detection model in the prior art lacks noise training samples, resulting in low model detection accuracy, poor detection result accuracy and reduced model detection performance.
[0007] According to a first aspect of an embodiment of the present application, a virtual obstacle detection method based on data enhancement is provided, comprising: acquiring surround view image data from multiple perspectives around a vehicle, and using a laser radar to scan the area around the vehicle to obtain point cloud data; inputting the surround view image data and the point cloud data into a trained multimodal fusion perception model, wherein the multimodal fusion perception model is a virtual obstacle detection model obtained by training the point cloud data containing virtual obstacle noise after enhancing the point cloud data using a predetermined data enhancement method; in the multimodal fusion perception model, using a feature extraction module to extract features from the surround view image data and the point cloud data; converting the extracted features into a unified bird's-eye view space, and performing time series fusion on the features in the bird's-eye view space to obtain time-related multimodal features; inputting the multimodal features into a decoder related to a preset task, using the decoder to identify and filter virtual obstacles in the point cloud data, and outputting a detection result.
[0008] According to a second aspect of an embodiment of the present application, a virtual obstacle detection device based on data enhancement is provided, comprising: an acquisition module, configured to acquire surround view image data of multiple perspectives around a vehicle, and use a laser radar to scan the surroundings of the vehicle to obtain point cloud data; an input module, configured to input the surround view image data and the point cloud data into a trained multimodal fusion perception model, wherein the multimodal fusion perception model is a virtual obstacle detection model obtained by training the enhanced point cloud data containing virtual obstacle noise after enhancing the point cloud data using a predetermined data enhancement method; an extraction module, configured to extract features from the surround view image data and the point cloud data using a feature extraction module in the multimodal fusion perception model; a fusion module, configured to convert the extracted features into a unified bird's-eye view space, and perform time-series fusion on the features in the bird's-eye view space to obtain time-series-related multimodal features; a detection module, configured to input the multimodal features into a decoder related to a preset task, use the decoder to identify and filter virtual obstacles in the point cloud data, and output a detection result.
[0009] According to a third aspect of an embodiment of the present application, an electronic device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the steps of the above method are implemented when the processor executes the computer program.
[0010] According to a fourth aspect of an embodiment of the present application, a computer-readable storage medium is provided, which stores a computer program, and when the computer program is executed by a processor, the steps of the above method are implemented.
[0011] At least one of the above technical solutions adopted in the embodiments of the present application can achieve the following beneficial effects: By acquiring surround image data from multiple perspectives around the vehicle, and using a laser radar to scan the area around the vehicle to obtain point cloud data; inputting the surround image data and point cloud data into the trained multimodal fusion perception model, wherein the multimodal fusion perception model is a virtual obstacle detection model obtained by training the enhanced point cloud data containing virtual obstacle noise after enhancing the point cloud data using a predetermined data enhancement method; in the multimodal fusion perception model, a feature extraction module is used to extract features from the surround image data and point cloud data; the extracted features are converted into a unified bird's-eye view space, and the features in the bird's-eye view space are time-series fused to obtain time-series-related multimodal features; the multimodal features are input into a decoder related to a preset task, and the decoder is used to identify and filter virtual obstacles in the point cloud data, and output the detection results. This application can improve the accuracy of model detection, the accuracy of detection results and the performance of model detection. BRIEF DESCRIPTION OF THE DRAWINGS
[0012] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.
[0013] Figure 1 is a flow chart of a virtual obstacle detection method based on data enhancement provided in an embodiment of the present application; Figure 2 is a structural schematic diagram of a virtual obstacle detection device based on data enhancement provided in an embodiment of the present application; Figure 3 It is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0014] In the following description, specific details such as specific system structures, technologies, etc. are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present application. However, it should be clear to those skilled in the art that the present application may also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to prevent unnecessary details from obstructing the description of the present application.
[0015] Perception is an important part of the autonomous driving system. It refers to the modeling of the surrounding environment based on sensor data, such as giving the position / size / direction / speed of obstacles, and the position / type / length of lane lines. Most of the algorithms in the perception module are implemented by AI-dominated neural networks. The three most core elements for neural networks are data, algorithms, and computing power. Data occupies the most core position in the research and development of perception modules. Its quality directly determines the actual performance of the perception module. In the production environment of the industrial sector, data usually costs millions to billions of dollars for labeling. Even if a huge amount of labeling funds are invested, it is difficult to cover the long-tail problems of perception. Solving the last 1% of the long-tail problems may require the same cost as the other 99% of the problems. Therefore, how to reduce the cost of the long-tail problems of perception is an urgent problem to be solved.
[0016] The current mainstream perception architecture is BEV (Bird's Eye View), which has strong scalability and extensibility and is a data-driven architecture. However, the BEV architecture requires a large amount of data to train the model, thus facing a more serious data long-tail problem.
[0017] As one of the core sensors for perception, Lidar has excellent ranging advantages and mature obstacle detection algorithms. However, the noise in real scenes causes the Lidar detection model to give virtual obstacles that do not exist. These virtual obstacles can cause the vehicle to brake suddenly and cause serious traffic accidents. The industry has tried to introduce image semantic information to solve the problem of virtual obstacles, but the methods currently used have strict requirements on time synchronization between sensors, making these methods generally ineffective. In principle, the BEV multi-mode fusion solution can completely solve the problem of virtual obstacles, but in actual scenes, there is usually a lack of sufficient training samples containing Lidar noise, making it difficult for the BEV front-mid fusion architecture to completely solve the problem of virtual obstacles. This application intends to use data enhancement methods and solve the problem of virtual obstacles based on the BEV multi-modal front-mid fusion architecture.
[0018] The root cause of virtual obstacles is that there are indeed point clouds related to virtual objects in Lidar. For example, there will be echoes when Lidar scans water mist, and there may be secondary reflections when Lidar scans highly reflective metal objects. For a multimodal front-center fusion architecture such as BEV, the image and Lidar imaging of virtual objects are inconsistent. There are indeed point clouds of related objects in Lidar, but there are nothing in the image. During the training process, if there is a lack of data containing virtual obstacles, the images of almost all training samples are consistent with Lidar, and because Lidar is easier to train, the trained BEV model is more confident in the features of Lidar point clouds. When using a model with a BEV architecture on the vehicle side, if there is noise in the Lidar, the model will tend to give a virtual object regardless of whether the object actually exists in the real scene / image.
[0019] In view of the problems existing in the prior art, the present application provides a virtual obstacle detection method based on data enhancement. The present application uses a data enhancement method and solves the problem of Lidar virtual obstacles based on the BEV multimodal architecture. The present application can synthesize some data containing virtual obstacles, that is, the Lidar contains noise points of virtual obstacles but there is no imaging of the obstacle in the image. During the BEV model training process, because there is no label of this virtual obstacle in the label of the data, the model will not simply look at the features of the Lidar point cloud. Furthermore, the model needs to more deeply integrate the Lidar point cloud features and the image visual features to give more accurate results, that is, the model has the ability to identify virtual noise points. After training with these noisy data, when the model encounters noise points during reasoning on the vehicle side, it can automatically filter out the interference caused by these noise points and output the correct detection results.
[0020] The contents of the technical solution of the present application are described in detail below with reference to the accompanying drawings and specific embodiments.
[0021] Figure 1 It is a flow chart of a virtual obstacle detection method based on data enhancement provided in an embodiment of the present application. Figure 1 The data enhancement-based virtual obstacle detection method can be performed by the autonomous driving system. Figure 1 As shown, the virtual obstacle detection method based on data enhancement may specifically include: S101, acquiring surround image data from multiple perspectives around the vehicle, and using a laser radar to scan the area around the vehicle to obtain point cloud data; S102, inputting the surround image data and the point cloud data into the trained multimodal fusion perception model, wherein the multimodal fusion perception model is a virtual obstacle detection model obtained by training the point cloud data including virtual obstacle noise after enhancing the point cloud data using a predetermined data enhancement method; S103, in the multimodal fusion perception model, using a feature extraction module to extract features from the surround view image data and the point cloud data; S104, converting the extracted features into a unified bird's-eye view space, and performing temporal fusion on the features in the bird's-eye view space to obtain temporally related multimodal features; S105, inputting the multimodal features into a decoder related to a preset task, using the decoder to identify and filter virtual obstacles in the point cloud data, and outputting a detection result.
[0022] In some embodiments, the training process of the multimodal fusion perception model includes: Determine the model structure and model parameters of the pre-built multimodal fusion perception model; Obtain a pre-labeled training data set, wherein the training data set includes surround image data, point cloud data, and labeled detection boxes; Using a predetermined data enhancement method, the point cloud data in the training data set is enhanced to generate enhanced point cloud data containing virtual obstacle noise points; The enhanced point cloud data and the corresponding surround view image data are input into the multimodal fusion perception model, and the multimodal fusion perception model is trained to obtain a trained multimodal fusion perception model.
[0023] Specifically, the multimodal fusion perception model of the embodiment of the present application adopts a bird's-eye view (BEV) architecture model, which identifies and filters virtual obstacle noise by fusing lidar point cloud features and surround image visual features. First, in the model training stage, the model structure G and initial learning parameters θ of the pre-built multimodal fusion perception model are determined. The model structure G includes a multimodal feature extraction module, a feature conversion module, a temporal fusion module and a decoder.
[0024] Furthermore, a pre-labeled training data set is obtained, wherein the training data set includes surround view image data, point cloud data, and labeled detection boxes. The training data set should cover a variety of driving scenarios to ensure the robustness of the model in different environments.
[0025] Further, the point cloud data in the training data set is enhanced using a predetermined data enhancement method to generate enhanced point cloud data containing virtual obstacle noise. For example, a detection box is randomly selected from the annotated detection boxes in the training data set as a virtual obstacle, and a 3D position is randomly generated in the lidar space, and the virtual obstacle is offset according to the 3D position to generate enhanced point cloud data containing virtual obstacle noise.
[0026] Furthermore, the enhanced point cloud data and the corresponding surround image data are input into the multimodal fusion perception model to train the multimodal fusion perception model. Given the model structure G and the learning parameter θ, the main task of model training is to obtain the value of the learning parameter θ. Using the labeled training data {(x1, y1), (x2, y2), · · · , (x m ,y m )}, where x i is the input data of the model, including enhanced point cloud data and surround image data, y i is the corresponding manually annotated label. The gradient descent algorithm (SGD) is used to optimize the model weight θ (i.e., learning parameter), and the model parameters are adjusted by minimizing the loss function.
[0027] Furthermore, during the model training process, the gradient descent algorithm is repeatedly used to optimize the model parameters until the loss function converges to obtain the final trained multimodal fusion perception model. The performance of the trained multimodal fusion perception model is verified on a data set containing virtual obstacles to ensure that the model can identify and filter virtual obstacles.
[0028] In some embodiments, training the multimodal fusion perception model includes: Based on the enhanced point cloud data and the corresponding surround image data, the model parameters are optimized using the gradient descent algorithm, and the model parameters are adjusted by minimizing the loss function; During the model training process, the gradient descent algorithm is repeatedly used to optimize the model parameters until the loss function converges, and the final trained multimodal fusion perception model is obtained; The performance of the trained multimodal fusion perception model is verified on a data set containing virtual obstacles to verify whether the trained multimodal fusion perception model can identify and filter virtual obstacles.
[0029] Specifically, enhanced point cloud data and corresponding surround view image data are used as training data input, wherein the enhanced point cloud data includes data containing virtual obstacle noise points generated by a data enhancement method, and the surround view image data is image data from multiple perspectives around the vehicle.
[0030] Furthermore, the model parameters are optimized using the gradient descent algorithm based on the input enhanced point cloud data and surround image data. For example, the gradient descent algorithm (SGD) is used to optimize the model weights θ (i.e., learning parameters), and the model parameters are adjusted by minimizing the loss function. The loss function is defined to measure the difference between the model prediction results and the actual labels, and the performance of the model is improved by minimizing the difference.
[0031] Furthermore, during the model training process, the gradient descent algorithm is repeatedly used to perform the optimization operation of the model parameters. In each iteration, the gradient of the loss function is calculated, and the model parameters are updated to gradually reduce the loss function value. The training process continues until the loss function converges, that is, the preset convergence condition is reached or the maximum number of iterations is reached, and the final trained multimodal fusion perception model is obtained.
[0032] Furthermore, the performance of the trained multimodal fusion perception model is verified on data sets containing virtual obstacles. By verifying the performance of the model on these data sets, its ability to identify and filter virtual obstacles is evaluated. The verification process includes inputting test data into the trained model, analyzing the detection results output by the model, comparing them with the actual labels, and calculating performance indicators (such as accuracy, recall, etc.).
[0033] Through the method of the above embodiment, the process of training the multimodal fusion perception model includes data preparation, gradient descent algorithm optimization, loss function minimization, iterative optimization and performance verification. This process ensures that the trained multimodal fusion perception model can effectively identify and filter virtual obstacles, and improve the perception accuracy and safety of the autonomous driving system.
[0034] In some embodiments, the point cloud data in the training data set is enhanced using a predetermined data enhancement method to generate enhanced point cloud data containing virtual obstacle noise points, including: A detection box is randomly selected from the labeled detection boxes in the training data set as a virtual obstacle, and a 3D position is randomly generated in the lidar space. The virtual obstacle is offset according to the 3D position to generate enhanced point cloud data containing virtual obstacle noise.
[0035] Specifically, obtain the surround image data of n viewing angles {I1, I2, · · · , I n}, point cloud data P, and annotated detection box B. These data form the basis of the training data set.
[0036] Furthermore, a detection box B′ is randomly selected from the labeled detection boxes B in the training data set as a virtual obstacle. The selected detection box B′ contains the point cloud data detected by Lidar in the real scene. The point cloud data P′ contained in the detection box B′ is found from the point cloud data P. These point cloud data P′ correspond to the representation of the virtual obstacle in Lidar.
[0037] Furthermore, a 3D position Q is randomly generated in the Lidar space, and the 3D position Q is used to define the new position of the virtual obstacle in the enhanced point cloud data. The point cloud data P′ of the virtual obstacle is offset according to the randomly generated 3D position Q to generate enhanced point cloud data P′′ containing the noise of the virtual obstacle. The offset operation ensures that the point cloud data P′ of the virtual obstacle will not be completely blocked by other detection frames at the new position Q.
[0038] Furthermore, the offset point cloud data P′′ is updated to the original point cloud data P to obtain the final enhanced point cloud data. The enhanced point cloud data at this time contains randomly generated virtual obstacle noise points, which causes inconsistency between the Lidar point cloud data and the surround image data.
[0039] Through the method of the present embodiment described above, the point cloud data in the training data set is processed using a predetermined data enhancement method to generate enhanced point cloud data containing virtual obstacle noise points. When training a multimodal fusion perception model, this enhanced data can effectively improve the model's ability to identify and filter virtual obstacles, thereby improving the perception accuracy and safety of the autonomous driving system.
[0040] In some embodiments, the virtual obstacle is offset according to the 3D position to generate enhanced point cloud data containing virtual obstacle noise, including: Determine whether the detection frame of the virtual obstacle will not be completely blocked by other detection frames when placed at the 3D position; If the detection frame of the virtual obstacle is not completely blocked at the 3D position, the point cloud data contained in the detection frame is offset according to the 3D position to generate virtual obstacle noise; The offset point cloud data is updated to the original point cloud data of the training data set to obtain enhanced point cloud data containing virtual obstacle noise points; Returns the updated surround view image data, enhanced point cloud data containing virtual obstacle noise, and detection box.
[0041] Specifically, the surround image data {I1, I2, · · · , I n}, point cloud data P and a training data set of labeled detection boxes B. A detection box B′ is randomly selected from the labeled detection box B as a virtual obstacle. The detection box B′ contains the point cloud data P′ detected by Lidar.
[0042] Furthermore, a 3D position Q is randomly generated in the Lidar space, which is used to define the new position of the virtual obstacle in the enhanced point cloud data. It is determined whether the detection frame B′ of the virtual obstacle will not be completely blocked by other detection frames B when placed at the 3D position Q. If the detection frame B′ of the virtual obstacle will not be completely blocked at the 3D position Q, proceed to the next step.
[0043] Furthermore, the point cloud data P′ contained in the detection frame B′ is offset according to the 3D position Q to generate the noise point P′′ of the virtual obstacle. Specifically, when the detection frame B′ is placed at the position Q, the offset of the point cloud data P′ relative to the original position is calculated and converted to the new position Q. The offset point cloud data P′′ is updated to the original point cloud data P of the training data set to obtain the enhanced point cloud data P containing the noise point of the virtual obstacle.
[0044] Finally, the updated surround image data, enhanced point cloud data containing virtual obstacle noise, and updated detection box are returned.
[0045] Through the above method of this embodiment, this embodiment realizes offsetting the virtual obstacle according to the randomly generated 3D position, thereby generating enhanced point cloud data containing virtual obstacle noise. This method ensures that the virtual obstacle in the enhanced data will not be completely blocked by other objects, and can effectively improve its ability to identify and filter virtual obstacles when training the multimodal fusion perception model, thereby improving the perception accuracy and safety of the autonomous driving system.
[0046] In some embodiments, the extracted features are converted into a unified bird's-eye view space, and the features in the bird's-eye view space are temporally fused to obtain temporally related multimodal features, including: Mapping the surround image features and point cloud features extracted by the feature extraction module into a unified bird's-eye view space; Spatial alignment is performed on the surround image features and point cloud features mapped into the bird's-eye view space so that the multimodal features have a consistent spatial representation in the same coordinate system; In the bird's-eye view space, the multimodal features at different time points are accumulated and fused in chronological order to form a temporal feature sequence; Perform time series analysis on the time series feature sequence and extract time-related multimodal features to capture dynamic change information.
[0047] Specifically, this embodiment extracts surround view image features and point cloud features through various feature extraction modules. The surround view image features include image features of multiple perspectives around the vehicle, and the point cloud features include environmental point cloud data collected by the laser radar.
[0048] Furthermore, the extracted surround image features and point cloud features are mapped into a unified bird's-eye view (BEV) space. During the mapping process, considering the data type and resolution differences of each sensor, an appropriate conversion algorithm is used to unify the multimodal features into a standard coordinate system.
[0049] Furthermore, the surround image features and point cloud features mapped to the BEV space are spatially aligned so that the multimodal features have a consistent spatial representation in the same coordinate system. The spatial alignment step ensures the correspondence between the data of different sensors in spatial position, laying the foundation for subsequent time series fusion.
[0050] Furthermore, in the BEV space, the multimodal features at different time points are accumulated and fused in chronological order to form a temporal feature sequence. Specifically, the surround image features and point cloud features at multiple consecutive time points are superimposed to generate multimodal features with a time dimension.
[0051] Furthermore, the time series feature sequence is subjected to time series analysis to extract time-related multimodal features to capture dynamic change information. The time series analysis step includes applying a time series model (such as LSTM, GRU, etc.) to process the accumulated multimodal features and extract features that can reflect the dynamic changes of the environment.
[0052] Finally, time-related multimodal features are obtained, which can reflect the temporal changes of the vehicle's surroundings and provide accurate input data for subsequent perception tasks.
[0053] Through the method of the present embodiment described above, the surround image features and point cloud features are converted into a unified bird's-eye view space, and time-related multimodal features are obtained through time fusion. This process ensures the consistency of multimodal features in space and time, and improves the accuracy and robustness of the perception system in a dynamic environment.
[0054] In some embodiments, the multimodal features are input into a decoder associated with a preset task, and the decoder is used to identify and filter virtual obstacles in the point cloud data, and output detection results, including: Input the time-series-related multimodal features into the decoder, use the decoder to process the multimodal features, and extract feature information related to virtual obstacle detection; The decoder is used to analyze and identify the feature information to determine the virtual obstacles existing in the point cloud data; The identified virtual obstacles are filtered to remove noise in the point cloud data, and the point cloud data after the virtual obstacles are filtered and the position information of the virtual obstacles are output.
[0055] Specifically, the time-related multimodal features obtained by time fusion are input into the decoder. The multimodal features include time-series fusion features from surround view images and Lidar point clouds, which can reflect the dynamic change information of the vehicle's surrounding environment.
[0056] Furthermore, the decoder is used to process the input multimodal features and extract feature information related to virtual obstacle detection. The decoder includes multiple task-related modules for different perception tasks, such as 3D object detection, lane line recognition, and occupancy grid recognition.
[0057] Furthermore, the decoder is used to analyze and identify the feature information to determine the virtual obstacles in the point cloud data. The decoder identifies the location and shape of the virtual obstacles by analyzing the abnormal point cloud features and image features in the multimodal features.
[0058] Furthermore, the identified virtual obstacles are filtered to remove noise in the point cloud data. Specifically, by comparing the features of the virtual obstacles with the reasonable features in the actual scene, the point cloud data corresponding to the virtual obstacles are filtered out, thereby reducing misleading the perception system.
[0059] Furthermore, the point cloud data after the virtual obstacle filtering and the position information of the virtual obstacle are output. The output result includes: the point cloud data after the virtual obstacle filtering, the noise of the virtual obstacle is removed, and the position information of the virtual obstacle is used for further processing and analysis.
[0060] Through the method of the present embodiment described above, the time-series-related multimodal features are input into the decoder, which realizes the recognition and filtering of virtual obstacles in the point cloud data, and improves the accuracy and robustness of the perception system. The decoder can flexibly process multimodal features according to different preset tasks, complete tasks such as 3D target detection, lane line recognition, and occupied grid recognition, thereby improving the perception ability of the autonomous driving system in complex environments.
[0061] According to the technical solution provided in the embodiments of the present application, the technical solution of the present application has at least the following advantages: The technical solution of this application generates enhanced point cloud data containing virtual obstacles through a predetermined data enhancement method, so that the BEV model no longer relies solely on the Lidar point cloud features during the training process. Instead, the model needs to more deeply integrate the visual features of the Lidar point cloud and the surround image to have the ability to identify virtual noise points.
[0062] Introducing training data containing virtual obstacles during the training process of the BEV model enables the model to effectively identify and filter out interference caused by Lidar noise in practical applications. This improvement enables the model to output more accurate detection results and reduces the misleading of the autonomous driving system by virtual obstacles.
[0063] By synthesizing training data containing virtual obstacles, the technical solution of this application effectively improves the performance of the autonomous driving perception system in dealing with virtual obstacle problems. Using these enhanced data, the perception system can more reliably identify and filter virtual obstacles in complex environments, improving the accuracy and robustness of overall perception.
[0064] The multimodal fusion perception model achieves more accurate virtual obstacle detection by fusing the lidar point cloud features and the surround image visual features. By comprehensively analyzing the data from multiple sensors, the model can more accurately identify virtual obstacles, thereby providing more reliable perception results.
[0065] Through the above improvements, the technical solution of this application significantly improves the safety of the autonomous driving system. It effectively identifies and filters virtual obstacle noise points, reduces the risk of emergency braking and potential traffic accidents caused by false obstacles, and ensures the stable operation of autonomous driving vehicles in complex environments.
[0066] In summary, the technical solution of the present application significantly improves the detection accuracy and robustness of the autonomous driving perception system by introducing training data containing virtual obstacles and optimizing the training process of the multimodal fusion perception model, thereby enhancing the performance and safety of the system in practical applications.
[0067] The following is an embodiment of the device of the present application at the bottom of the window, which can be used to execute the embodiment of the method of the present application. For details not disclosed in the embodiment of the device of the present application, please refer to the embodiment of the method of the present application. Figure 2 is a schematic diagram of the structure of a virtual obstacle detection device based on data enhancement provided in an embodiment of the present application. Figure 2 As shown, the virtual obstacle detection device based on data enhancement includes: The acquisition module 201 is configured to acquire surround view image data from multiple perspectives around the vehicle, and use a laser radar to scan the area around the vehicle to obtain point cloud data; The input module 202 is configured to input the surround image data and the point cloud data into the trained multimodal fusion perception model, wherein the multimodal fusion perception model is a virtual obstacle detection model obtained by training the enhanced point cloud data containing virtual obstacle noise after enhancing the point cloud data using a predetermined data enhancement method; The extraction module 203 is configured to extract features from the surround image data and the point cloud data using the feature extraction module in the multimodal fusion perception model; A fusion module 204 is configured to convert the extracted features into a unified bird's-eye view space, and perform temporal fusion on the features in the bird's-eye view space to obtain temporally related multimodal features; The detection module 205 is configured to input the multimodal features into a decoder associated with a preset task, use the decoder to identify and filter virtual obstacles in the point cloud data, and output a detection result.
[0068] In some embodiments, Figure 2 The training module 206 determines the model structure and model parameters of the pre-built multimodal fusion perception model; obtains a pre-labeled training data set, wherein the training data set includes surround view image data, point cloud data and labeled detection frames; uses a predetermined data enhancement method to enhance the point cloud data in the training data set to generate enhanced point cloud data including virtual obstacle noise; inputs the enhanced point cloud data and the corresponding surround view image data into the multimodal fusion perception model, trains the multimodal fusion perception model, and obtains the trained multimodal fusion perception model.
[0069] In some embodiments, Figure 2 The training module 206 optimizes the model parameters using the gradient descent algorithm according to the enhanced point cloud data and the corresponding surround image data, and adjusts the model parameters by minimizing the loss function; during the model training process, the gradient descent algorithm is repeatedly used to perform the optimization operation of the model parameters until the loss function converges to obtain the finally trained multimodal fusion perception model; the performance of the trained multimodal fusion perception model on the data set containing virtual obstacles is verified to verify whether the trained multimodal fusion perception model can identify and filter virtual obstacles.
[0070] In some embodiments, Figure 2 The training module 206 randomly selects a detection box from the labeled detection boxes in the training data set as a virtual obstacle, and randomly generates a 3D position in the lidar space, offsets the virtual obstacle according to the 3D position, and generates enhanced point cloud data containing virtual obstacle noise points.
[0071] In some embodiments, Figure 2The training module 206 determines whether the detection frame of the virtual obstacle will not be completely blocked by other detection frames when it is placed at the 3D position; if the detection frame of the virtual obstacle will not be completely blocked at the 3D position, the point cloud data contained in the detection frame is offset according to the 3D position to generate virtual obstacle noise; the offset point cloud data is updated to the original point cloud data of the training data set to obtain enhanced point cloud data containing virtual obstacle noise; and the updated surround image data, the enhanced point cloud data containing virtual obstacle noise and the detection frame are returned.
[0072] In some embodiments, Figure 2 The fusion module 204 maps the surround image features and point cloud features extracted by the feature extraction module into a unified bird's-eye view space; spatially aligns the surround image features and point cloud features mapped into the bird's-eye view space so that the multimodal features have a consistent spatial representation in the same coordinate system; in the bird's-eye view space, accumulates and fuses the multimodal features at different time points in chronological order to form a time series feature sequence; performs time series analysis on the time series feature sequence to extract time-related multimodal features to capture dynamic change information.
[0073] In some embodiments, Figure 2 The detection module 205 inputs the time-series-related multimodal features into the decoder, processes the multimodal features using the decoder, and extracts feature information related to virtual obstacle detection; uses the decoder to analyze and identify the feature information to determine the virtual obstacles present in the point cloud data; filters the identified virtual obstacles to remove noise in the point cloud data, and outputs the point cloud data after filtering the virtual obstacles and the location information of the virtual obstacles.
[0074] It should be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.
[0075] Figure 3 Schematic diagram of the structure of the electronic device 3 provided in the embodiment of the present application. Figure 3 As shown, the electronic device 3 of this embodiment includes: a processor 301, a memory 302, and a computer program 303 stored in the memory 302 and executable on the processor 301. When the processor 301 executes the computer program 303, the steps in the above-mentioned method embodiments are implemented. Alternatively, when the processor 301 executes the computer program 303, the functions of the modules / units in the above-mentioned device embodiments are implemented.
[0076] Exemplarily, the computer program 303 may be divided into one or more modules / units, which are stored in the memory 302 and executed by the processor 301 to complete the present application. The one or more modules / units may be a series of computer program instruction segments capable of completing specific functions, which are used to describe the execution process of the computer program 303 in the electronic device 3.
[0077] The electronic device 3 may be a desktop computer, a notebook, a PDA, a cloud server, or other electronic device. The electronic device 3 may include, but is not limited to, a processor 301 and a memory 302. Those skilled in the art will appreciate that Figure 3 It is only an example of the electronic device 3 and does not constitute a limitation of the electronic device 3. It may include more or fewer components than shown in the figure, or a combination of certain components, or different components. For example, the electronic device may also include input and output devices, network access devices, buses, etc.
[0078] The processor 301 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or the processor may be any conventional processor, etc.
[0079] The memory 302 may be an internal storage unit of the electronic device 3, for example, a hard disk or memory of the electronic device 3. The memory 302 may also be an external storage device of the electronic device 3, for example, a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the electronic device 3. Further, the memory 302 may also include both an internal storage unit and an external storage device of the electronic device 3. The memory 302 is used to store computer programs and other programs and data required by the electronic device. The memory 302 may also be used to temporarily store data that has been output or is to be output.
[0080] The technicians in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In practical applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated in a processing unit, or each unit can exist physically separately, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here.
[0081] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.
[0082] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.
[0083] In the embodiments provided in the present application, it should be understood that the disclosed devices / computer equipment and methods can be implemented in other ways. For example, the device / computer equipment embodiments described above are only schematic. For example, the division of modules or units is only a logical function division. There may be other division methods in actual implementation. Multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.
[0084] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
[0085] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.
[0086] If the integrated module / unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. The computer program may include computer program code, which may be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may include: any entity or device capable of carrying computer program code, recording medium, U disk, mobile hard disk, disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium, etc.
[0087] The above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.
Claims
1. A virtual obstacle detection method based on data enhancement, characterized in that: include: Obtain surround image data from multiple perspectives around the vehicle, and use lidar to scan the area around the vehicle to obtain point cloud data; Inputting the surround image data and the point cloud data into a trained multimodal fusion perception model, wherein the multimodal fusion perception model is a virtual obstacle detection model obtained by training the point cloud data containing virtual obstacle noise after enhancing the point cloud data using a predetermined data enhancement method; In the multimodal fusion perception model, a feature extraction module is used to extract features from the surround view image data and the point cloud data; The extracted features are converted into a unified bird's-eye view space, and the features in the bird's-eye view space are temporally fused to obtain time-related multimodal features; The multimodal features are input into a decoder associated with a preset task, the decoder is used to identify and filter virtual obstacles in the point cloud data, and a detection result is output.
2. The method according to claim 1, characterized in that The training process of the multimodal fusion perception model includes: Determining the model structure and model parameters of the pre-built multimodal fusion perception model; Obtain a pre-labeled training data set, wherein the training data set includes surround image data, point cloud data, and a labeled detection frame; Using a predetermined data enhancement method, the point cloud data in the training data set is enhanced to generate enhanced point cloud data containing virtual obstacle noise points; The enhanced point cloud data and the corresponding surround image data are input into the multimodal fusion perception model, and the multimodal fusion perception model is trained to obtain a trained multimodal fusion perception model.
3. The method according to claim 2, characterized in that The training of the multimodal fusion perception model includes: According to the enhanced point cloud data and the corresponding surround image data, the model parameters are optimized using a gradient descent algorithm, and the model parameters are adjusted by minimizing a loss function; During the model training process, the gradient descent algorithm is repeatedly used to optimize the model parameters until the loss function converges, and the final trained multimodal fusion perception model is obtained; The performance of the trained multimodal fusion perception model is verified on a data set containing virtual obstacles to verify whether the trained multimodal fusion perception model can identify and filter virtual obstacles.
4. The method according to claim 2, characterized in that: The method of enhancing the point cloud data in the training data set by using a predetermined data enhancement method to generate enhanced point cloud data containing virtual obstacle noise points includes: A detection box is randomly selected from the labeled detection boxes in the training data set as a virtual obstacle, and a 3D position is randomly generated in the lidar space. The virtual obstacle is offset according to the 3D position to generate enhanced point cloud data containing virtual obstacle noise points.
5. The method according to claim 4, characterized in that The step of offsetting the virtual obstacle according to the 3D position to generate enhanced point cloud data containing virtual obstacle noise includes: Determining whether the detection frame of the virtual obstacle will not be completely blocked by other detection frames when placed at the 3D position; If the detection frame of the virtual obstacle is not completely blocked at the 3D position, offsetting the point cloud data contained in the detection frame according to the 3D position to generate virtual obstacle noise; Updating the offset point cloud data to the original point cloud data of the training data set to obtain enhanced point cloud data containing virtual obstacle noise points; Returns the updated surround view image data, enhanced point cloud data containing virtual obstacle noise, and detection box.
6. The method according to claim 1, characterized in that The extracted features are converted into a unified bird's-eye view space, and the features in the bird's-eye view space are temporally fused to obtain time-related multimodal features, including: Mapping the surround image features and point cloud features extracted by the feature extraction module into a unified bird's-eye view space; Spatially aligning the surround image features and the point cloud features mapped into the bird's-eye view space so that the multimodal features have a consistent spatial representation in the same coordinate system; In the bird's-eye view space, multimodal features at different time points are accumulated and fused in chronological order to form a temporal feature sequence; A time series analysis is performed on the time series feature sequence to extract time-related multimodal features to capture dynamic change information.
7. The method according to claim 1, characterized in that The step of inputting the multimodal features into a decoder associated with a preset task, using the decoder to identify and filter virtual obstacles in the point cloud data, and outputting a detection result includes: Inputting the time-series-related multimodal features into the decoder, and using the decoder to process the multimodal features to extract feature information related to virtual obstacle detection; Analyzing and identifying the feature information using the decoder to determine virtual obstacles present in the point cloud data; The identified virtual obstacles are filtered to remove noise in the point cloud data, and the point cloud data after the virtual obstacles are filtered and the position information of the virtual obstacles are output.
8. A virtual obstacle detection device based on data enhancement, characterized in that: include: An acquisition module is configured to acquire surround image data from multiple perspectives around the vehicle and use a laser radar to scan the area around the vehicle to obtain point cloud data; An input module is configured to input the surround image data and the point cloud data into a trained multimodal fusion perception model, wherein the multimodal fusion perception model is a virtual obstacle detection model obtained by training the enhanced point cloud data containing virtual obstacle noise after enhancing the point cloud data using a predetermined data enhancement method; an extraction module, configured to perform feature extraction on the surround view image data and the point cloud data using a feature extraction module in the multimodal fusion perception model; A fusion module is configured to convert the extracted features into a unified bird's-eye view space, and perform temporal fusion on the features in the bird's-eye view space to obtain temporally related multimodal features; The detection module is configured to input the multimodal features into a decoder associated with a preset task, use the decoder to identify and filter virtual obstacles in the point cloud data, and output a detection result.
9. An electronic device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.
10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.
Citation Information
Patent Citations
Multi-sensor information fusion method of automatic driving system
CN116051818A
Suspension ghosting point cloud removal method, automatic driving method and related equipment
CN116386001A
Cited By
Height detection method and device, vehicle detection method and device, vehicle and storage medium
CN121214393A
