Data screening method, device, system, electronic equipment and storage medium

Abstract

The application provides a data screening method, device and system, electronic equipment and storage medium, and relates to the technical field of automatic driving. The data screening method can comprise: acquiring initial screening incremental data; wherein the initial screening incremental data is obtained based on at least one of prediction data and decision data of an automatic driving vehicle and perception data of the automatic driving vehicle; and performing secondary screening on the initial screening incremental data through a first preset screening mode to obtain final incremental data; wherein the final incremental data at least comprises at least one of abnormal data and dangerous driving scene data. The technical scheme provided by the application can enrich valuable data screened.

Description

Technical Field

[0001] This application relates to the field of autonomous driving technology, specifically to a data filtering method, apparatus, system, electronic device, and storage medium. Background Technology

[0002] Autonomous driving is a long-tail problem with a large number of rare but critical safety scenarios. In order for autonomous driving algorithm models to better cope with such scenarios, data from these scenarios is urgently needed for model training.

[0003] In existing technologies, autonomous vehicles typically perform this type of data filtering on the driving data they collect themselves. However, on-board data filtering is generally based on data collected by various sensors, which may miss some valuable data and affect the optimization of model performance. Summary of the Invention

[0004] Based on the deficiencies and shortcomings of the prior art, this application proposes a data filtering method, apparatus, system, electronic device, and storage medium that can enrich the valuable data obtained through filtering.

[0005] According to a first aspect of the embodiments of this application, a data filtering method is provided, implemented by a server, the method comprising:

[0006] Acquire initial screening incremental data; wherein the initial screening incremental data is obtained based on at least one of the prediction data and decision data of the autonomous vehicle, as well as the perception data of the autonomous vehicle.

[0007] The initial incremental data is further filtered using a first preset filtering method to obtain the final incremental data; wherein the final incremental data includes at least one of abnormal data and dangerous driving scenario data.

[0008] According to a second aspect of the embodiments of this application, a data filtering method is provided, implemented by an autonomous vehicle, the method comprising:

[0009] Collect vehicle driving data;

[0010] The vehicle driving data is filtered using a second preset filtering method to obtain initial incremental data. The initial incremental data is obtained by filtering based on at least one of the prediction data and decision data of the autonomous vehicle, as well as the perception data of the autonomous vehicle. The initial incremental data is sent to the server, which then performs a second filtering on it using a first preset filtering method to obtain final incremental data. The final incremental data includes at least one of abnormal data and dangerous driving scenario data.

[0011] According to a third aspect of the embodiments of this application, a data filtering device is provided, deployed on a server, the device comprising:

[0012] The data acquisition module is used to acquire preliminary screening incremental data; wherein the preliminary screening incremental data is obtained by filtering based on at least one of the prediction data and decision data of the autonomous vehicle, as well as the perception data of the autonomous vehicle.

[0013] The first data filtering module is used to perform secondary filtering on the initial incremental data obtained by the data acquisition module through a first preset filtering method to obtain the final incremental data; wherein the final incremental data includes at least one of abnormal data and dangerous driving scenario data.

[0014] According to a fourth aspect of the embodiments of this application, a data filtering device is provided, deployed in an autonomous vehicle, the device comprising:

[0015] The data acquisition module is used to collect vehicle driving data;

[0016] The second data filtering module is used to filter the vehicle driving data through a second preset filtering method to obtain preliminary incremental data. The preliminary incremental data is obtained by filtering based on at least one of the prediction data and decision data of the autonomous vehicle, as well as the perception data of the autonomous vehicle. The preliminary incremental data is sent to the server, so that the server performs a second filtering on it through a first preset filtering method to obtain final incremental data. The final incremental data includes at least one of abnormal data and dangerous driving scenario data.

[0017] According to a fifth aspect of the embodiments of this application, a data filtering system is provided, including: an autonomous vehicle and a server;

[0018] The autonomous vehicle is used to filter the vehicle driving data it collects through a second preset filtering method to obtain preliminary incremental data; wherein the preliminary incremental data is obtained by filtering based on at least one of the autonomous vehicle's prediction data and decision data, as well as the autonomous vehicle's perception data.

[0019] The server is used to perform a second screening of the initial incremental data obtained from the screening of the autonomous vehicles using a first preset screening method to obtain the final incremental data; wherein the final incremental data includes at least one of abnormal data and dangerous driving scenario data.

[0020] According to a sixth aspect of the embodiments of this application, an electronic device is provided, including: a memory and a processor;

[0021] The memory is connected to the processor and is used to store programs;

[0022] The processor is used to implement the data filtering method as described in the first or second aspect by running a program in the memory.

[0023] According to a seventh aspect of the present application, a storage medium is provided, on which a computer program is stored, and when the computer program is run by a processor, it implements the data filtering method as described in the first or second aspect.

[0024] In this embodiment, data filtering is performed not only based on vehicle perception data but also on at least one of vehicle prediction data and decision data. This allows for data filtering at different levels, enriching the incremental data obtained and reducing the omission of valuable data. Furthermore, in this implementation, in addition to the incremental data filtering at the vehicle end, the server performs a secondary filtering of the incremental data filtered at the vehicle end. Because the server has stronger computing power, it can run more complex data filtering algorithms to perform more precise data filtering, thereby improving the accuracy of data filtering and reducing the workload of subsequent data annotation and model training. Attached Figure Description

[0025] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of this application. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0026] Figure 1 This is a schematic diagram of a data closed-loop system in the prior art.

[0027] Figure 2 This is a flowchart illustrating a data filtering method implemented via a server, as provided in an embodiment of this application.

[0028] Figure 3 A block diagram illustrating the server-side data filtering mechanism provided in this application embodiment.

[0029] Figure 4 A schematic diagram of the hardware framework of the vehicle-side data acquisition system provided in this application embodiment.

[0030] Figure 5 This is a block diagram of the architecture of a single-class classifier provided in an embodiment of this application.

[0031] Figure 6 A block diagram of the sensing module provided in an embodiment of this application.

[0032] Figure 7 Another block diagram of the sensing module provided in an embodiment of this application.

[0033] Figure 8 A block diagram of the perception module and prediction module provided in the embodiments of this application.

[0034] Figure 9 A block diagram of the perception module, prediction module, and planning module provided in the embodiments of this application.

[0035] Figure 10 This is a flowchart illustrating a data filtering method implemented using an autonomous vehicle, as provided in an embodiment of this application.

[0036] Figure 11 A block diagram illustrating the vehicle-side data filtering mechanism provided in this application embodiment.

[0037] Figure 12 A block diagram illustrating the construction of the scenario library provided in the embodiments of this application.

[0038] Figure 13 This is a block diagram of a data filtering device deployed on a server, as provided in an embodiment of this application.

[0039] Figure 14 This is a block diagram of a data filtering device deployed in an autonomous vehicle, as provided in an embodiment of this application.

[0040] Figure 15 A schematic diagram of a data closed-loop system provided in an embodiment of this application.

[0041] Figure 16 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0042] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0043] Application Overview

[0044] The safety of autonomous vehicles is a major issue that urgently needs to be addressed. As perception technologies and autonomous driving algorithm models mature, the key factors affecting vehicle safety have shifted from common, general cases to various uncommon but constantly emerging special cases, such as driving scenarios in extreme situations.

[0045] In related technologies, the above problems are generally solved through a data closed-loop approach. Specifically, the data closed-loop framework for autonomous driving can be as follows: Figure 1 As shown, by repeatedly collecting data, training the model, and deploying the model, the model is continuously optimized and its processing capabilities are improved.

[0046] In this process, data collection and filtering are performed by the vehicle. However, since the occurrence of the aforementioned special scenarios is unpredictable and may be fleeting, the vehicle needs to process the data promptly and perform data filtering. Therefore, the data filtering algorithm deployed on the vehicle is lightweight and relatively simple to quickly complete the data filtering in a short time. However, this also means that the vehicle can only perform coarse filtering, and the accuracy of data filtering needs to be improved. In addition, the vehicle generally performs data filtering based on data collected by various sensors, which may miss some valuable data and affect the optimization of model performance. Therefore, this application provides a technical solution for data filtering to solve the above problems.

[0047] Exemplary application scenarios

[0048] The technical solutions provided in this application can be applied to data screening during the vehicle R&D stage, as well as during the vehicle mass production stage.

[0049] Exemplary methods

[0050] This application provides a data filtering method that can be implemented by a server, i.e., the server executes the method.

[0051] like Figure 2 As shown, the data filtering method may include:

[0052] Step 201: The server obtains the initial screening incremental data.

[0053] The aforementioned initial incremental data is obtained by autonomous vehicles filtering the driving data they collect themselves.

[0054] Specifically, the initial incremental data can be obtained by filtering based on at least one of the prediction data and decision data of autonomous vehicles, as well as the perception data of autonomous vehicles.

[0055] The aforementioned perception data may include, but is not limited to: data collected by various sensors on autonomous vehicles, and perception results output by onboard autonomous driving algorithm models.

[0056] The aforementioned prediction data may include, but is not limited to, trajectory prediction results output for surrounding obstacles through in-vehicle autonomous driving algorithm models or other means.

[0057] The aforementioned decision data may include, but is not limited to: data on the decision-making behavior of vehicle drivers (such as safety drivers during the vehicle development phase) and decision results output by onboard autonomous driving algorithms for the vehicle's driving trajectory.

[0058] In autonomous driving technologies, perception is fundamental. Therefore, data filtering based on perception data is essential. The applicant has discovered that vehicle prediction and decision-making data can also demonstrate data value to a certain extent. For example, in driving scenarios including mass protests, if an autonomous driving algorithm model fails to predict the trajectories of obstacles (i.e., numerous pedestrians), or if the vehicle trajectory in the decision result shows a high risk, it indicates that the model lacks the ability to handle such driving scenarios. In this case, the data from such driving scenarios is valuable data, i.e., incremental data. Based on this discovery, this application proposes data filtering not only based on vehicle perception data but also on at least one of vehicle prediction and decision-making data. This allows for data filtering at different levels, enriching the incremental data obtained through filtering and reducing the omission of valuable data.

[0059] The incremental data mentioned in this application embodiment can be understood as missing data in the training data of the autonomous driving algorithm model, or as abnormal data (such as abnormal perception data such as incorrect target matching in different cameras, target jitter in consecutive frames, or sudden disappearance of the target) or dangerous driving scenario data (such as scenario data of driving on a highway in bad weather). The aforementioned initial screening incremental data may include at least one set of data, each set of data corresponding to a driving scenario.

[0060] like Figure 3 As shown, vehicle 301 can send the filtered data to server 302 in real time or at regular intervals. Server 302 can save the received data, for example, by saving the data to a temporary storage hard disk 3021 for later use.

[0061] The aforementioned vehicle driving data can be the autonomous vehicle's own driving data collected, or it can be obtained by the autonomous vehicle from other autonomous vehicles. For example, after vehicle A and vehicle B form a network, vehicle A can send its own driving data to vehicle B even if the communication connection between vehicle A and the server is disconnected, and vehicle B can then filter vehicle A's driving data.

[0062] The aforementioned vehicle driving data may include, but is not limited to, data collected by various sensors on the vehicle, such as data collected by at least one of cameras, lidar, millimeter-wave radar, Global Navigation Satellite System (GNSS), Inertial Measurement Unit (IMU), and other onboard sensors.

[0063] like Figure 4 As shown, the cameras on the vehicle may include, but are not limited to: front camera 401, left front camera 402, right front camera 403, rear camera 404, left rear camera 405, and right rear camera 406. The positional relationship of each camera on the vehicle can be found by reference. Figure 4 (The black circles in the image represent cameras), but not limited to... Figure 4 As shown, these cameras can capture environmental images within a 360-degree radius around the vehicle.

[0064] like Figure 4 As shown, the LiDAR 407 on the vehicle can be installed at the rear of the vehicle. It can perform 360-degree scanning to obtain 3D point cloud data. The installation location of the LiDAR 407 on the vehicle can be found in [reference needed]. Figure 4 (The circular shape filled with diagonal lines in the image), but not limited to... Figure 4 As shown. It is understandable that this lidar can also be replaced by a combination of multiple lidars capable of performing limited-view scanning.

[0065] like Figure 4 As shown, the millimeter-wave radar on the vehicle may include, but is not limited to: forward radar 408, left front radar 409, right front radar 410, left rear radar 411, and right rear radar 412. The positional relationship of each millimeter-wave radar on the vehicle can be referenced. Figure 4 (Circular shapes filled with grid in the image), but not limited to Figure 4 As shown. These millimeter-wave radars and lidars can detect obstacles within a 360-degree range around the vehicle. When the vehicle is not equipped with lidar, a rear-facing radar 413 can be installed at the rear of the vehicle.

[0066] like Figure 4 As shown, the IMU 414 on a vehicle can be installed at the rear of the vehicle, enabling more reliable vehicle navigation without relying on external information. The installation location of the IMU 414 on the vehicle can be found in [reference needed]. Figure 4 (The circular shape filled with black dots in the image), but not limited to... Figure 4 As shown.

[0067] The vehicle driving data mentioned above is collected by multiple similar sensors (such as multiple cameras or multiple millimeter-wave radars), or it is multimodal data collected by multiple sensors. When the vehicle driving data is multimodal, sensor calibration is required. Specifically, calibration can be performed by establishing a three-dimensional coordinate system with a point on the vehicle as the origin, and then calibrating the coordinate position of each sensor in the three-dimensional coordinate system. This coordinate position can be used to determine the coordinate relationships between the sensor data, facilitating feature matching or feature merging operations.

[0068] In addition, a unified clock needs to be used among the sensors, and then a certain signal is used to trigger the sensors to work. For example, in order to facilitate feature matching or feature merging between point cloud data collected by LiDAR and images collected by multiple cameras, the correspondence between the orientation of the LiDAR and each camera can be preset. Then, the camera that matches the LiDAR is determined according to its orientation, and the camera is controlled to collect images.

[0069] Step 202: The server performs a second screening of the initial incremental data using the first preset screening method to obtain the final incremental data.

[0070] The aforementioned final incremental data includes at least one of the following: abnormal data and dangerous driving scenario data. The concepts of abnormal data and dangerous driving scenario data have already been explained above and will not be repeated here.

[0071] After the server receives the initial screening incremental data sent by the autonomous vehicle, for example... Figure 3 After obtaining the initial incremental data from the temporary storage hard drive 3021, the initial incremental data can be further filtered using a preset filtering algorithm (i.e., the first preset filtering method) to obtain the final incremental data. This final incremental data can be stored in the database 3028 for purposes such as building a scene library or training models. Because the server has stronger computing power and does not need to filter the data collected from vehicles in real time, it can run more complex data filtering algorithms to perform more precise data filtering, thereby improving the accuracy of data filtering and reducing subsequent useless data annotation and useless model training work.

[0072] As an optional embodiment, the number of the above-mentioned first preset filtering methods is at least one, and may include at least one of the following: first filtering method, second filtering method, and third filtering method.

[0073] The first screening method is the autonomous driving algorithm screening method, which involves inputting the initial screening incremental data into the autonomous driving algorithm model on the server side, and then screening the data based on the model output results and the single-class classifier.

[0074] The single-class classifier described here, also known as a one-class classifier, means that it can only identify one class of data. The output of a single-class classifier is either "belongs to the target class" or "does not belong to the target class," where the target class is the category the single-class classifier can recognize. For example, a single-class classifier can be trained to recognize objects like pens. To do this, image features of various pens can be collected beforehand and used as training samples for the single-class classifier. After training, when the image features of an eraser are input into the single-class classifier, its output will be "the object is not a pen"; when the image features of a pen are input into the single-class classifier, its output will be "the object is a pen."

[0075] The single-class classifier in this embodiment can be trained using target training data, which is the output result obtained after inputting historical training data of the autonomous driving algorithm model into the model. When using this single-class classifier for data filtering, the model output result can be input into the single-class classifier, which then classifies the output result. If the output result belongs to the target class, it is determined that the data to be filtered is not the required data, that is, the data to be filtered is not incremental data; if the output result does not belong to the target class, it is determined that the data to be filtered is the required data, that is, the data to be filtered is incremental data.

[0076] Optionally, the single-class classifier architecture in the embodiments of this application can be as follows: Figure 5 As shown, it includes: a feature encoding module 501, a feature decoding module 502, and a classifier 503 based on convolutional neural networks (CNN).

[0077] In this embodiment, the single-class classifier is trained based on the historical training data of the autonomous driving algorithm model, which helps to better distinguish between incremental and non-incremental data and provides a powerful tool for understanding vehicle driving data. Therefore, incremental data can be effectively filtered through the first filtering method.

[0078] The second screening method is clustering screening, which involves screening data based on the clustering results of the initial incremental data and at least one preset scenario cluster.

[0079] The preset scene clusters mentioned here are obtained by clustering historical driving scene data and represent known driving scenes. Optionally, unsupervised learning methods or density estimation methods can be used to generate scene clusters.

[0080] Since the preset scenario clusters are obtained by clustering historical driving scenario data and represent known driving scenarios, the relationship between the data to be filtered and the preset scenario clusters can be used to determine whether the data to be filtered is incremental data. If the data to be filtered belongs to one of the preset scenario clusters, it is determined that the data to be filtered is not incremental data; if the data to be filtered does not belong to any preset scenario cluster, it is determined that the data to be filtered is incremental data. For example, the preset scenario clusters include: rainy driving scenario and foggy driving scenario. When the data to be filtered corresponds to the extreme weather driving scenario of a typhoon, the data to be filtered does not belong to any preset scenario cluster, so the data to be filtered is incremental data.

[0081] In the second filtering method, the preset scene clusters are pre-generated. By analyzing and judging the relationship between the data to be filtered and the preset scene clusters, it can be determined whether the data to be filtered is incremental data. The algorithm is simple, can quickly obtain filtering results, and has high filtering efficiency.

[0082] The third filtering method is a search filtering method, which involves filtering data by finding data that matches the preset scenario information in the initial incremental data.

[0083] The preset scene information mentioned here includes the description information of newly added scenes. The description information of newly added scenes can be obtained based on expert knowledge, that is, human-defined scene information, in order to discover a certain target, context (i.e., surrounding environment) or traffic behavior, such as motorcycles appearing on the street at night, large trucks on the highway in bad weather, vehicles and pedestrians in roundabouts, highway lane changes, and U-turns at street intersections.

[0084] In this embodiment of the application, the third filtering method can be used to filter data as needed, reducing useless data in the final filtering results.

[0085] It should be noted that, in this embodiment of the application, the data to be screened is the initial screening incremental data.

[0086] Preferably, the number of the first preset screening methods is at least two, including at least one of the first screening method and the second screening method, and a third screening method, that is: the first preset screening method includes the first screening method and the third screening method, or includes the second screening method and the third screening method, or includes the first screening method, the second screening method and the third screening method.

[0087] The first and second screening methods are data-based, while the third screening method is knowledge-based. By combining data with knowledge, data can be filtered from different perspectives, thus enriching the filtered data.

[0088] It should be noted that when using multiple filtering methods to filter data, the multiple filtering methods are implemented in parallel, that is, the incremental data is filtered a second time using different filtering methods.

[0089] The following explains the process of filtering data using the first, second, or third filtering method.

[0090] As an optional embodiment, when the first preset filtering method includes the first filtering method, step 202: the server performs a second filtering on the initially screened incremental data using the first preset filtering method to obtain the final incremental data, which may include:

[0091] Step A1: The server inputs the initial screening of incremental data into the autonomous driving algorithm model to obtain the target result.

[0092] The number of target results is at least one, including at least one of the following: perception results, prediction results, and decision results (also known as planning results).

[0093] The autonomous driving algorithm model described herein is stored on a server. In this embodiment, the autonomous driving algorithm can be run step-by-step, meaning the output of the autonomous driving algorithm model does not affect the vehicle. This autonomous driving algorithm can be a modular pipeline, including a perception module 3022, a prediction module 3023, and a planning module 3024, such as... Figure 3 As shown. The perception module 3022 is used to output the perception result, the prediction module 3023 is used to output the prediction result, and the planning module 3024 is used to output the decision result.

[0094] Step A2: The server inputs each target result into the corresponding preset single-class classifier, and determines whether each target result is abnormal based on the output of the preset single-class classifier.

[0095] The preset single-class classifier mentioned here is... Figure 3 The single-class classifier 3025 in the example.

[0096] Different target results correspond to different preset single-class classifiers. The preset single-class classifier is trained using target training data, which is the target result obtained by inputting the historical training data of the autonomous driving algorithm model into the model and corresponding to the preset single-class classifier. That is: the preset single-class classifier corresponding to the perception result is trained using the perception result obtained by inputting the historical training data of the autonomous driving algorithm model into the model; the preset single-class classifier corresponding to the prediction result is trained using the prediction result obtained by inputting the historical training data of the autonomous driving algorithm model into the model; and the preset single-class classifier corresponding to the decision result is trained using the decision result obtained by inputting the historical training data of the autonomous driving algorithm model into the model.

[0097] As mentioned earlier, a single-class classifier can identify data of one category. Therefore, after inputting the target result into the corresponding preset single-class classifier, the output of the classifier can be used to determine whether the input target result is abnormal. That is, if the target result belongs to the target class that the corresponding preset single-class classifier can identify, the target result is determined to be normal; if the target result does not belong to the target class that the corresponding preset single-class classifier can identify, the target result is determined to be abnormal.

[0098] Step A3: If at least one target result is abnormal, the server will determine the initial incremental data as the final incremental data.

[0099] If any of the target results is abnormal, the initial screening incremental data will be determined as the final incremental data. For example, the target results include perception results, prediction results, and decision results. If any of these three results is abnormal, the initial screening incremental data will be determined as the final incremental data. Figure 3 As shown, the server can store the incremental data obtained through the first filtering method into database 3028.

[0100] In this embodiment, data can be filtered as needed based on the different output results of different autonomous driving algorithm models. When using the filtered data for model training, more emphasis can be placed on optimizing the corresponding functions. For example, when using incremental data filtered based on perception results for model training, more emphasis can be placed on optimizing the model's perception function; when using incremental data filtered based on prediction results for model training, more emphasis can be placed on optimizing the model's prediction function; and when using incremental data filtered based on decision results for model training, more emphasis can be placed on optimizing the model's planning function, thereby better achieving model optimization.

[0101] Preferably, the number of target results is at least two, including at least one of a prediction result and a decision result, and the perception result, that is: the target result may include the perception result and the prediction result, or the perception result and the decision result, or the perception result, the prediction result and the decision result.

[0102] In autonomous driving technologies, perception is fundamental. Therefore, it is essential to filter data based on perception results. In this application embodiment, data is not only filtered from the perception level, but also from at least one of the prediction and planning levels, thereby enabling the selection of valuable data from different levels and enriching the model training data.

[0103] The generation process for different target results will be further described below.

[0104] In some embodiments where the target result includes the perception result, step A1: The server inputs the initial screening incremental data into the autonomous driving algorithm model to obtain the target result, which may include:

[0105] Step A111: If the initial screening of incremental data includes image sequences, perform perspective transformation processing on the image sequences to obtain the first target bird's-eye view.

[0106] The image sequence described here is a series of image frames captured by cameras on autonomous vehicles.

[0107] Step A112: Perform map element detection on the first target bird's-eye view and generate a vector map.

[0108] Step A113: Perform target obstacle detection on the first target bird's-eye view to obtain a second target bird's-eye view with the target obstacles marked.

[0109] Step A114: Use the vector map and the second target bird's-eye view as the perception result.

[0110] The following is combined with Figure 6 Steps A111 to A114 will be explained.

[0111] Figure 6 The diagram shows the perception module 3022 in the autonomous driving algorithm model. The perception module 3022 may include: a backbone network module 601, a view conversion module 602, a bird's-eye view target detector 603, a map element detection module 604, and a polyline generator 605.

[0112] like Figure 6As shown, the backbone network module 601 encodes the image sequences in the initial incremental data. Then, the viewpoint conversion module 602 performs viewpoint conversion processing on the image encoding results to obtain a bird's-eye view (i.e., the first target bird's-eye view). The obtained bird's-eye view is then input into the bird's-eye target detector 603 and the map element detection module 604, respectively. The bird's-eye target detector 603 detects targets (i.e., target obstacles) in the bird's-eye view, identifies the detected targets, and outputs bird's-eye target bounding boxes, i.e., the second target bird's-eye view with the identified targets. The map element detection module 604 detects the map elements included in the bird's-eye view, and then inputs the detection results into the polyline generator 605. The polyline generator 605 generates a vector map based on the map element information in the detection results and outputs it. The vector map and the bird's-eye target bounding boxes are the perception results output by the perception module 3022.

[0113] In autonomous driving technologies, cameras are standard equipment in both vehicles used for data collection during the R&D phase and in mass-produced vehicles. Therefore, the solution of obtaining perception results through steps A111 to A114 is applicable to both the vehicle R&D phase and the vehicle mass production phase.

[0114] In some other embodiments where the target result includes the perception result, step A1: The server inputs the initial screening incremental data into the autonomous driving algorithm model to obtain the target result, which may include:

[0115] Step A121: If the initial screening of incremental data includes image sequences and point cloud data, perform perspective transformation processing on the image sequences and point cloud data to obtain the first bird's-eye view and the second bird's-eye view, respectively.

[0116] The image sequence consists of continuous image frames captured by cameras on the autonomous vehicle, while the point cloud data is collected by lidar on the autonomous vehicle.

[0117] Step A122: Perform feature merging processing on the first bird's-eye view and the second bird's-eye view to obtain the first target bird's-eye view.

[0118] Step A123: Perform map element detection on the first target bird's-eye view and generate a vector map.

[0119] Step A124: Perform target obstacle detection on the first target bird's-eye view to obtain a second target bird's-eye view with the target obstacles marked.

[0120] Step A125: Use the vector map and the second target bird's-eye view as the perception result.

[0121] The following is combined with Figure 7 Steps A121 to A125 will be explained.

[0122] Figure 7 Another block diagram of the perception module 3022 in the autonomous driving algorithm model is shown. The perception module 3022 may include: a backbone network module 701, a first-view conversion module 702, a voxelization module 703, a feature encoding module 704, a second-view conversion module 705, a feature merging module 706, a bird's-eye view target detector 707, a map element detection module 708, and a polyline generator 709.

[0123] like Figure 7 As shown, the backbone network module 701 performs image encoding on the image sequences in the initial incremental data. Then, the first viewpoint conversion module 702 performs viewpoint conversion processing on the image encoding results to obtain the first bird's-eye view. The point cloud data in the initial incremental data is voxelized by the voxelization module 703, and the feature encoding module 704 performs feature encoding on the voxelization results. Then, the second viewpoint conversion module 705 performs viewpoint conversion processing on the feature encoding results to obtain the second bird's-eye view. Next, the first and second bird's-eye views are input to the feature merging module 706, which performs feature merging processing on the two bird's-eye views to obtain the first target bird's-eye view. Then, the first target bird's-eye view is input to the bird's-eye target detector 707 and the map element detection module 708, respectively. The bird's-eye target detector 707 detects targets (i.e., target obstacles) in the bird's-eye view, identifies the detected targets, and outputs bird's-eye target bounding boxes, i.e., the second target bird's-eye view with the targets identified. The map element detection module 708 detects map elements included in the bird's-eye view, and then inputs the detection results into the polyline generator 709. The polyline generator 709 generates a vector map based on the map element information in the detection results and outputs it. The vector map and the bird's-eye view target bounding box are the perception results output by the perception module 3022.

[0124] In autonomous driving technologies, both cameras and LiDAR are standard equipment during the vehicle development phase. However, in mass-produced vehicles, cameras are standard, but LiDAR is not. Therefore, the solution for obtaining perception results through steps A111 to A114 is suitable for the vehicle development phase. Of course, it is also suitable for the mass production phase when LiDAR is installed in mass-produced vehicles.

[0125] In some embodiments where the target result includes the predicted result, step A1: The server inputs the initial screening incremental data into the autonomous driving algorithm model to obtain the target result, which may include:

[0126] Step A131: Perform temporal encoding on the bird's-eye view in the perception results output by the autonomous driving algorithm model to obtain spatiotemporal bird's-eye view features.

[0127] The bird's-eye view includes identification information of the target obstacle. The bird's-eye view is obtained based on the image sequence in the initial screening incremental data, as shown in steps A111 to A114; or it is obtained based on the image sequence and point cloud data in the initial screening incremental data, as shown in steps A121 to A125.

[0128] Step A132: Perform motion decoding on the spatiotemporal bird's-eye view features to predict the trajectory of the target obstacle.

[0129] Step A133: Use the predicted trajectory as the prediction result.

[0130] The following is combined with Figure 8 Steps A131 to A133 will be explained.

[0131] Figure 8 This is a block diagram of the perception module 3022 and the prediction module 3023 in an autonomous driving algorithm model. The perception module 3022 shown here is... Figure 7 The sensing module shown. Of course. Figure 8 The prediction module 3023 shown can also be used with Figure 6 The perception module shown is combined. The prediction module 3023 may include a time-domain coding module 801 and a motion decoding module 802.

[0132] like Figure 8 As shown, the second target bird's-eye view output by the feature merging module 706 is input to the temporal decoding module 801, which encodes the second target bird's-eye view to obtain a temporal encoding result, which is a spatiotemporal bird's-eye view feature. The temporal encoding result is then input to the motion decoding module 802 for decoding, i.e., understanding the spatiotemporal bird's-eye view feature to obtain the predicted trajectory of the target obstacle. Optionally, in this implementation, IMU data can be input into the model to participate in temporal encoding. The IMU data includes the vehicle's driving trajectory; combining the vehicle's driving trajectory to predict the trajectory of the target obstacle can improve the accuracy of trajectory prediction. The IMU data can be input into the model through the backbone network module 701.

[0133] The scheme for obtaining the prediction results through steps A131 to A133 is applicable to both the vehicle development stage and the vehicle mass production stage.

[0134] In some embodiments where the target result includes the decision result, step A1: The server inputs the initial screening incremental data into the autonomous driving algorithm model to obtain the target result, which may include:

[0135] Step A141: Plan and decode the spatiotemporal bird's-eye view features to obtain the spatiotemporal feature information of the driving scene.

[0136] Among them, the spatiotemporal bird's-eye view feature is the result obtained by temporal encoding of the bird's-eye view in the perception results output by the autonomous driving algorithm model. The bird's-eye view here includes the identification information of target obstacles. This bird's-eye view is obtained based on the image sequence in the initial screening incremental data, as shown in steps A111 to A114; or based on the image sequence and point cloud data in the initial screening incremental data, as shown in steps A121 to A125.

[0137] The spatiotemporal characteristics of the driving scenario described here are denoted as cost volume, which includes not only the current position information of the target obstacle, but also the future position information of the target obstacle.

[0138] Step A142: Calculate the cost of at least one vehicle trajectory by using a preset cost function and combining it with the spatiotemporal characteristics of the driving scenario.

[0139] Step A143: Use the vehicle trajectory with the lowest cost as the decision result.

[0140] The following is combined with Figure 9 Steps A141 to A143 will be explained.

[0141] Figure 9 This is a block diagram of the perception module 3022, prediction module 3023, and planning module 3024 in an autonomous driving algorithm model. The perception module 3022 shown here is... Figure 7 The sensing module shown. Of course. Figure 9 The prediction module 3023 and planning module 3024 shown can also be used with Figure 6 The sensing modules shown are combined. The planning module 3023 may include: a planning encoding module 901, a minimum cost calculation module 902, and a sampler 903.

[0142] like Figure 9 As shown, the planning decoding module 901 decodes the output of the time-domain encoding module 801 to obtain the cost body. Then, the cost function (i.e., the preset cost function) in the minimum cost module 902, combined with the cost body, calculates the cost of various vehicle trajectories emitted by the sampler 903 and determines the vehicle trajectory with the minimum cost, outputting it. The cost function includes factors such as safety (obstacle avoidance), traffic rules, and trajectory smoothness (acceleration and curvature), meaning that cost is calculated from multiple aspects to obtain the optimal vehicle trajectory.

[0143] The solution obtained through steps A141 to A143 is applicable to both the vehicle development stage and the vehicle mass production stage.

[0144] In the embodiments of this application, the encoding and decoding processes involved in the above-mentioned methods for obtaining perception results, prediction results, and decision results are all implemented using encoders and decoders based on deep learning, which is more intelligent.

[0145] As an optional embodiment, when the first preset filtering method includes a second filtering method, step 202: the server performs a second filtering on the initially screened incremental data using the first preset filtering method to obtain the final incremental data, which may include:

[0146] Step B1: The server determines the distance between the initial screening incremental data and the cluster center of the at least one preset scene cluster.

[0147] Step B2: If the distance between the initial screening incremental data and the cluster center of at least one preset scene cluster is greater than or equal to the distance threshold, the server determines the initial screening incremental data as the final incremental data.

[0148] If the initial screening incremental data is far from the cluster center of a preset scene cluster, it means that the initial screening incremental data does not belong to the preset scene cluster, and the initial screening incremental data is considered abnormal data, i.e., incremental data. Therefore, if the distance between the initial screening incremental data and the cluster center of all preset scene clusters is greater than or equal to the distance threshold, it means that the initial screening incremental data does not belong to any preset scene cluster, and the server determines the initial screening incremental data as the final incremental data.

[0149] The distance threshold mentioned here can be set according to actual needs, and this application embodiment does not limit it.

[0150] In this embodiment of the application, it can be achieved through... Figure 3 The clustering module 3026 shown performs clustering processing on the initial screening incremental data. Afterwards, the server can determine whether the initial screening incremental data is abnormal based on the clustering results, and if abnormal, store the initial screening incremental data in the database 3028.

[0151] As an optional embodiment, when the first preset filtering method includes a third filtering method, step 202: the server performs a second filtering on the initially screened incremental data using the first preset filtering method to obtain the final incremental data, which may include:

[0152] Step C1: The server searches for incremental data that matches the preset scenario information in the initial screening of incremental data.

[0153] Step C2: The server determines the incremental data that matches the preset scene information as the final incremental data.

[0154] In this embodiment, the required incremental scene data can be predefined manually, so that the server can search for matching data in the initial screening of incremental data based on the predefined scene information, thereby completing the data filtering. For example, if the preset scene information includes descriptive information about a high-speed lane-changing scene, such as speed value and road conditions, the server can search for matching feature data in the initial screening of incremental data.

[0155] In this embodiment of the application, it can be achieved through... Figure 3 The search module 3027 shown searches for data that matches the preset scene information in the initial screening of incremental data. After that, the server can store the found data in the database 3028.

[0156] This application also provides a data filtering method, which can be implemented by an autonomous vehicle, i.e., the method is executed by an autonomous vehicle.

[0157] like Figure 10 As shown, the data filtering method may include:

[0158] Step 1001: Autonomous vehicles collect vehicle driving data.

[0159] The aforementioned vehicle driving data can be the autonomous vehicle's own driving data collected by the autonomous vehicle itself, or it can be obtained by the autonomous vehicle from other autonomous vehicles.

[0160] The aforementioned vehicle driving data may include, but is not limited to, data collected through at least one of the following vehicle-mounted sensors: cameras, lidar, millimeter-wave radar, GNSS, IMU, and others. For a detailed description of these sensors, please refer to the previous embodiment; further details will not be provided here.

[0161] Step 1002: The autonomous vehicle filters the vehicle driving data through a second preset filtering method to obtain preliminary incremental data.

[0162] The initial incremental data described here is obtained by filtering at least one of the prediction data and decision data of autonomous vehicles, as well as the perception data of autonomous vehicles.

[0163] The aforementioned perception data may include, but is not limited to: data collected by various sensors on autonomous vehicles, and perception results output by onboard autonomous driving algorithm models.

[0164] The aforementioned prediction data may include, but is not limited to, trajectory prediction results output for surrounding obstacles through in-vehicle autonomous driving algorithm models or other means.

[0165] The aforementioned decision data may include, but is not limited to: data on the decision-making behavior of vehicle drivers (such as safety drivers during the vehicle development phase) and decision results output by onboard autonomous driving algorithms for the vehicle's driving trajectory.

[0166] In autonomous driving technologies, perception is fundamental. Therefore, data filtering based on perception data is essential. The applicant has discovered that vehicle prediction and decision-making data can also demonstrate data value to a certain extent. For example, in driving scenarios including mass protests, if an autonomous driving algorithm model fails to predict the trajectories of obstacles (i.e., numerous pedestrians), or if the vehicle trajectory in the decision result shows a high risk, it indicates that the model lacks the ability to handle such driving scenarios. In this case, the data from such driving scenarios is valuable data, i.e., incremental data. Based on this discovery, this application proposes data filtering not only based on vehicle perception data but also on at least one of vehicle prediction and decision-making data. This allows for data filtering at different levels, enriching the incremental data obtained through filtering and reducing the omission of valuable data.

[0167] The initial incremental data mentioned here is sent to the server, which then performs a second screening on it using a first preset screening method to obtain the final incremental data.

[0168] Because the server has stronger computing power and does not need to filter the data collected from vehicles in real time, it can run more complex data filtering algorithms to filter the data more accurately. Therefore, the first preset filtering method is more complex and more accurate than the second preset filtering method, thus making the data filtering more precise.

[0169] It should be noted that the incremental data mentioned in this application embodiment can be understood as missing data in the training data of the autonomous driving algorithm model, or as abnormal data (such as abnormal perception data such as incorrect target matching in different cameras, target jitter in consecutive frames, or sudden disappearance of the target) or dangerous driving scenario data (such as scenario data of driving on a highway in bad weather). The aforementioned initial screening incremental data may include at least one set of data, each set of data corresponding to a driving scenario.

[0170] As an optional embodiment, the number of second preset filtering methods is at least one, including at least one of the following: first filtering method, second filtering method, third filtering method, and fourth filtering method.

[0171] The first screening method is similar to the aforementioned server-side first screening method, which filters data based on the output of the in-vehicle autonomous driving algorithm model and a single-class classifier. For example... Figure 11As shown, the vehicle can input the output of the autonomous driving algorithm model 1101 into the single-class classifier 1102. Based on the output of the single-class classifier 1102, it can determine whether the vehicle driving data is abnormal. If it is abnormal, it will be filtered out. For details on the first filtering method, please refer to the previous embodiment, which will not be repeated here.

[0172] In this embodiment, the single-class classifier is trained based on the historical training data of the autonomous driving algorithm model, which helps to better distinguish between incremental and non-incremental data and provides a powerful tool for understanding vehicle driving data. Therefore, incremental data can be effectively filtered through the first filtering method.

[0173] Alternatively, for lightweight operation on the vehicle side, a one-class support vector machine (SVM) model can be used to train the single-class classifier.

[0174] The second filtering method involves filtering data using a shadow mode. For example... Figure 11 As shown, the shadow mode module 1103 can filter data based on the output of the onboard autonomous driving algorithm. For example, if the perception module of the autonomous driving algorithm model detects that a target (such as an obstacle) is mismatched in different cameras, there is jitter or sudden disappearance in consecutive frames, or there are strong changes in lighting when entering or exiting a tunnel, then the vehicle driving data can be considered abnormal and filtered out. Furthermore, abnormal situations such as a vehicle accelerating while cutting in or decelerating while cutting out, failing to avoid an obstacle in front, or being nearly collided with by a vehicle detected by the side and rear cameras when changing lanes can also be considered abnormal vehicle driving data and filtered out.

[0175] The third filtering method is a search filtering method, which involves filtering data by searching for data that matches preset scenario information in vehicle driving data.

[0176] The preset scene information mentioned here includes the description information of newly added scenes. The description information of newly added scenes can be obtained based on expert knowledge, that is, human-defined scene information, in order to discover a certain target, context (i.e., surrounding environment) or traffic behavior, such as motorcycles appearing on the street at night, large trucks on the highway in bad weather, vehicles and pedestrians in roundabouts, highway lane changes, and U-turns at street intersections.

[0177] like Figure 11 As shown, the search module 1104 can search for data in the vehicle driving data that matches the preset scenario information and filter it out.

[0178] The fourth screening method is based on data filtering based on driving operations.

[0179] In this embodiment of the application, it can be achieved through... Figure 11 The driving operation module 1105 shown detects abnormal driving behavior by obtaining data such as yaw rate and speed from the vehicle's CAN bus. This includes abnormal driving behavior such as the vehicle's trajectory exhibiting a "dragon-like" phenomenon, excessive acceleration or braking, large-angle steering wheel turns or excessive turning angles, and triggering the Autonomous Emergency Braking (AEB) system.

[0180] In this embodiment, the first and second filtering methods are data-based filtering methods. The third and fourth filtering methods are knowledge-based data filtering methods, which can filter data as needed and reduce useless data in the final filtering results.

[0181] Optionally, in this embodiment of the application, the initial screening incremental data can be labeled to identify the screening method and the reason for screening, etc. Figure 11 As shown, the initial screening incremental data can be tagged by the data acquisition module, and then the tagged initial screening incremental data can be sent to the server for storage.

[0182] As an optional embodiment, when the second preset screening method includes the first screening method, step 1002: the autonomous vehicle filters the vehicle driving data through the second preset screening method to obtain preliminary incremental data, which may include:

[0183] Step D1: The autonomous vehicle inputs the vehicle driving data into the onboard autonomous driving algorithm model to obtain the target result.

[0184] The number of target results is at least one, including at least one of the following: perception result, prediction result, and decision result. The target results described here are similar to the server-side target results in the previous embodiment, and will not be repeated here to avoid repetition.

[0185] Step D2: The autonomous vehicle inputs each target result into the corresponding preset single-class classifier to determine whether each target result is abnormal.

[0186] The preset single-class classifiers differ for different target results. These preset single-class classifiers are obtained by training the model with the target results obtained after inputting historical training data of the autonomous driving algorithm model. The preset single-class classifiers described here are similar to the server-side preset single-class classifiers in the previous embodiment, and will not be described again here to avoid repetition.

[0187] Step D3: In the event of an anomaly in at least one target result, the autonomous vehicle will identify the vehicle driving data as the initial incremental data for screening.

[0188] In this embodiment, data can be filtered as needed based on the different output results of different autonomous driving algorithm models. When using the filtered data for model training, more emphasis can be placed on optimizing the corresponding functions. For example, when using incremental data filtered based on perception results for model training, more emphasis can be placed on optimizing the model's perception function; when using incremental data filtered based on prediction results for model training, more emphasis can be placed on optimizing the model's prediction function; and when using incremental data filtered based on decision results for model training, more emphasis can be placed on optimizing the model's planning function, thereby better achieving model optimization.

[0189] Preferably, the number of target results is at least two, including at least one of a prediction result and a decision result, and the perception result, that is: the target result may include the perception result and the prediction result, or the perception result and the decision result, or the perception result, the prediction result and the decision result.

[0190] In autonomous driving technologies, perception is fundamental. Therefore, it is essential to filter data based on perception results. In this application embodiment, data is not only filtered from the perception level, but also from at least one of the prediction and planning levels, thereby enabling the selection of valuable data from different levels and enriching the model training data.

[0191] The above is a description of the data filtering method provided in the embodiments of this application.

[0192] In summary, this application's embodiments employ a data filtering mechanism combining vehicle-side and server-side (i.e., cloud-based) approaches. On the vehicle side, incremental data collection is triggered online in a lightweight manner, while on the server side, more complex offline incremental data mining algorithms are performed based on deep learning. This provides a more comprehensive system filtering mechanism for incremental data mining, helping to improve the accuracy of data filtering. Furthermore, this application's embodiments can also perform data filtering at three levels: perception, prediction, and planning, enabling better optimization of different model functions. Further, this application's embodiments can perform data filtering based on knowledge and data separately, enriching the incremental data, such as... Figure 12 As shown, data can be filtered using knowledge-based methods (such as search filtering and driving operation filtering) and data-based methods (such as shadow mode, single-classification filtering and clustering filtering). The filtered data can then be used to build a scenario library, thereby enriching the scenario data within the scenario library.

[0193] Exemplary device

[0194] Accordingly, this application also provides a data filtering device deployed on a server.

[0195] like Figure 13 As shown, the device may include:

[0196] The data acquisition module 1301 is used to acquire incremental data from the initial screening.

[0197] The initial incremental data is obtained by filtering based on at least one of the prediction data and decision data of the autonomous vehicle, as well as the perception data of the autonomous vehicle.

[0198] The first data filtering module 1302 is used to perform secondary filtering on the initial incremental data obtained by the data acquisition module 1301 through a first preset filtering method to obtain the final incremental data.

[0199] The final incremental data includes at least one of abnormal data and dangerous driving scenario data.

[0200] Optionally, the first preset filtering method includes a first filtering method, which involves inputting the initial screening incremental data into the autonomous driving algorithm model on the server side, and then filtering the data based on the model output results and a single-class classifier.

[0201] Optionally, if the first preset filtering method includes the first filtering method, the first data filtering module 1302 may include:

[0202] The model processing unit is used to input the initial screening incremental data into the autonomous driving algorithm model to obtain the target result.

[0203] The number of target results is at least one, including at least one of the following: perception results, prediction results, and decision results.

[0204] An anomaly detection unit is used to input each target result into a corresponding preset single-class classifier, and determine whether each target result is abnormal based on the output result of the preset single-class classifier.

[0205] The preset single-class classifiers corresponding to different target results are different. These preset single-class classifiers are trained using target training data, which consists of target results obtained by inputting historical training data from the autonomous driving algorithm model into the model and corresponding to the preset single-class classifier. If a target result does not belong to a target class that the corresponding preset single-class classifier can recognize, the target result is determined to be abnormal.

[0206] The first incremental data filtering unit is used to determine the initial incremental data as the final incremental data in the event that at least one of the target results is abnormal.

[0207] Optionally, the number of target results is at least two, including at least one of the prediction result and the decision result, and the perception result.

[0208] Optionally, when the target result includes a perception result, the model processing unit is specifically used for:

[0209] If the initial screening incremental data includes an image sequence, the image sequence is subjected to a perspective transformation process to obtain a first target bird's-eye view; wherein, the image sequence is acquired by a camera on an autonomous vehicle;

[0210] Perform map element detection on the first target's bird's-eye view and generate a vector map;

[0211] Target obstacle detection is performed on the first target bird's-eye view to obtain a second target bird's-eye view in which the target obstacles are marked;

[0212] The vector map and the second target bird's-eye view are used as the perception results.

[0213] Optionally, when the target result includes a perception result, the model processing unit is specifically used for:

[0214] When the initial screening incremental data includes image sequences and point cloud data, the image sequences and point cloud data are subjected to perspective transformation processing to obtain a first bird's-eye view and a second bird's-eye view, respectively; wherein, the image sequences are acquired by cameras on autonomous vehicles, and the point cloud data are acquired by lidar on autonomous vehicles;

[0215] The first bird's-eye view and the second bird's-eye view are combined to obtain the first target bird's-eye view.

[0216] Perform map element detection on the first target's bird's-eye view and generate a vector map;

[0217] Target obstacle detection is performed on the first target bird's-eye view to obtain a second target bird's-eye view in which the target obstacles are marked;

[0218] The vector map and the second target bird's-eye view are used as the perception results.

[0219] Optionally, when the target result includes the prediction result, the model processing unit is specifically used for:

[0220] The bird's-eye view in the perception results output by the autonomous driving algorithm model is temporally encoded to obtain spatiotemporal bird's-eye view features; wherein, the bird's-eye view includes identification information of target obstacles, and the bird's-eye view is obtained based on the image sequence in the initial screening incremental data, or based on the image sequence and point cloud data in the initial screening incremental data, wherein the image sequence is obtained by the camera on the autonomous driving vehicle, and the point cloud data is obtained by the lidar on the autonomous driving vehicle.

[0221] Motion decoding is performed on the spatiotemporal bird's-eye view features to predict the trajectory of the target obstacle;

[0222] The predicted trajectory is used as the prediction result.

[0223] Optionally, when the target result includes a decision result, the model processing unit is specifically used for:

[0224] The spatiotemporal bird's-eye view features are planned and decoded to obtain the spatiotemporal feature information of the driving scene; wherein, the spatiotemporal bird's-eye view features are the result of temporal coding of the bird's-eye view in the perception results output by the autonomous driving algorithm model; the bird's-eye view includes the identification information of target obstacles, and the bird's-eye view is obtained based on the image sequence in the initial screening incremental data, or based on the image sequence and point cloud data in the initial screening incremental data, wherein the image sequence is obtained by the camera on the autonomous driving vehicle, and the point cloud data is obtained by the lidar on the autonomous driving vehicle;

[0225] By using a preset cost function and combining the spatiotemporal feature information, the cost of at least one vehicle trajectory is calculated.

[0226] The vehicle trajectory with the lowest cost is taken as the decision result.

[0227] Optionally, the first preset screening method includes a second screening method, which performs data screening based on the clustering results of the initial screening incremental data and at least one preset scenario cluster, wherein the preset scenario cluster is obtained by clustering historical driving scenario data.

[0228] Optionally, the first preset filtering method includes a third filtering method, which filters data by searching for data that matches preset scenario information in the initial incremental data. The preset scenario information includes descriptive information of the newly added driving scenario.

[0229] The data filtering device deployed on a server provided in this embodiment belongs to the same application concept as the server-implemented data filtering method provided in the above embodiments of this application. It can execute the server-implemented data filtering method provided in any of the above embodiments of this application and has the corresponding functional modules and beneficial effects of the execution method. Technical details not described in detail in this embodiment can be found in the specific processing content of the server-implemented data filtering method provided in the above embodiments of this application, and will not be repeated here.

[0230] Accordingly, this application also provides a data filtering device deployed in an autonomous vehicle.

[0231] like Figure 14 As shown, the device may include:

[0232] The data acquisition module 1401 is used to collect vehicle driving data.

[0233] The second data filtering module 1402 is used to filter the vehicle driving data collected by the data acquisition module 1401 through a second preset filtering method to obtain preliminary incremental data.

[0234] The initial incremental data is obtained by filtering at least one of the prediction data and decision data of the autonomous vehicle, as well as the perception data of the autonomous vehicle; the initial incremental data is sent to the server, so that the server performs a second filtering on it through a first preset filtering method to obtain the final incremental data; the final incremental data includes at least one of abnormal data and dangerous driving scenario data.

[0235] Optionally, the number of the second preset filtering methods is at least one, including at least one of the following: a first filtering method, a second filtering method, a third filtering method, and a fourth filtering method.

[0236] The first screening method is to screen data based on the output results of the in-vehicle autonomous driving algorithm model and a single-class classifier.

[0237] The second filtering method is to filter data using shadow mode.

[0238] The third filtering method involves filtering data by searching for data that matches preset scenario information in the vehicle driving data. The preset scenario information includes descriptive information of the newly added scenario.

[0239] The fourth filtering method is based on data filtering according to driving operations.

[0240] Optionally, when the second preset filtering method includes the first filtering method, the second data filtering module 1402 may include:

[0241] The model processing unit is used to input the vehicle driving data into the on-board autonomous driving algorithm model to obtain the target result.

[0242] The number of target results is at least one, including at least one of the following: perception results, prediction results, and decision results.

[0243] An anomaly detection unit is used to input each target result into the corresponding preset single-class classifier to determine whether each target result is abnormal.

[0244] The preset single-class classifiers corresponding to different target results are different. These preset single-class classifiers are trained using target training data, which are target results obtained by inputting historical training data from the autonomous driving algorithm model into the model and corresponding to the preset single-class classifier. If a target result does not belong to a target class that the corresponding preset single-class classifier can recognize, the target result is determined to be abnormal.

[0245] An incremental data filtering unit is used to identify the vehicle driving data as the initial incremental data in the event that at least one of the target results is abnormal.

[0246] The data filtering device deployed on an autonomous vehicle provided in this embodiment belongs to the same application concept as the data filtering method implemented by an autonomous vehicle provided in the above embodiments of this application. It can execute the data filtering method implemented by an autonomous vehicle provided in any of the above embodiments of this application, and has the corresponding functional modules and beneficial effects of the execution method. Technical details not described in detail in this embodiment can be found in the specific processing content of the data filtering method implemented by an autonomous vehicle provided in the above embodiments of this application, and will not be repeated here.

[0247] Exemplary System

[0248] This application also provides a data filtering system, which may include: an autonomous vehicle and a server.

[0249] The autonomous vehicle is used to filter its own collected vehicle driving data through a second preset filtering method to obtain preliminary incremental data. The preliminary incremental data is obtained based on at least one of the autonomous vehicle's prediction data and decision data, as well as the autonomous vehicle's perception data.

[0250] The server is used to perform a secondary screening of the initial incremental data obtained from the screening of the autonomous vehicles using a first preset screening method, to obtain the final incremental data. The final incremental data includes at least one of the following: abnormal data and dangerous driving scenario data.

[0251] like Figure 15 As shown, compared to Figure 1 The data closed-loop system shown in this application's embodiment adds a secondary data filtering process on the server side. Because the server has stronger computing power and doesn't need to filter vehicle-collected data in real time, it can run more complex data filtering algorithms to perform more precise data filtering, thereby improving the accuracy of data filtering and reducing the workload of subsequent data labeling and model training.

[0252] The data filtering system provided in this embodiment belongs to the same application concept as the data filtering method provided in the above embodiments of this application. It can execute the data filtering method provided in any of the above embodiments of this application and has the corresponding functional modules and beneficial effects of the execution method. Technical details not described in detail in this embodiment can be found in the specific processing content of the data filtering method implemented by the server and the data filtering method implemented by the autonomous vehicle provided in the above embodiments of this application, which will not be repeated here.

[0253] Exemplary electronic devices

[0254] This application also provides an electronic device, such as... Figure 16 As shown, the device includes a memory 1600 and a processor 1610.

[0255] The memory 1600 is connected to the processor 1610 and is used to store programs.

[0256] When the electronic device is a server, the processor 1610 is used to implement the data filtering method implemented by the server in the above embodiments by running the program stored in the memory 1600. When the electronic device is an autonomous vehicle, the processor 1610 is used to implement the data filtering method implemented by the autonomous vehicle in the above embodiments by running the program stored in the memory 1600.

[0257] Specifically, the aforementioned electronic device may also include: a bus, a communication interface 1620, an input device 1630, and an output device 1640.

[0258] The processor 1610, memory 1600, communication interface 1620, input device 1630, and output device 1640 are interconnected via a bus. Among them:

[0259] A bus can include a pathway for transmitting information between various components of a computer system.

[0260] The processor 1610 can be a general-purpose processor, such as a general-purpose central processing unit (CPU), a microprocessor, etc., or an application-specific integrated circuit (ASIC), or one or more integrated circuits used to control the execution of the program of the present invention. It can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), an off-the-shelf programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0261] The processor 1610 may include a main processor, as well as a baseband chip, modem, etc.

[0262] The memory 1600 stores a program for executing the technical solution of this invention, and may also store an operating system and other key business functions. Specifically, the program may include program code, which includes computer operation instructions. More specifically, the memory 1600 may include read-only memory (ROM), other types of static storage devices capable of storing static information and instructions, random access memory (RAM), other types of dynamic storage devices capable of storing information and instructions, disk storage, flash memory, etc.

[0263] Input device 1630 may include a device for receiving user input data and information, such as a keyboard, mouse, camera, scanner, light pen, voice input device, touch screen, pedometer, or gravity sensor.

[0264] Output device 1640 may include devices that allow information to be output to a user, such as a display screen, printer, speaker, etc.

[0265] The communication interface 1620 may include a device that uses any transceiver to communicate with other devices or communication networks, such as Ethernet, Radio Access Network (RAN), Wireless Local Area Network (WLAN), etc.

[0266] The processor 1610 executes the program stored in the memory 1600 and calls other devices, which can be used to implement the various steps of the data filtering method provided in the above embodiments of this application.

[0267] Exemplary computer program products and storage media

[0268] In addition to the methods and devices described above, embodiments of this application may also be computer program products, which include computer program instructions that, when executed by a processor, cause the processor to perform the steps in the data filtering method described in the embodiments of this application.

[0269] The computer program product can be written in any combination of one or more programming languages ​​to perform the operations of the embodiments of this application. The programming languages ​​include object-oriented programming languages ​​such as Java and C++, as well as conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on the user's computing device, partially on the user's computing device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0270] Furthermore, embodiments of this application may also be storage media storing a computer program, which is executed by a processor of the steps in the data filtering method described in the embodiments of this application.

[0271] For the foregoing method embodiments, in order to simplify the description, they are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, because according to this application, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily essential to this application.

[0272] It should be noted that the various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For apparatus embodiments, since they are basically similar to method embodiments, the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments.

[0273] The steps in the methods of the various embodiments of this application can be adjusted, merged, or deleted in order according to actual needs, and the technical features described in each embodiment can be replaced or combined.

[0274] The modules and sub-modules in the various embodiments of the present application's devices and terminals can be merged, divided, and deleted according to actual needs.

[0275] It should be understood that the disclosed terminals, devices, and methods can be implemented in other ways, given the several embodiments provided in this application. For example, the terminal embodiments described above are merely illustrative. For instance, the division of modules or sub-modules is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple sub-modules or modules may be combined or integrated into another module, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be indirect coupling or communication connection through some interfaces, devices, or modules, and may be electrical, mechanical, or other forms.

[0276] The modules or submodules described as separate components may or may not be physically separate. The components that constitute a module or submodule may or may not be physical modules or submodules; that is, they may be located in one place or distributed across multiple network modules or submodules. Some or all of the modules or submodules can be selected to achieve the purpose of this embodiment's solution, depending on actual needs.

[0277] Furthermore, the functional modules or sub-modules in the various embodiments of this application can be integrated into one processing module, or each module or sub-module can exist physically separately, or two or more modules or sub-modules can be integrated into one module. The integrated modules or sub-modules described above can be implemented in hardware or in the form of software functional modules or sub-modules.

[0278] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0279] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein can be implemented directly by hardware, a software unit executed by a processor, or a combination of both. The software unit can be located in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art.

[0280] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0281] The above description of the disclosed embodiments enables those skilled in the art to make or use this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A data filtering method, characterized in that, Implemented via a server, the method includes: Acquire initial screening incremental data; wherein the initial screening incremental data is obtained based on at least one of the prediction data and decision data of the autonomous vehicle, as well as the perception data of the autonomous vehicle. The initial incremental data is further filtered using a first preset filtering method to obtain final incremental data. The final incremental data includes at least one of abnormal data and dangerous driving scenario data. The first preset filtering method includes a first filtering method, which involves inputting the initial incremental data into the autonomous driving algorithm model on the server and filtering the data based on the model output and a single-class classifier. The step of performing a second screening of the initial incremental data using a first preset screening method to obtain the final incremental data includes: The initial screening incremental data is input into the autonomous driving algorithm model to obtain the target result; wherein, the number of the target result is at least one, including at least one of the following: perception result, prediction result, and decision result; Each target result is input into a corresponding preset single-class classifier. Based on the output of the preset single-class classifier, it is determined whether each target result is abnormal. Different target results correspond to different preset single-class classifiers, which are trained using target training data. The target training data consists of target results corresponding to the preset single-class classifier obtained by inputting historical training data of the autonomous driving algorithm model into the model. If a target result does not belong to a target class that the corresponding preset single-classifier can recognize, the target result is determined to be abnormal. In the event that at least one of the target results is abnormal, the initial screening incremental data will be determined as the final incremental data; When the target result includes the perception result, the step of inputting the initial screening incremental data into the autonomous driving algorithm model to obtain the target result includes: If the initial screening incremental data includes an image sequence, the image sequence is subjected to a perspective transformation process to obtain a first target bird's-eye view; wherein, the image sequence is acquired by a camera on an autonomous vehicle; Perform map element detection on the first target's bird's-eye view and generate a vector map; Target obstacle detection is performed on the first target bird's-eye view to obtain a second target bird's-eye view in which the target obstacles are marked; The vector map and the second target bird's-eye view are used as the perception results.

2. The data filtering method according to claim 1, characterized in that, The number of target results is at least two, including at least one of the prediction result and the decision result, and the perception result.

3. The data filtering method according to claim 1 or 2, characterized in that, When the target result includes the perception result, the step of inputting the initial screening incremental data into the autonomous driving algorithm model to obtain the target result includes: When the initial screening incremental data includes image sequences and point cloud data, the image sequences and point cloud data are subjected to perspective transformation processing to obtain a first bird's-eye view and a second bird's-eye view, respectively; wherein, the image sequences are acquired by cameras on autonomous vehicles, and the point cloud data are acquired by lidar on autonomous vehicles; The first bird's-eye view and the second bird's-eye view are combined to obtain the first target bird's-eye view. Perform map element detection on the first target's bird's-eye view and generate a vector map; Target obstacle detection is performed on the first target bird's-eye view to obtain a second target bird's-eye view in which the target obstacles are marked; The vector map and the second target bird's-eye view are used as the perception results.

4. The data filtering method according to claim 1 or 2, characterized in that, When the target result includes the predicted result, the step of inputting the initial screening incremental data into the autonomous driving algorithm model to obtain the target result includes: The bird's-eye view in the perception results output by the autonomous driving algorithm model is temporally encoded to obtain spatiotemporal bird's-eye view features; wherein, the bird's-eye view includes identification information of target obstacles, and the bird's-eye view is obtained based on the image sequence in the initial screening incremental data, or based on the image sequence and point cloud data in the initial screening incremental data, wherein the image sequence is obtained by the camera on the autonomous driving vehicle, and the point cloud data is obtained by the lidar on the autonomous driving vehicle. Motion decoding is performed on the spatiotemporal bird's-eye view features to predict the trajectory of the target obstacle; The predicted trajectory is used as the prediction result.

5. The data filtering method according to claim 1 or 2, characterized in that, When the target result includes a decision result, the step of inputting the initial screening incremental data into the autonomous driving algorithm model to obtain the target result includes: The spatiotemporal bird's-eye view features are planned and decoded to obtain the spatiotemporal feature information of the driving scene; wherein, the spatiotemporal bird's-eye view features are the result of temporal coding of the bird's-eye view in the perception results output by the autonomous driving algorithm model; the bird's-eye view includes the identification information of target obstacles, and the bird's-eye view is obtained based on the image sequence in the initial screening incremental data, or based on the image sequence and point cloud data in the initial screening incremental data, wherein the image sequence is obtained by the camera on the autonomous driving vehicle, and the point cloud data is obtained by the lidar on the autonomous driving vehicle; By using a preset cost function and combining the spatiotemporal feature information, the cost of at least one vehicle trajectory is calculated. The vehicle trajectory with the lowest cost is taken as the decision result.

6. The data filtering method according to claim 1, characterized in that, The first preset filtering method includes a second filtering method. The second filtering method is to filter data based on the clustering results of the initial screening incremental data and at least one preset scenario cluster. The preset scenario cluster is obtained by clustering historical driving scenario data.

7. The data filtering method according to claim 1, characterized in that, The first preset filtering method includes a third filtering method, which filters data by searching for data that matches preset scenario information in the initial incremental data. The preset scenario information includes descriptive information of the newly added driving scenario.

8. A data filtering method, characterized in that, The method, implemented via autonomous vehicles, includes: Collect vehicle driving data; The vehicle driving data is filtered using a second preset filtering method to obtain initial incremental data. This initial incremental data is obtained by filtering based on at least one of the autonomous vehicle's prediction data and decision data, as well as the autonomous vehicle's perception data. The initial incremental data is sent to a server, which then performs a second filtering using a first preset filtering method to obtain final incremental data. The final incremental data includes at least one of abnormal data and dangerous driving scenario data. The second preset filtering method includes a first filtering method, which involves inputting the vehicle driving data into an onboard autonomous driving algorithm model and then filtering the data based on the model's output and a single-class classifier. The step of filtering the vehicle driving data using a second preset filtering method to obtain preliminary incremental data includes: The vehicle driving data is input into the on-board autonomous driving algorithm model to obtain the target result; wherein, the number of the target result is at least one, including at least one of the following: perception result, prediction result, and decision result; Each target result is input into a corresponding preset single-class classifier to determine whether each target result is abnormal. The preset single-class classifiers are different for different target results. These preset single-class classifiers are trained using target training data, which is the target result corresponding to the preset single-class classifier obtained by inputting historical training data of the autonomous driving algorithm model into the model. If a target result does not belong to a target class that the corresponding preset single-class classifier can recognize, the target result is determined to be abnormal. If at least one of the target results is abnormal, the vehicle driving data will be identified as the initial screening incremental data; When the target result includes a perception result, the step of inputting the vehicle driving data into the on-board autonomous driving algorithm model to obtain the target result includes: If the vehicle driving data includes an image sequence, the image sequence is subjected to a perspective transformation process to obtain a first target bird's-eye view; wherein, the image sequence is acquired by a camera on the autonomous vehicle. Perform map element detection on the first target's bird's-eye view and generate a vector map; Target obstacle detection is performed on the first target bird's-eye view to obtain a second target bird's-eye view in which the target obstacles are marked; The vector map and the second target bird's-eye view are used as the perception results.

9. The data filtering method according to claim 8, characterized in that, The number of target results is at least two, including at least one of the prediction result and the decision result, and the perception result.

10. A data filtering device, characterized in that, Deployed on a server, the device includes: The data acquisition module is used to acquire preliminary screening incremental data; wherein the preliminary screening incremental data is obtained by filtering based on at least one of the prediction data and decision data of the autonomous vehicle, as well as the perception data of the autonomous vehicle. The first data filtering module is used to perform secondary filtering on the initial screening incremental data acquired by the data acquisition module through a first preset filtering method to obtain final incremental data; wherein, the final incremental data includes at least one of abnormal data and dangerous driving scenario data; the first preset filtering method includes a first filtering method, which is to input the initial screening incremental data into the autonomous driving algorithm model on the server side and then perform data filtering based on the model output results and a single classifier; The first data filtering module includes: The model processing unit is used to input the initial screening incremental data into the autonomous driving algorithm model to obtain the target result; wherein the number of the target result is at least one, including at least one of the following: perception result, prediction result, and decision result; An anomaly detection unit is used to input each target result into a corresponding preset single-class classifier, and determine whether each target result is abnormal based on the output of the preset single-class classifier; wherein, different target results correspond to different preset single-class classifiers, and the preset single-class classifier is trained using target training data, wherein the target training data is the target result corresponding to the preset single-class classifier obtained by inputting the historical training data of the autonomous driving algorithm model into the model; if the target result does not belong to the target class that the corresponding preset single-class classifier can recognize, the target result is determined to be abnormal; The first incremental data filtering unit is used to determine the initial incremental data as the final incremental data in the case of at least one of the target results being abnormal. When the target result includes a perception result, the model processing unit is specifically used for: If the initial screening incremental data includes an image sequence, the image sequence is subjected to a perspective transformation process to obtain a first target bird's-eye view; wherein, the image sequence is acquired by a camera on an autonomous vehicle; Perform map element detection on the first target's bird's-eye view and generate a vector map; Target obstacle detection is performed on the first target bird's-eye view to obtain a second target bird's-eye view in which the target obstacles are marked; The vector map and the second target bird's-eye view are used as the perception results.

11. A data filtering device, characterized in that, Deployed in autonomous vehicles, the device includes: The data acquisition module is used to collect vehicle driving data; The second data filtering module is used to filter the vehicle driving data using a second preset filtering method to obtain initial incremental data. The initial incremental data is obtained based on at least one of the autonomous vehicle's prediction data and decision data, as well as the autonomous vehicle's perception data. The initial incremental data is sent to a server, which then performs a second filtering using a first preset filtering method to obtain final incremental data. The final incremental data includes at least one of abnormal data and dangerous driving scenario data. The second preset filtering method includes a first filtering method, which involves inputting the vehicle driving data into an onboard autonomous driving algorithm model and then filtering the data based on the model's output and a single-class classifier. The second data filtering module is specifically used for: The vehicle driving data is input into the on-board autonomous driving algorithm model to obtain the target result; wherein, the number of the target result is at least one, including at least one of the following: perception result, prediction result, and decision result; Each target result is input into a corresponding preset single-class classifier to determine whether each target result is abnormal. The preset single-class classifiers are different for different target results. These preset single-class classifiers are trained using target training data, which is the target result corresponding to the preset single-class classifier obtained by inputting historical training data of the autonomous driving algorithm model into the model. If a target result does not belong to a target class that the corresponding preset single-class classifier can recognize, the target result is determined to be abnormal. If at least one of the target results is abnormal, the vehicle driving data will be identified as the initial screening incremental data; When the target result includes a perception result, the second data filtering module is further specifically used for: If the vehicle driving data includes an image sequence, the image sequence is subjected to a perspective transformation process to obtain a first target bird's-eye view; wherein, the image sequence is acquired by a camera on the autonomous vehicle. Perform map element detection on the first target's bird's-eye view and generate a vector map; Target obstacle detection is performed on the first target bird's-eye view to obtain a second target bird's-eye view in which the target obstacles are marked; The vector map and the second target bird's-eye view are used as the perception results.

12. A data filtering system, characterized in that, include: Autonomous vehicles and servers; The autonomous vehicle is used to implement the data filtering method as described in claim 8 or 9; The server is used to implement the data filtering method as described in any one of claims 1 to 7.

13. An electronic device, characterized in that, include: Memory and processor; The memory is connected to the processor and is used to store programs; The processor is used to implement the data filtering method as described in claims 1 to 7 or as described in claims 8 or 9 by running a program in the memory.

14. A storage medium, characterized in that, The storage medium stores a computer program, which, when executed by a processor, implements the data filtering method as described in claims 1 to 7 or as described in claims 8 or 9.

Images