A scene-aware training data acquisition method, related apparatus and system

Abstract

This invention discloses a method, related apparatus, and related system for acquiring scene perception training data, relating to the field of autonomous driving technology. The method includes: receiving vehicle-mounted sensor data for a target time period uploaded by the vehicle and roadside sensor data for the same target time period uploaded by the roadside, with the target time period covering the target moment triggered by user commands or scene matching; reconstructing a 3D scene based on the dual-source data to obtain scene samples characterized by dynamic occupancy grids and traffic participant trajectory sets; and annotating the scene samples with scene semantic information, using these annotated samples to train a scene perception model. This invention achieves real-time automatic capture of high-value scenes through a dual-trigger mechanism that complements subjective and objective data, integrates multi-source vehicle and road data for multi-view reconstruction, improves the quality of scene samples and annotations, and thus enhances the model's perception accuracy, robustness, and generalization ability.

Description

Technical Field

[0001] This invention relates to the field of autonomous driving technology, and in particular to a method, related apparatus and system for acquiring scene perception training data. Background Technology

[0002] Within the existing technological framework, the performance and safety improvements of autonomous driving systems heavily rely on the acquisition and utilization of massive amounts of high-quality training data. Among these, the scene perception model, as the core of the system, directly depends on the quality and distribution of the training data for its perception accuracy, robustness, and generalization ability.

[0003] Currently, the training data for scene perception models is obtained by first collecting autonomous driving scene data, then relying on engineers to manually filter logs or perform post-event backtracking to select scene fragments that can be used for training from massive amounts of data, and then constructing training data based on the selected scene fragments.

[0004] However, high-value scenarios are sparse and difficult to capture. Long-tail scenarios such as ghost peeks, unprotected left turns, and detours around construction zones account for less than 0.1% of the real road environment, but directly determine the safety limit of autonomous driving systems. Existing solutions rely on engineers to manually filter logs or retrospectively review them, which makes it difficult to effectively capture such scenarios that do not cause significant changes in vehicle behavior but contain high-value information. This results in a severe shortage of high-value scenario samples, which in turn affects the model training effect. Summary of the Invention

[0005] In view of this, the present invention provides a method, related apparatus and system for acquiring scene-aware training data, to solve the problem that existing technologies are unable to capture high-value scenes, thus affecting the model training effect. The technical solution is as follows:

[0006] The first aspect of this invention provides a method for acquiring scene-aware training data, applied in the cloud, the method comprising:

[0007] The system receives vehicle-mounted sensor data for the target time period uploaded by the vehicle and roadside sensor data for the target time period uploaded by the roadside. The target time period is a continuous time period covering the target moment, and the target moment is the moment when the vehicle detects a user command to trigger scene capture, or the moment when the vehicle or roadside detects that the current scene matches a preset high-value scene.

[0008] A 3D scene is reconstructed based on vehicle-mounted sensor data and roadside sensor data during the target time period, resulting in a scene sample characterized by a dynamic occupancy grid and a set of traffic participant trajectories.

[0009] The scene samples are annotated with scene semantic information to obtain scene samples annotated with scene semantic information. The scene samples annotated with scene semantic information are used to train the scene perception model.

[0010] In one possible implementation, a 3D scene is reconstructed based on vehicle-mounted sensor data and roadside sensor data for the target time period, resulting in a scene sample characterized by a dynamic occupancy grid and a set of traffic participant trajectories, including:

[0011] The vehicle-mounted sensor data and the roadside sensor data for the target time period are synchronized in time to obtain time-synchronized sensor data.

[0012] The time-synchronized sensor data is spatially aligned to obtain spatially aligned sensor data.

[0013] Based on the spatially aligned sensor data, a dynamic occupancy grid and a set of traffic participant trajectories are generated using a neural radiation field or a multi-view fusion model based on a bird's-eye view transformer, resulting in scene samples characterized by the dynamic occupancy grid and the set of traffic participant trajectories.

[0014] In one possible implementation, the scene-aware training data acquisition method further includes:

[0015] Using a conditional generation model, with the scene sample, random seed, and control conditions representing environmental attributes as input, a variant scene of the scene sample is generated to obtain a variant scene sample.

[0016] The variant scene samples are annotated with scene semantic information to obtain variant scene samples annotated with scene semantic information. The scene samples annotated with scene semantic information and the variant scene samples annotated with scene semantic information are used to train the scene perception model.

[0017] In one possible implementation, the random seed follows a standard normal distribution;

[0018] The control conditions include at least one of the following environmental attribute parameters:

[0019] Weather type used to indicate the weather conditions of a scene;

[0020] Lighting periods used to indicate the lighting conditions of a scene;

[0021] Traffic density is used to indicate the density of traffic participants in a scene.

[0022] In one possible implementation, the scene-aware training data acquisition method further includes:

[0023] The scene perception model is trained using the scene samples labeled with scene semantic information, and the trained scene perception model is sent to the vehicle for deployment.

[0024] The step of training the scene perception model using the scene samples labeled with scene semantic information includes:

[0025] The scene sample is input into the current scene perception model to obtain the first output distribution corresponding to the scene sample, and the scene sample is input into the previous version scene perception model that has been trained to obtain the second output distribution corresponding to the scene sample. The previous version scene perception model has been trained on historical scene samples.

[0026] Calculate the new scene learning loss based on the first output distribution corresponding to the scene sample and the scene semantic information labeled by the scene sample;

[0027] Calculate the knowledge distillation loss based on the first output distribution and the second output distribution;

[0028] The parameters of the current scene perception model are updated based on the new scene learning loss and the knowledge distillation loss.

[0029] In one possible implementation, the scene-aware training data acquisition method further includes:

[0030] Determine the data value score of the scene sample, wherein the data value score characterizes the degree of contribution of the scene sample to the training of the scene perception model;

[0031] If the data value score of the scene sample is greater than the preset score threshold, and if the target time is the time when the vehicle detects a user instruction to trigger scene capture, then keywords related to scene capture are extracted from the user instruction, wherein the vehicle sensor data received from the vehicle is accompanied by the user instruction.

[0032] An update instruction carrying the keyword is sent to the vehicle terminal, the update instruction being used to instruct the vehicle terminal to update the scene capture trigger word set based on the keyword; wherein, the vehicle terminal, in conjunction with the scene capture trigger word set, decides whether to detect a user instruction to trigger scene capture.

[0033] In one possible implementation, determining the data value score of the scene sample includes:

[0034] Obtain the performance gain evaluation value of the trained scene perception model;

[0035] Obtain the coverage evaluation value of the scene sample relative to the existing scene sample library;

[0036] Obtain the redundancy assessment value of the scene sample relative to the existing scene sample library;

[0037] The data value score of the scene sample is determined based on the performance gain evaluation value, the coverage evaluation value, and the redundancy evaluation value.

[0038] In one possible implementation, the scene-aware training data acquisition method further includes:

[0039] If the data value score of the scene sample is greater than a preset score threshold, the scene sample will be added to the high-value sample library for subsequent incremental model training.

[0040] A second aspect of the present invention provides a method for acquiring scene perception training data, applied to a vehicle, the method comprising:

[0041] Real-time acquisition of vehicle-side sensor data, and real-time determination of whether a first trigger condition or a second trigger condition is met. The first trigger condition is a user issuing an instruction to trigger scene capture, and the second trigger condition is the current scene matching a preset high-value scene template.

[0042] If the first triggering condition or the second triggering condition is met, then vehicle sensor data for the target time period is extracted from the acquired vehicle sensor data. The target time period is a continuous time period covering the target time. The target time is the moment when the first triggering condition or the second triggering condition is met.

[0043] The vehicle-mounted sensor data for the target time period is uploaded to the cloud, and the roadside is notified to upload the roadside sensor data for the target time period to the cloud. The cloud then reconstructs a 3D scene based on the vehicle-mounted sensor data and the roadside sensor data for the target time period, and annotates the scene semantic information of the reconstructed scene samples represented by dynamic occupancy grids and traffic participant trajectory sets.

[0044] In one possible implementation, determining whether the first triggering condition is met includes:

[0045] Collect natural language speech from the user, perform speech recognition on the natural language speech, and obtain the recognized text;

[0046] The fine-tuned natural language understanding model is invoked, and combined with the scene capture trigger word set, it is determined whether the recognized text is an instruction that triggers scene capture;

[0047] If the identified text is an instruction to trigger scene capture, then the first triggering condition is determined to be met.

[0048] In one possible implementation, determining whether the second triggering condition is met includes:

[0049] Based on the acquired vehicle-side sensor data, a current scene representation vector is obtained, wherein the current scene representation vector represents the current dynamic traffic environment and the motion state of traffic participants;

[0050] Calculate the similarity between the current scene representation vector and the scene representation vector of each high-value scene template in the preset high-value scene template set;

[0051] If the maximum similarity among the calculated similarities is greater than the preset similarity threshold, then the second triggering condition is satisfied.

[0052] In one possible implementation, the scene-aware training data acquisition method further includes:

[0053] When an update instruction containing keywords related to scene capture is received from the cloud, the keywords are extracted from the update instruction;

[0054] The set of trigger words for scene capture is updated based on the keywords.

[0055] A third aspect of the present invention provides a method for acquiring scene perception training data, applied to roadside environments, the method comprising:

[0056] Real-time acquisition of roadside sensor data, and real-time determination of whether the second triggering condition is met, wherein the second triggering condition is that the current scene matches a preset high-value scene template;

[0057] If the second triggering condition is met, then the roadside sensor data for the target time period is extracted from the acquired roadside sensor data. The target time period is a continuous time period covering the target time, and the target time is the time when the second triggering condition is met.

[0058] The roadside sensor data for the target time period is uploaded to the cloud, and the vehicle is notified to upload its own sensor data for the target time period to the cloud. The cloud then reconstructs a 3D scene based on the vehicle sensor data and roadside sensor data for the target time period, and annotates the scene semantic information of the reconstructed scene samples represented by dynamic occupancy grids and traffic participant trajectory sets.

[0059] A fourth aspect of the present invention provides a cloud server, comprising at least one processor and a memory connected to the processor, wherein:

[0060] The memory is used to store computer programs;

[0061] The processor is used to execute the computer program so that the cloud server can implement any of the above-described methods for acquiring scene perception training data applied in the cloud.

[0062] A fifth aspect of the present invention provides an autonomous driving vehicle, comprising at least one processor and a memory connected to the processor, wherein:

[0063] The memory is used to store computer programs;

[0064] The processor is used to execute the computer program so that the autonomous vehicle can implement any of the above-described methods for acquiring scene perception training data applied to the vehicle.

[0065] A sixth aspect of the present invention provides a roadside device, comprising at least one processor and a memory connected to the processor, wherein:

[0066] The memory is used to store computer programs;

[0067] The processor is used to execute the computer program so that the roadside device can implement the above-described method for acquiring scene perception training data applied to the roadside.

[0068] The seventh aspect of the present invention provides an autonomous driving intelligent agent system, comprising: a cloud-based intelligent agent, a vehicle-side intelligent agent, and a roadside intelligent agent;

[0069] The cloud-based intelligent agent is used to execute any of the above-mentioned methods for acquiring scene perception training data applied to the cloud.

[0070] The vehicle-side intelligent agent is used to execute any of the above-mentioned methods for acquiring scene perception training data applied to the vehicle.

[0071] The roadside agent is used to execute the above-mentioned method for acquiring scene perception training data applied to the roadside.

[0072] By employing the above technical solutions, the scene perception training data acquisition method provided by this invention, on the one hand, can actively capture high-value scenes subjectively perceived by users in real time by supporting user command triggering and scene matching triggering, and can also actively capture scenes that match preset high-value scene templates in real time. This effectively solves the problem that existing technologies rely on engineers manually screening logs or retrospective analysis, making it difficult to capture long-tail scenes, and achieves effective acquisition of high-value scenes. On the other hand, by fusing vehicle-side sensor data and roadside sensor data for multi-view fusion reconstruction, high-quality 3D scenes can be reconstructed, which can significantly reduce information loss and annotation errors caused by occlusion or viewpoint limitations, and significantly improve the quality of scene samples and their annotations. Furthermore, by using the high-value scene samples with high-quality annotations to train the scene perception model, the model's perception accuracy, robustness, and generalization ability can be significantly improved, thereby enhancing the perception reliability of the autonomous driving system in complex long-tail scenes. Attached Figure Description

[0073] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0074] Figure 1 A flowchart illustrating the method for acquiring scene perception training data provided in an embodiment of the present invention;

[0075] Figure 2 This is a schematic diagram of the process for determining whether the first triggering condition is met in the vehicle-side according to an embodiment of the present invention;

[0076] Figure 3 This is a schematic diagram of the process for determining whether the second triggering condition is met in the vehicle-side according to an embodiment of the present invention;

[0077] Figure 4 A schematic diagram illustrating the process of reconstructing a 3D scene in the cloud based on vehicle-side sensor data and roadside sensor data during the target time period, as provided in an embodiment of the present invention.

[0078] Figure 5 This is a schematic diagram of the process of training a scene perception model using scene samples labeled with scene semantic information, as provided in an embodiment of the present invention.

[0079] Figure 6 A schematic diagram illustrating the process of scene sample value assessment and scene capture trigger word set update provided in the embodiments of the present invention;

[0080] Figure 7This is a schematic diagram of the structure of a scene perception training data acquisition device applied to the cloud, provided in an embodiment of the present invention.

[0081] Figure 8 This is a schematic diagram of the structure of a scene perception training data acquisition device applied to a vehicle, as provided in an embodiment of the present invention.

[0082] Figure 9 A schematic diagram of the structure of a scene perception training data acquisition device for roadside applications provided in an embodiment of the present invention;

[0083] Figure 10 This is a schematic diagram of the structure of an autonomous driving intelligent agent system provided in an embodiment of the present invention. Detailed Implementation

[0084] The embodiments of the present invention will now be described with reference to the accompanying drawings. The terminology used in the embodiments section is for illustrative purposes only and is not intended to limit the scope of the invention.

[0085] The embodiments of the present invention will now be described with reference to the accompanying drawings. Those skilled in the art will recognize that, with technological advancements and the emergence of new scenarios, the technical solutions provided by the embodiments of the present invention are equally applicable to similar technical problems.

[0086] The terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such terms are interchangeable where appropriate; this is merely a way of distinguishing objects with the same attributes in the description of embodiments of the invention. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion, so that a process, method, system, product, or apparatus that comprises a series of units is not necessarily limited to those units, but may include other units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0087] In the process of developing this invention, the inventors discovered that the current acquisition of scene perception training data mainly relies on data collected by the vehicle and manual post-processing, which has the following drawbacks:

[0088] First, high-value scenario data relies on manual post-screening, which is inefficient and highly subjective. Existing solutions typically involve large-scale collection of vehicle logs, followed by engineers manually browsing, judging, and extracting potentially high-value scenario fragments using playback tools. This approach has significant drawbacks. Firstly, high-value scenarios (such as pedestrians suddenly appearing from behind obstacles, unprotected left turns, and detouring through construction zones) account for a very small percentage of actual road data. Engineers need to spend a lot of time sifting through massive amounts of mundane data, resulting in extremely low efficiency. Secondly, manual screening heavily relies on personal experience and subjective judgment. The standards of different screeners are inconsistent, and it is easy to miss "gray area" scenarios that do not cause significant abnormal vehicle behavior but contain high-value information (such as pedestrians hesitating to cross the road, and construction cones at a distance). As a result, a large number of long-tail scenarios that are crucial to the robustness of the model are missed or misscreened, resulting in a severe shortage of high-value samples and making it difficult for the model to cover extreme situations on real roads.

[0089] Second, the existing data acquisition architecture has a limited perception range, which makes it difficult to obtain complete global scene information in complex scenarios such as multi-vehicle interaction and occlusion crossing due to the limited perspective. This results in the construction of training data with blind spots or missing information, poor quality, and affects the model training effect.

[0090] Third, the data processing workflow is fragmented and lacks an automated closed loop. The typical workflow of existing solutions is: data collection → manual screening → manual / semi-automatic annotation → offline training → manual verification. Each step relies on manual intervention, and the iteration cycle is usually as long as several weeks. There is a lack of effective feedback mechanisms to evaluate the actual gain of new data on the model, which leads to low-value data being repeatedly annotated, while high-value scenarios are not prioritized for learning. The consequence is low data utilization and limited evolution speed of scene-aware models.

[0091] In view of the numerous problems existing in the current solutions, the inventors of this invention, through continuous research, have proposed a method for acquiring scene perception training data, which overcomes the shortcomings of the existing solutions. The following embodiments will describe the scene perception training data acquisition method provided by this invention.

[0092] Please see Figure 1 The diagram illustrates a flowchart of a scene perception training data acquisition method provided in an embodiment of the present invention. This scene perception training data acquisition method may include:

[0093] Step S101a: The vehicle acquires vehicle sensor data in real time and determines in real time whether the first trigger condition or the second trigger condition is met.

[0094] Among them, vehicle-side sensor data refers to raw or pre-processed perception data collected by sensors deployed on the vehicle (including but not limited to cameras, lidar, millimeter-wave radar, inertial measurement units, etc.).

[0095] In this embodiment, the vehicle acquires vehicle sensor data in real time and determines in real time whether the first triggering condition or the second triggering condition is met. The first triggering condition is that the user issues an instruction to trigger scene capture, that is, the vehicle detects that the user (such as the driver or safety officer) actively issues an instruction (such as pressing a button or a voice command) to request the capture of the current scene. The second triggering condition is that the current scene matches a preset high-value scene template (such as a right vehicle forcibly cutting in, or a left turn conflict without protection).

[0096] This step aims to enable proactive, real-time collection of high-value scenario data, avoiding passive sifting from massive amounts of data afterward, and improving the targeting and efficiency of high-value scenario data collection.

[0097] Step S102a: If the vehicle determines that the first triggering condition or the second triggering condition is met, the vehicle sensor data for the target time period is extracted from the acquired vehicle sensor data, the vehicle sensor data for the target time period is uploaded to the cloud, and the roadside is notified to upload the roadside sensor data for the target time period to the cloud.

[0098] The vehicle can collect user voice through voice acquisition devices (such as microphone arrays), and then determine whether the user voice is an instruction to trigger scene capture. If the user voice is an instruction to trigger scene capture, then the first triggering condition is met.

[0099] The vehicle can also determine in real time whether the current scene matches the preset high-value scene template based on the vehicle sensor data it acquires. If the current scene matches the preset high-value scene template, it is determined that the second triggering condition is met.

[0100] If the vehicle determines that either the first or second triggering condition is met, it proactively initiates high-value scenario data capture. This involves extracting the target time period's vehicle-side sensor data from the acquired sensor data, packaging and uploading this data to the cloud. Simultaneously, the vehicle can send a notification to the roadside via vehicle-to-infrastructure communication (e.g., V2X) to inform the roadside equipment that a high-value scenario has been triggered and to also upload the target time period's roadside sensor data to the cloud. It should be noted that after receiving the data upload notification from the vehicle, the roadside equipment will also upload the target time period's roadside sensor data to the cloud.

[0101] The target time period is a continuous period of time covering the target moment. The target moment is the moment when the first triggering condition is met (i.e., the moment when the user issues an instruction to trigger scene capture) or the moment when the second triggering condition is met (i.e., the moment when the current scene is detected to match a preset high-value scene template). For example, the target time period can be a time period formed by tracing back a first preset duration from the target moment and extending forward a second preset duration, such as [t-10s, t+5s], where t is the target moment.

[0102] Step S101b: The roadside acquires roadside sensor data in real time and determines in real time whether the second triggering condition is met.

[0103] Among them, roadside sensor data is raw or pre-processed data collected by sensing devices deployed on the roadside (including but not limited to roadside cameras, lidar, millimeter-wave radar, etc.), used to cover a wider range and multiple perspectives of scene perception.

[0104] Roadside equipment can run scene matching algorithms independently and detect in real time whether the current scene matches the preset high-value scene template. When the roadside equipment detects that the current scene matches the preset high-value scene template, it considers that a high-value scene has been found. At this time, the roadside can actively initiate the high-value scene data capture, thereby avoiding the omissions that may be caused by relying solely on vehicle-side detection and realizing the complementary triggering capability of vehicle-road cooperation.

[0105] Step S102b: If the roadside determination meets the second triggering condition, the roadside sensor data for the target time period is extracted from the acquired roadside sensor data, the roadside sensor data for the target time period is uploaded to the cloud, and the vehicle is notified to upload the vehicle sensor data for the target time period to the cloud.

[0106] When the roadside device detects that the current scene matches the preset high-value scene template, it uploads the roadside sensor data for the target time period to the cloud. At the same time, it sends a notification to the vehicle (e.g., vehicles within the roadside coverage area) via vehicle-to-infrastructure communication, requesting the vehicle to upload its own sensor data for the same time period to the cloud.

[0107] It should be noted that after receiving the data upload notification from the roadside, the vehicle-mounted sensor data for the target time period is uploaded to the cloud.

[0108] Steps S101a / S102a and S101b / S102b are two parallel or complementary triggering paths. In practical applications, they can be enabled simultaneously or only one of them can be enabled. When steps S101b / S102b and S101a / S102a are enabled simultaneously, a bidirectional triggering mechanism between the vehicle and the roadside can be formed, improving the robustness of triggering coverage.

[0109] Steps S101b / S102b and S101a / S102a provide multi-source data for subsequent fusion reconstruction. Among them, the advantages of roadside beyond-line-of-sight and multi-view can make up for the blind spots of vehicle-side perception.

[0110] Step S103: The cloud receives the vehicle-mounted sensor data for the target time period uploaded by the vehicle and the roadside sensor data for the target time period uploaded by the roadside. Based on the vehicle-mounted sensor data and the roadside sensor data for the target time period, a 3D scene is reconstructed to obtain a scene sample characterized by a dynamic occupancy grid and a set of traffic participant trajectories.

[0111] The cloud serves as the data processing center, receiving vehicle-mounted sensor data for the target time period uploaded by the vehicle and roadside sensor data for the target time period uploaded by the roadside. It then performs 3D scene reconstruction to generate a dynamic occupancy grid and a set of traffic participant trajectories.

[0112] Generating a dynamic occupancy grid refers to dividing space into grids, with each grid updating its probability of being occupied over time, thus characterizing the spatiotemporal occupancy changes of static obstacles (such as guardrails and curbs) and dynamic objects (such as vehicles and pedestrians). Generating a traffic participant trajectory set refers to extracting continuous trajectory points (position, speed, orientation, etc.) for each traffic participant to form structured trajectory data.

[0113] The dynamic occupancy grid and the trajectory set of traffic participants together constitute a scene sample, which fully records the static structure of the traffic environment, the movement of dynamic objects, and their interactions within the target time period.

[0114] This step aims to transform raw sensor data (such as point clouds, images, etc.) into a structured scene representation that is easy for models to process.

[0115] Step S104: The vehicle end annotates the scene samples with scene semantic information to obtain scene samples annotated with scene semantic information.

[0116] The semantic information of the scene sample annotation may include, but is not limited to, some or all of the following: scene geometric semantic information, scene attribute semantic information, and scene behavior semantic information.

[0117] Scene geometric semantic information may include, but is not limited to, the following: traffic participant bounding boxes b n (t)=[x,y,w,h,θ], where (x,y) are the coordinates of the center point, w is the width, h is the height, θ is the orientation angle, the subscript n indicates the nth traffic participant, and t is the timestamp.

[0118] Scene attribute semantic information may include, but is not limited to, some or all of the following information: traffic light status. t =parseFrom(IRSU The drivable area F = RoadSegmentation(O) is calculated using a road segmentation model, where O is a dynamically occupied grid.

[0119] Scene behavior semantic information may include, but is not limited to, the following: actions of traffic participants. n (t)∈{stop, turn-left, yield, …}, action n (t) represents the behavior category of the nth traffic participant at time t.

[0120] It should be noted that by continuously performing trigger detection, uploading data from the vehicle and roadside, reconstructing 3D scenes, and annotating scene semantic information, multiple scene samples labeled with scene semantic information can be accumulated.

[0121] After obtaining scene samples labeled with scene semantic information, the cloud can use these samples to train a scene perception model and then send the trained model to the vehicle. Specifically, after training, the cloud will distribute the model parameters or complete model file of the trained scene perception model to the vehicle via OTA or other means, enabling the vehicle to have real-time scene perception capabilities without relying on real-time inference from the cloud (reducing latency and bandwidth requirements).

[0122] The vehicle receives and deploys the trained scene perception model from the cloud. Specifically, after receiving the scene perception model from the cloud, the vehicle integrates it into its autonomous driving software stack for local loading and inference. Subsequently, the vehicle can run the model in real time during driving to perform semantic perception of the current scene, which can be used to assist in decision-making, early warning, or planning.

[0123] This invention enables a collaborative model of "cloud-based training and vehicle-side inference," balancing model performance and real-time capabilities. Furthermore, the scene-aware model can be continuously iterated and upgraded as cloud-based training data accumulates, and the vehicle-side model can be updated via OTA (Over-The-Air) updates, forming a data loop.

[0124] The scene perception training data acquisition method provided in this invention has several advantages. First, by supporting user command triggering and scene matching triggering, it can capture high-value scene data subjectively perceived by users in real time, and can also capture scene data matching preset high-value scene templates in real time. This effectively solves the problem that existing technologies rely on engineers manually screening logs or retrospective analysis, making it difficult to capture long-tail scenes, and achieves effective acquisition of high-value scenes. Second, by fusing vehicle-side sensor data and roadside sensor data for multi-view fusion reconstruction, it can reconstruct high-quality 3D scenes, thereby significantly reducing information loss and annotation errors caused by occlusion or viewpoint limitations, and significantly improving the quality of scene samples and their annotations. Third, by using the high-value scene samples with high-quality annotations to train the scene perception model, it can significantly improve the model's perception accuracy, robustness, and generalization ability, thereby enhancing the perception reliability of the autonomous driving system in complex long-tail scenes.

[0125] As mentioned in the above embodiments, the vehicle can determine in real time whether the first triggering condition (the user issues an instruction to trigger scene capture) is met. If the first triggering condition is met, the vehicle sensor data for the target time period is extracted from the acquired vehicle sensor data, and then the vehicle sensor data for the target time period is uploaded to the cloud.

[0126] In one possible implementation, the user can trigger scene capture by issuing a voice command. In some embodiments of the present invention, the process of determining whether the first triggering condition (the user issues a command to trigger scene capture) is described.

[0127] like Figure 2 As shown, the process by which the vehicle determines whether the first triggering condition (the user issues an instruction to trigger scene capture) is met may include:

[0128] Step S201: The vehicle collects the natural language speech emitted by the user, performs speech recognition on the collected natural language speech, and obtains the recognized text.

[0129] The vehicle uses a voice acquisition device (such as a microphone array) to collect the user's natural language speech (e.g., "That electric car suddenly darted out just now, write it down!"), and calls the speech recognition (ASR) module to perform speech recognition on the collected natural language speech to obtain the recognized text.

[0130] Step S202: Call the fine-tuned natural language understanding model, combine it with the scene capture trigger word set, and determine whether the recognized text is an instruction that triggers scene capture.

[0131] The fine-tuned natural language understanding model (e.g., a fine-tuned BERT model) is invoked, and a set of trigger words C is captured based on a pre-defined scenario. trigger(For example, C) trigger ={“Record this”, “Save this”, “Dangerous scene”,…}, perform semantic classification on the identified text to determine whether it is an instruction to trigger scene capture:

[0132] (1).

[0133] Specifically, according to the preset prompt template, the recognized text and scene capture trigger word set C are... trigger Fill in the prompt template to construct a complete prompt. This prompt is used to guide the fine-tuned natural language understanding model (e.g., a fine-tuned BERT model) to combine with the scene capture trigger word set to determine whether the recognized text is an instruction to trigger scene capture, and obtain the judgment result. Below is an example of a prompt template:

[0134] "You are a scene capture command recognition assistant. Please determine whether the user's input is a command that triggers scene capture."

[0135] Examples of commands that trigger scene capture include, but are not limited to, the following keywords or phrases: {List of capture trigger words};

[0136] User input: {Recognize text}.

[0137] Please output:

[0138] If it is a trigger command, output "1";

[0139] If not, output "0".

[0140] Step S203a: If the recognized text is an instruction to trigger scene capture, then the first triggering condition is satisfied.

[0141] If the fine-tuned natural language understanding model determines that the recognized text is an instruction to trigger scene capture, then it can be determined that the user issued an instruction to trigger scene capture.

[0142] Step S203b: If the identified text is not an instruction to trigger scene capture, then the first triggering condition is not met.

[0143] If the fine-tuned natural language understanding model determines that the recognized text is not an instruction to trigger scene capture, then it can be determined that the natural language speech emitted by the user is not an instruction to trigger scene capture.

[0144] As mentioned in the above embodiments, the vehicle can determine in real time whether the second triggering condition (the current scene matches a preset high-value scene template) is met. If the second triggering condition is met, the vehicle sensor data for the target time period is extracted from the acquired vehicle sensor data, and then the vehicle sensor data for the target time period is uploaded to the cloud. In some embodiments of the present invention, the process of the vehicle determining whether the second triggering condition is met is described.

[0145] like Figure 3 As shown, the process by which the vehicle determines whether the second triggering condition (the current scene matches the preset high-value scene template) is met may include:

[0146] Step S301: Obtain the current scene representation vector based on the acquired vehicle-side sensor data.

[0147] Based on the acquired vehicle-side sensor data, a current scene representation vector is obtained through a multi-source fusion algorithm. The current scene representation vector is a vector that represents the current scene, specifically representing the current dynamic traffic environment and the motion state of traffic participants. The components of the current scene representation vector include, but are not limited to: the speed and acceleration of the vehicle and surrounding traffic participants (such as the vehicle in front, the vehicle next to it, and pedestrians), the relative distance and relative speed between the vehicle and surrounding traffic participants, the traffic light status (red, yellow, green, countdown, etc.), lane line information, and the boundary of the drivable area.

[0148] Step S302: Calculate the similarity between the current scene representation vector and the scene representation vector of each high-value scene template in the preset high-value scene template set.

[0149] The vehicle can pre-build a set of high-value scenario templates. The set of high-value scenario templates contains one or more high-value scenario templates, such as "right vehicle forcibly cutting in" and "unprotected left turn conflict". Each high-value scenario template in the set of high-value scenario templates corresponds to a scenario representation vector.

[0150] Obtain the current scene representation vector x t Then, the current scene representation vector x can be calculated. t The scene representation vector p of each high-value scene template in the preset high-value scene template set k The similarity. Optionally, the current scene representation vector x can be calculated. t The scene representation vector p of each high-value scene template in the preset high-value scene template set k The cosine similarity. A similarity set can be obtained by calculation. K represents the number of high-value scenario templates in the high-value scenario template set.

[0151] Step S303: Determine whether the maximum similarity among the calculated similarities is greater than the preset similarity threshold.

[0152] In the calculated similarity set Take the maximum similarity Determine whether the maximum similarity is greater than the preset similarity threshold. If the maximum similarity is greater than the preset similarity threshold, proceed to step S304a; otherwise, proceed to step S304b.

[0153] Step S304a: Determine that the second triggering condition is met.

[0154] If the maximum similarity among the calculated similarities is greater than the preset similarity threshold, then the current scene is determined to match the preset high-value scene template (the current scene is a high-value scene), which satisfies the second triggering condition.

[0155] Step S304b: Determine that the second triggering condition is not met.

[0156] If the maximum similarity among the calculated similarities is less than or equal to the preset similarity threshold, it is determined that the current scene does not match the preset high-value scene template (the current scene is not a high-value scene), that is, the second triggering condition is not met.

[0157] As mentioned in the above embodiments, the roadside can also determine in real time whether the second triggering condition (the current scene matches a preset high-value scene template) is met, and upload the roadside sensor data for the target time period to the cloud when the second triggering condition is determined to be met. It should be noted that the method by which the roadside determines whether the second triggering condition is met is the same as the method by which the vehicle determines whether the second triggering condition is met, and will not be described in detail here.

[0158] After obtaining vehicle-mounted sensor data and roadside sensor data for the target time period, the cloud can reconstruct a three-dimensional scene based on the vehicle-mounted sensor data and roadside sensor data for the target time period, and obtain a scene sample characterized by dynamic occupancy grid and traffic participant trajectory set. This process is described in some embodiments of the present invention.

[0159] like Figure 4 As shown, the process by which the cloud reconstructs a 3D scene based on vehicle-side sensor data and roadside sensor data for the target time period, and obtains scene samples characterized by dynamic occupancy grids and traffic participant trajectory sets, can include:

[0160] Step S401: Synchronize the vehicle-side sensor data and the roadside sensor data for the target time period to obtain the time-synchronized sensor data.

[0161] Vehicle-mounted and roadside equipment collect data independently. Due to clock drift or transmission delay, the data from the two may deviate on the time axis (usually by tens of milliseconds or even seconds). To ensure the accuracy of subsequent fusion, time synchronization is performed first. By aligning the vehicle-mounted sensor data (such as camera images, LiDAR point clouds, and millimeter-wave radar target lists) with the roadside sensor data (such as roadside camera images and radar data) to the same time reference, the time error between the two is controlled within a preset time range (such as within 10 milliseconds).

[0162] In this embodiment, a preset time synchronization method can be used to synchronize the vehicle-mounted sensor data and the roadside sensor data for the target time period. The preset time synchronization method can be based on the IEEE 1588 Precise Time Protocol (PTP). However, this embodiment is not limited to this; for example, the preset time synchronization method can also be based on a time synchronization method using joint timestamp calibration of GNSS (Global Navigation Satellite System) and IMU (Inertial Measurement Unit).

[0163] Step S402: Spatially align the time-synchronized sensor data to obtain spatially aligned sensor data.

[0164] Vehicle-mounted and roadside equipment are typically located in different physical locations and use different local coordinate systems (vehicle-mounted equipment uses its own vehicle as the origin, while roadside equipment uses a fixed world coordinate system as the origin). In order to merge data within the same spatial framework, spatial alignment (i.e., coordinate system one) is required.

[0165] By solving for an optimal rigid body transformation, the vehicle-side sensor data can be processed. Data from roadside sensors Transforming the data to the same coordinate system, for example, transforming the vehicle-mounted sensor data to the roadside's global coordinate system, minimizes the registration error.

[0166] (2).

[0167] Here, T represents a rigid body transformation, which typically consists of a rotation matrix and a translation vector. This represents the optimal rigid transformation.

[0168] The optimal rigid body transformation between the vehicle-side coordinate system and the roadside global coordinate system can be solved using algorithms such as Iterative Closest Point (ICP) and semantic feature matching. This enables spatial alignment of data from vehicle-mounted and roadside sensors.

[0169] After coordinate transformation, the vehicle-mounted sensor data is placed in the same global coordinate system as the roadside sensor data, and the spatial error can be controlled within the centimeter level.

[0170] Step S403: Based on the spatially aligned sensor data, reconstruct the 3D scene using a neural radiation field or a multi-view fusion model based on a bird's-eye view transformer to obtain scene samples characterized by dynamic occupancy grids and traffic participant trajectory sets.

[0171] After time synchronization and spatial alignment are completed, the cloud can obtain multi-view, multi-modal sensor data (vehicle images, point clouds, roadside images, radar data, etc.), and then use this data to reconstruct 3D scenes.

[0172] In one possible implementation, a 3D scene can be reconstructed using neural radiation fields (NeRF) based on time-synchronized and spatially aligned sensor data. Specifically, the scene is implicitly represented in 3D using NeRF, and multi-frame temporal information is fused to model the motion of dynamic objects. Furthermore, detection results at different times (boundaries, types, etc. of traffic participants) are correlated, that is, the detection results of the same target at different time points (or different frames) are concatenated to form a continuous trajectory for each traffic participant.

[0173] In another possible implementation, a 3D scene can be reconstructed using a multi-view fusion model based on a bird's-eye view transformer, based on time-synchronized and spatially aligned sensor data. Specifically, using the Transformer-based BEVFormer model, image features from multiple cameras are projected onto a unified bird's-eye view (BEV) space. After fusing temporal information, a dynamic occupancy grid and the trajectory of each traffic participant are output.

[0174] Dynamic occupancy grid is a spatiotemporal raster representation where each grid cell records the probability of that location being occupied at different times (distinguishing between static obstacles and dynamic objects). The traffic participant trajectory set contains the temporal bounding box sequence of each traffic participant, including attributes such as position, speed, and orientation. Together, these two constitute a scene sample, which serves as the basic data for subsequent semantic annotation and model training.

[0175] After obtaining scene samples, semantic information can be annotated to generate training data with supervised labels. Specifically, the semantic information annotated for scene samples may include, but is not limited to, one or more of the following: traffic participant bounding boxes (the position, size, and orientation of each traffic participant, such as vehicles, pedestrians, and cyclists, in three-dimensional space), traffic participant behavior categories (the movement intentions or actions of each traffic participant, including but not limited to stopping, turning left, turning right, going straight, yielding, and cutting in), traffic light status (the color (red, yellow, green), arrow direction, and countdown value of each signal light group), and drivable areas (such as areas where vehicles can legally pass).

[0176] In order to expand the scale of training data and improve the generalization ability of the scene perception model, in some embodiments of the present invention, after scene reconstruction and semantic annotation are completed, controllable data augmentation and generalization generation steps can be further performed.

[0177] Specifically, using a conditional generation model, a reconstructed scene sample, a random seed, and control conditions representing environmental attributes are taken as input to generate variant scenes of the reconstructed scene sample, resulting in variant scene samples. Subsequently, scene semantic information is annotated on these variant scene samples to obtain variant scene samples annotated with scene semantic information. Assuming the scene sample is S, then the variant scenes of scene sample S are... for:

[0178] (3).

[0179] In the above formula, Let c be a conditional generative model, c be the control condition, and z be the random seed.

[0180] The control conditions may include, but are not limited to, at least one of the following environmental attribute parameters: weather type indicating the weather state of the scene, illumination period indicating the illumination state of the scene, and traffic density indicating the density of traffic participants in the scene. For example, control condition c = [weather type, illumination period, traffic density], where weather type ∈ {sunny, rainy, foggy}, time ∈ {daytime, nighttime}, and traffic density ∈ [0,1].

[0181] The random seed z ~ N(0, I) means that z follows a standard normal distribution. In the context of the conditional generative model, z serves as a random seed to introduce controllable random variations, so that the model can still generate diverse variant scenes when given the same scene samples and control conditions c.

[0182] It should be noted that the generated results output by the conditional generation model maintain physical consistency. For example, under rainy conditions, point cloud data exhibits sparse characteristics, and under nighttime conditions, image data exhibits light attenuation characteristics. This setting can avoid the distribution shift problem caused by data augmentation and ensure that the generated scene is consistent with the physical laws of the real scene.

[0183] Through the controllable generation process described above, a large number of diverse variant scene samples that conform to physical laws can be obtained. Scene semantic information is then annotated on the obtained variant scene samples to obtain variant scene samples annotated with scene semantic information.

[0184] After obtaining the original reconstructed scene samples (already labeled) and the aforementioned variant scene samples (already labeled), the scene perception model is trained using both types of labeled samples, which can improve the model's generalization ability and robustness.

[0185] In some embodiments of the present invention, the process of training a scene perception model using scene samples labeled with scene semantic information (including original reconstructed scene samples, and possibly variant scene samples of the original reconstructed scene samples) is described.

[0186] like Figure 5 As shown, the process of training a scene perception model using scene samples labeled with scene semantic information can include:

[0187] Step S501: Input the scene sample into the current scene perception model to obtain the first output distribution corresponding to the scene sample, and input the scene sample into the previously trained scene perception model to obtain the second output distribution corresponding to the scene sample.

[0188] The previous version of the scene perception model has been trained on historical scene samples.

[0189] Scene samples are input into the current scene perception model (i.e., the scene perception model currently being trained) to obtain the first output distribution (i.e., the semantic category probability distribution predicted by the model) corresponding to the scene samples. Simultaneously, the same scene samples are input into the previously trained scene perception model to obtain the second output distribution. The previous version of the scene perception model has already been trained on historical scene samples, and its parameters are frozen during this training round to provide a reference for old knowledge.

[0190] Step S502a: Calculate the learning loss for the new scene based on the first output distribution corresponding to the scene sample and the scene semantic information labeled with the scene sample.

[0191] Based on the first output distribution corresponding to the scene sample (i.e., the output distribution of the current scene perception model for the scene sample) and the semantic annotation information of the scene sample, the new scene learning loss is calculated. The new scene learning loss can be the cross-entropy loss function, which is used to drive the current scene perception model to make correct predictions on the new scene sample.

[0192] Step S502b: Calculate the knowledge distillation loss based on the first output distribution and the second output distribution.

[0193] Based on the first output distribution and the second output distribution corresponding to the scene sample, the knowledge distillation loss is calculated. Optionally, the KL divergence between the first output distribution and the second output distribution corresponding to the scene sample (KL divergence measures the difference between the two distributions) can be calculated as the knowledge distillation loss.

[0194] Step S503: Update the parameters of the current scene perception model based on the new scene learning loss and knowledge distillation loss.

[0195] Specifically, the new scene learning loss and knowledge distillation loss are fused to obtain the total loss. The parameters of the current scene perception model are then updated based on the total loss (the goal is to minimize the total loss). It should be noted that the parameters of the previous version of the scene perception model are frozen during this training round.

[0196] Optionally, the new scene learning loss and knowledge distillation loss can be fused in the manner shown in the following formula:

[0197] (4).

[0198] Where θ represents the model parameters. Indicates the learning loss in the new scenario. This represents the first output distribution corresponding to scene sample x. This represents the second output distribution corresponding to scene sample x. This represents the knowledge distillation loss. Let λ represent the total loss, and λ be the hyperparameter that balances the two losses.

[0199] It should be noted that this embodiment is not limited to using the above method to fuse the new scene learning loss and the knowledge distillation loss. For example, weights can be set for the new scene learning loss and the knowledge distillation loss respectively, and then a weighted summation fusion method can be used to fuse the new scene learning loss and the knowledge distillation loss to obtain the total loss.

[0200] In some embodiments of the present invention, the vehicle-side uses scene samples labeled with scene semantic information (after training the scene perception model, it can perform scene sample value evaluation and scene capture trigger word set update process).

[0201] like Figure 6 As shown, the process of scene sample value assessment and scene capture trigger word set update may include:

[0202] Step S601: Determine the data value score of the scene sample.

[0203] Among them, the data value score represents the degree of contribution of scene samples to the training of scene perception model.

[0204] The cloud performs a quantitative evaluation of scene samples (scene samples reconstructed from sensor data after spatiotemporal alignment based on the target time period) and calculates their data value score. The data value score is used to measure the degree of contribution of scene samples to the training of scene perception models, that is, the "usefulness" or "scarcity" of scene samples. The higher the data value score of a scene sample, the more the scene sample can improve the performance of the scene perception model.

[0205] In one possible implementation, the process of determining the data value score of a scene sample may include: obtaining the performance gain evaluation value of the trained scene perception model; obtaining the coverage evaluation value of the scene sample relative to an existing scene sample library; obtaining the redundancy evaluation value of the scene sample relative to an existing scene sample library; and determining the data value score of the scene sample based on the performance gain evaluation value, coverage evaluation value, and redundancy evaluation value.

[0206] Among them, the performance gain evaluation value evaluates the performance improvement of the scene perception model after training the scene perception model with scene samples; the coverage evaluation value evaluates the degree of coverage of the scene samples with the existing scene sample library; and the redundancy evaluation value evaluates the similarity between the scene samples and the scene samples in the existing scene sample library. If there is a high degree of repetition with the existing scene samples, the redundancy is high.

[0207] After obtaining the performance gain assessment value, coverage assessment value, and redundancy assessment value, these values ​​can be merged. In one possible implementation, the performance gain assessment value, coverage assessment value, and redundancy assessment value can be merged in the manner shown in the following formula:

[0208] V(D) = ∆mAP val +α·Coverage(D) - β·MMD(D,D archive (5).

[0209] Where, ∆mAP val MMD(D,D) is the performance gain evaluation value, Coverage(D) is the coverage evaluation value, and MMD(D,D) is the coverage evaluation value. archive V(D) is the redundancy assessment value, V(D) is the data value score of the scene sample, α is the weight of the coverage assessment value, and β is the weight of the redundancy assessment value.

[0210] Step S602: Determine whether the data value score of the scene sample is greater than the preset score threshold.

[0211] If the data value score of the scene sample is greater than the preset score threshold, then step S603 is executed.

[0212] Step S603: If the target time is the time when the vehicle detects the user command that indicates the triggering of scene capture, then extract keywords related to scene capture from the user command.

[0213] It should be noted that if the target time is the time when the user command that triggers the scene capture is given, the vehicle will send the user command and the vehicle sensor data together to the cloud when sending the vehicle sensor data for the target time period (if the user command is a voice command, the voice command or the recognized text of the voice command can be sent).

[0214] When the source of scene collection is a user command and the data value score of the scene sample is high, keywords related to scene capture are extracted from the user command.

[0215] Step S604: Send an update command carrying keywords to the vehicle terminal.

[0216] The cloud-based system encapsulates the extracted keywords into an update command and sends it to the vehicle.

[0217] The update instruction is used to instruct the vehicle to update the set of trigger words for scene capture based on keywords.

[0218] After receiving the update command from the cloud, the vehicle parses the update command, extracts the keywords it carries, and updates the local scene capture trigger word set based on the extracted keywords, so as to realize the online evolution of the scene capture trigger word set.

[0219] The solution provided by this invention constitutes a closed-loop data value feedback mechanism: acquiring high-value scenario samples → evaluating their value → if the value is high and triggered by user voice commands, extracting new keywords → sending them to the vehicle → updating the scenario capture trigger word set on the vehicle → improving the accuracy and coverage of future user voice command triggers. This closed-loop data value feedback mechanism enables the system to learn from users' real language feedback, automatically enriching the scenario capture trigger word set, reducing manual maintenance costs, and continuously improving the capture efficiency of high-value scenarios.

[0220] Optionally, if the data value score of a scene sample is greater than a preset score threshold, the scene sample can be added to a high-value sample library for subsequent incremental model training.

[0221] The scene perception training data acquisition method provided in this embodiment of the invention has the following advantages:

[0222] Firstly, the accuracy and completeness of high-value scenario collection. Through a dual-trigger mechanism combining user command triggering and scenario matching triggering, it can capture both long-tail scenarios that users subjectively consider important (such as abnormal driving behavior and sudden danger) and automatically discover objective high-value scenarios that meet engineering definitions (such as forced entry and unprotected left turn conflicts). This achieves complementarity between subjective and objective factors, avoids scenario omissions caused by a single triggering method, and significantly improves the coverage and representativeness of training samples.

[0223] Secondly, the vehicle-road-cloud collaborative three-dimensional perception and reconstruction capabilities. By integrating multi-source data from vehicle-side sensors and roadside sensing devices, and utilizing time synchronization and spatial alignment technologies, vehicle-side sensing data and roadside sensing data are unified to the same spatiotemporal reference, thereby reconstructing high-quality scene samples and compensating for the blind spots and limited perspectives of single-end perception.

[0224] Third, automatic semantic annotation and controllable data augmentation. Multi-dimensional semantic information (traffic participant bounding boxes, traffic participant behavior categories, traffic light status, and drivable areas) is annotated on the reconstructed scene samples to directly generate labeled training data, reducing the cost of manual annotation. At the same time, physically consistent variant scene samples are generated based on conditional generation models (controlling weather, lighting, traffic density, etc.), significantly expanding the data scale and improving the model's generalization ability to different environments.

[0225] Fourth, model evolution and deployment under the continuous learning framework. A continuous learning strategy with knowledge distillation regularization is adopted. When introducing new scenario samples for training, the KL divergence constraint model is used to constrain the forgetting of old knowledge, so as to achieve balanced learning of new and old scenarios. After training, the model is pushed to the vehicle, forming a closed loop of "data collection → cloud training → vehicle deployment → new data triggering → model iteration", which enables the scenario perception model to continuously evolve with the fleet of vehicles.

[0226] Fifth, adaptive closed-loop optimization of the scene capture trigger word set. Determine the data value score (performance gain, coverage, redundancy) of scene samples. When the data value score is greater than the preset score threshold, extract keywords from user commands and send them to the vehicle to update the scene capture trigger word set, continuously improving the accuracy of voice triggering and the scene coverage.

[0227] In summary, this invention has significant advantages in terms of data acquisition quality, fusion and reconstruction accuracy, training data richness, model continuous evolution capability, and adaptive triggering mechanism.

[0228] This invention also provides a scene perception training data acquisition device applied in the cloud, such as... Figure 7 As shown, the device may include: a sensor data receiving module 701, a 3D scene reconstruction module 702, a sample annotation module 703, a model training module 704, and a model data sending module 705.

[0229] The sensor data receiving module 701 is used to receive vehicle-mounted sensor data for the target time period uploaded by the vehicle and roadside sensor data for the target time period uploaded by the roadside.

[0230] The target time period is a continuous time period covering the target time. The target time is the moment when the vehicle detects the user command that indicates the triggering of scene capture, or the moment when the vehicle or roadside detects that the current scene matches a preset high-value scene.

[0231] The 3D scene reconstruction module 702 is used to reconstruct a 3D scene based on vehicle-mounted sensor data and roadside sensor data during the target time period, and obtain scene samples characterized by dynamic occupancy grid and traffic participant trajectory set.

[0232] The sample annotation module 703 is used to annotate scene samples with scene semantic information to obtain scene samples annotated with scene semantic information.

[0233] In one possible implementation, the scene perception training data acquisition device provided in this embodiment may further include: a model training module 704 and a model data sending module 705.

[0234] The model training module 704 is used to train the scene perception model using scene samples labeled with scene semantic information.

[0235] The model data sending module 705 is used to send the trained scene perception model to the vehicle for deployment.

[0236] In one possible implementation, when the 3D scene reconstruction module 702 reconstructs a 3D scene based on vehicle-mounted sensor data and roadside sensor data for the target time period, and obtains scene samples characterized by a dynamic occupancy grid and a set of traffic participant trajectories, it is specifically used for:

[0237] The vehicle-mounted sensor data and roadside sensor data for the target time period are synchronized in time to obtain time-synchronized sensor data. The time-synchronized sensor data is then spatially aligned to obtain spatially aligned sensor data. Based on the spatially aligned sensor data, a 3D scene is reconstructed using a neural radiation field or a multi-view fusion model based on a bird's-eye view transformer, resulting in scene samples characterized by dynamic occupancy grids and traffic participant trajectory sets.

[0238] In one possible implementation, the scene-aware training data acquisition device provided in this embodiment may further include a sample augmentation module.

[0239] The sample enhancement module is used to generate variant scenes of the scene sample by taking the scene sample, random seed and control conditions representing environmental attributes as input using the conditional generation model, thus obtaining variant scene samples.

[0240] The sample annotation module 703 is also used to annotate the variant scene samples with scene semantic information to obtain variant scene samples annotated with scene semantic information.

[0241] When training the scene perception model using scene samples labeled with scene semantic information, the model training module 704 is specifically used to train the scene perception model using scene samples labeled with scene semantic information and variant scene samples labeled with scene semantic information.

[0242] In one possible implementation, the random seed follows a standard normal distribution, and the control conditions include at least one of the following environmental attribute parameters: weather type indicating the weather state of the scene, illumination period indicating the illumination state of the scene, and traffic density indicating the density of traffic participants in the scene.

[0243] In one possible implementation, when the model training module 704 trains the scene perception model using scene samples labeled with scene semantic information, it is specifically used for:

[0244] The scene samples are input into the current scene perception model to obtain the first output distribution corresponding to the scene samples, and the scene samples are input into the previously trained scene perception model to obtain the second output distribution corresponding to the scene samples. The previous version of the scene perception model has been trained on historical scene samples. Based on the first output distribution corresponding to the scene samples and the scene semantic information labeled on the scene samples, the new scene learning loss is calculated. Based on the first output distribution and the second output distribution, the knowledge distillation loss is calculated. Based on the new scene learning loss and the knowledge distillation loss, the parameters of the current scene perception model are updated.

[0245] In one possible implementation, the scene perception training data acquisition device provided in this embodiment may further include: a sample value scoring determination module, a keyword extraction module, and an update instruction sending module.

[0246] The sample value scoring module is used to determine the data value score of scene samples, where the data value score represents the degree of contribution of scene samples to the training of scene perception models.

[0247] The keyword extraction module is used to extract keywords related to scene capture from the user command if the data value score of the scene sample is greater than the preset score threshold and the target time is the time when the vehicle detects the user command that triggers scene capture. The user command is attached to the vehicle sensor data received from the vehicle.

[0248] The update instruction sending module is used to send update instructions carrying keywords to the vehicle-side terminal. These instructions instruct the terminal to update the scene capture trigger word set based on the keywords. The vehicle-side terminal then uses the scene capture trigger word set to decide whether to detect a user instruction to trigger scene capture.

[0249] In one possible implementation, the sample value scoring module, when determining the data value score of a scene sample, is specifically used for:

[0250] Obtain the performance gain evaluation value of the trained scene perception model; obtain the coverage evaluation value of the scene samples relative to the existing scene sample library; obtain the redundancy evaluation value of the scene samples relative to the existing scene sample library; determine the data value score of the scene samples based on the performance gain evaluation value, coverage evaluation value, and redundancy evaluation value.

[0251] In one possible implementation, the scene perception training data acquisition device provided in this embodiment may further include a sample processing module.

[0252] The sample processing module is used to add scene samples to a high-value sample library when the data value score of a scene sample is greater than a preset score threshold, for subsequent incremental model training.

[0253] This invention also provides a device for acquiring scene perception training data for vehicle applications, such as... Figure 8 As shown, the device may include: a vehicle-side sensor data acquisition module 801, a scene capture triggering and discrimination module 802, a vehicle-side sensor data interception module 803, and a vehicle-side sensor data transmission module 804.

[0254] The vehicle-side sensor data acquisition module 801 is used to acquire vehicle-side sensor data in real time.

[0255] The scene capture triggering and discrimination module 802 is used to determine in real time whether the first triggering condition or the second triggering condition is met. The first triggering condition is that the user issues an instruction to trigger scene capture, and the second triggering condition is that the current scene matches a preset high-value scene template.

[0256] The vehicle-side sensor data interception module 803 is used to intercept vehicle-side sensor data for a target time period from the acquired vehicle-side sensor data when a first trigger condition or a second trigger condition is met. The target time period is a continuous time period covering a target moment, and the target moment is the moment when the first trigger condition or the second trigger condition is met.

[0257] The vehicle-side sensor data transmission module 804 is used to upload vehicle-side sensor data for the target time period to the cloud and notify the roadside to upload roadside sensor data for the target time period to the cloud. This allows the cloud to reconstruct a 3D scene based on the vehicle-side sensor data and roadside sensor data for the target time period. After annotating the reconstructed scene samples with scene semantic information, the cloud uses the annotated scene samples to train a scene perception model.

[0258] The scene perception training data acquisition device provided in this embodiment of the invention may further include: a model data receiving module 805 and a model deployment module 806.

[0259] The model data receiving module 805 is used to receive the trained scene perception model sent from the cloud.

[0260] Model deployment module 806 is used to deploy the trained scene-aware model.

[0261] In one possible implementation, the scene capture trigger discrimination module 802, when determining whether the first trigger condition is met, is specifically used for:

[0262] Collect the natural language speech emitted by the user, perform speech recognition on the natural language speech to obtain the recognized text; call the fine-tuned natural language understanding model, combine it with the scene capture trigger word set, and determine whether the recognized text is an instruction to trigger scene capture; if the recognized text is an instruction to trigger scene capture, it is determined that the first trigger condition is met.

[0263] In one possible implementation, the scene capture trigger discrimination module 802, when determining whether the second trigger condition is met, is specifically used for:

[0264] Based on the acquired vehicle-side sensor data, the current scene representation vector is obtained, where the current scene representation vector represents the current dynamic traffic environment and the motion state of traffic participants; the similarity between the current scene representation vector and the scene representation vector of each high-value scene template in the preset high-value scene template set is calculated; if the maximum similarity among the calculated similarities is greater than the preset similarity threshold, it is determined that the second triggering condition is met.

[0265] In one possible implementation, the scene perception training data acquisition device provided in this embodiment may further include: a trigger word set update module.

[0266] The trigger word set update module is used to extract keywords from the update command sent by the cloud when it receives an update command that carries keywords related to scene capture, and update the scene capture trigger word set based on the extracted keywords.

[0267] This invention also provides a scene perception training data acquisition device for roadside applications, such as... Figure 9 As shown, it may include: a roadside sensor data acquisition module 901, a scene capture triggering and discrimination module 902, a roadside sensor data interception module 903, and a roadside sensor data transmission module 904.

[0268] The roadside sensor data acquisition module 901 is used to acquire roadside sensor data in real time.

[0269] The scene capture trigger discrimination module 902 is used to determine in real time whether the second trigger condition is met. The second trigger condition is that the current scene matches a preset high-value scene template.

[0270] The roadside sensor data interception module 903 is used to intercept roadside sensor data for a target time period from the acquired roadside sensor data when a second triggering condition is met. The target time period is a continuous time period covering a target time, and the target time is the moment when the second triggering condition is met.

[0271] The roadside sensor data transmission module 904 is used to upload roadside sensor data for the target time period to the cloud and notify the vehicle to upload vehicle sensor data for the target time period to the cloud. The cloud can then reconstruct a 3D scene based on the vehicle sensor data and roadside sensor data for the target time period, and after annotating the reconstructed scene samples with scene semantic information, use the annotated scene samples to train a scene perception model.

[0272] This invention also provides a cloud server, which may include at least one processor and a memory connected to the processor.

[0273] The processor may be a central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement embodiments of the present invention; the memory may include high-speed RAM, and may also include non-volatile memory, such as at least one disk storage.

[0274] The memory is used to store computer programs, and the processor is used to execute the computer programs, so that the cloud server can implement the scene perception training data acquisition method provided in the above embodiments.

[0275] This invention also provides an autonomous driving vehicle, which may include at least one processor and a memory connected to the processor.

[0276] The processor may be a central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement embodiments of the present invention; the memory may include high-speed RAM, and may also include non-volatile memory, such as at least one disk storage.

[0277] The memory is used to store computer programs, and the processor is used to execute the computer programs, so that the autonomous vehicle can implement the scene perception training data acquisition method provided in the above embodiments.

[0278] This invention also provides a roadside device, which may include at least one processor and a memory connected to the processor.

[0279] The processor may be a central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement embodiments of the present invention; the memory may include high-speed RAM, and may also include non-volatile memory, such as at least one disk storage.

[0280] The memory is used to store computer programs, and the processor is used to execute the computer programs so that the roadside equipment can implement the scene perception training data acquisition method provided in the above embodiments.

[0281] This invention also provides a computer storage medium carrying one or more computer programs. When the one or more computer programs are executed by an electronic device, the electronic device is able to implement the scene perception training data acquisition method provided in the above embodiments.

[0282] This invention also provides a computer program product, including computer-readable instructions, which, when executed on an electronic device, enable the electronic device to implement the scene perception training data acquisition method provided in the above embodiments.

[0283] This invention also provides an autonomous driving intelligent agent system, such as... Figure 10 As shown, the system may include a vehicle-side intelligent agent 1001, a roadside intelligent agent 1002, and a cloud-based intelligent agent 1003.

[0284] The vehicle-side intelligent agent 1001 can communicate with the roadside intelligent agent 1002 and the cloud-based intelligent agent 1003, and the roadside intelligent agent 1002 can communicate with the vehicle-side intelligent agent 1001 and the cloud-based intelligent agent 1003.

[0285] The vehicle-side intelligent agent 1001 is used to execute the method steps executed by the vehicle in the above method embodiments.

[0286] The roadside agent 1002 is used to execute the method steps performed by the roadside in the above method embodiments.

[0287] The cloud-based intelligent agent 1003 is used to execute the method steps executed by the cloud in the above method embodiments.

[0288] It should also be noted that the device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. In addition, in the accompanying drawings of the device embodiments provided by this invention, the connection relationship between modules indicates that they have a communication connection, which can be implemented as one or more communication buses or signal lines.

[0289] Through the above description of the embodiments, those skilled in the art can clearly understand that the present invention can be implemented by means of software plus necessary general-purpose hardware, and of course, it can also be implemented by special-purpose hardware including application-specific integrated circuits, special-purpose CPUs, special-purpose memory, special-purpose components, etc. Generally, any function performed by a computer program can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can also be diverse, such as analog circuits, digital circuits, or special-purpose circuits. However, for the present invention, software program implementation is more often the preferred implementation method. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a readable storage medium, such as a computer floppy disk, USB flash drive, mobile hard disk, ROM, RAM, magnetic disk, or optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, training equipment, or network device, etc.) to execute the methods described in the various embodiments of the present invention.

[0290] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product.

[0291] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of the present invention are generated. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions may be transmitted from one website, computer, training device, or data center to another website, computer, training device, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a training device or data center that integrates one or more available media. The available media may be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., DVDs), or semiconductor media (e.g., solid-state drives (SSDs)).

Claims

1. A method for acquiring scene-aware training data, characterized in that, Applied to the cloud, the method includes: The system receives vehicle-mounted sensor data for the target time period uploaded by the vehicle and roadside sensor data for the target time period uploaded by the roadside. The target time period is a continuous time period covering the target moment, and the target moment is the moment when the vehicle detects a user command to trigger scene capture, or the moment when the vehicle or roadside detects that the current scene matches a preset high-value scene. A 3D scene is reconstructed based on vehicle-mounted sensor data and roadside sensor data during the target time period, resulting in a scene sample characterized by a dynamic occupancy grid and a set of traffic participant trajectories. The scene samples are annotated with scene semantic information to obtain scene samples annotated with scene semantic information. The scene samples annotated with scene semantic information are used to train the scene perception model.

2. The method for acquiring scene perception training data according to claim 1, characterized in that, The process of reconstructing a 3D scene based on vehicle-mounted sensor data and roadside sensor data during the target time period, resulting in scene samples characterized by dynamic occupancy grids and traffic participant trajectory sets, includes: The vehicle-mounted sensor data and the roadside sensor data for the target time period are synchronized in time to obtain time-synchronized sensor data. The time-synchronized sensor data is spatially aligned to obtain spatially aligned sensor data. Based on the spatially aligned sensor data, a dynamic occupancy grid and a set of traffic participant trajectories are generated using a neural radiation field or a multi-view fusion model based on a bird's-eye view transformer, resulting in scene samples characterized by the dynamic occupancy grid and the set of traffic participant trajectories.

3. The method for acquiring scene perception training data according to claim 1, characterized in that, Also includes: Using a conditional generation model, with the scene sample, random seed, and control conditions representing environmental attributes as input, a variant scene of the scene sample is generated to obtain a variant scene sample. The variant scene samples are annotated with scene semantic information to obtain variant scene samples annotated with scene semantic information. The scene samples annotated with scene semantic information and the variant scene samples annotated with scene semantic information are used to train the scene perception model.

4. The method for acquiring scene perception training data according to claim 3, characterized in that, The random seed follows a standard normal distribution; The control conditions include at least one of the following environmental attribute parameters: Weather type used to indicate the weather conditions of a scene; Lighting periods used to indicate the lighting conditions of a scene; Traffic density is used to indicate the density of traffic participants in a scene.

5. The method for acquiring scene perception training data according to claim 1, characterized in that, Also includes: The scene perception model is trained using the scene samples labeled with scene semantic information, and the trained scene perception model is sent to the vehicle for deployment. The step of training the scene perception model using the scene samples labeled with scene semantic information includes: The scene sample is input into the current scene perception model to obtain the first output distribution corresponding to the scene sample, and the scene sample is input into the previous version scene perception model that has been trained to obtain the second output distribution corresponding to the scene sample. The previous version scene perception model has been trained on historical scene samples. Calculate the new scene learning loss based on the first output distribution corresponding to the scene sample and the scene semantic information labeled by the scene sample; Calculate the knowledge distillation loss based on the first output distribution and the second output distribution; The parameters of the current scene perception model are updated based on the new scene learning loss and the knowledge distillation loss.

6. The method for acquiring scene perception training data according to claim 1, characterized in that, Also includes: Determine the data value score of the scene sample, wherein the data value score characterizes the degree of contribution of the scene sample to the training of the scene perception model; If the data value score of the scene sample is greater than the preset score threshold, and if the target time is the time when the vehicle detects a user instruction to trigger scene capture, then keywords related to scene capture are extracted from the user instruction, wherein the vehicle sensor data received from the vehicle is accompanied by the user instruction. An update instruction carrying the keyword is sent to the vehicle terminal, the update instruction being used to instruct the vehicle terminal to update the scene capture trigger word set based on the keyword; wherein, the vehicle terminal, in conjunction with the scene capture trigger word set, decides whether to detect a user instruction to trigger scene capture.

7. The method for acquiring scene perception training data according to claim 6, characterized in that, The determination of the data value score for the scene sample includes: Obtain the performance gain evaluation value of the trained scene perception model; Obtain the coverage evaluation value of the scene sample relative to the existing scene sample library; Obtain the redundancy assessment value of the scene sample relative to the existing scene sample library; The data value score of the scene sample is determined based on the performance gain evaluation value, the coverage evaluation value, and the redundancy evaluation value.

8. The method for acquiring scene perception training data according to claim 6, characterized in that, Also includes: If the data value score of the scene sample is greater than a preset score threshold, the scene sample will be added to the high-value sample library for subsequent incremental model training.

9. A method for acquiring scene-aware training data, characterized in that, Applied to the vehicle end, the method includes: Real-time acquisition of vehicle-side sensor data, and real-time determination of whether a first trigger condition or a second trigger condition is met. The first trigger condition is a user issuing an instruction to trigger scene capture, and the second trigger condition is the current scene matching a preset high-value scene template. If the first triggering condition or the second triggering condition is met, then vehicle sensor data for the target time period is extracted from the acquired vehicle sensor data. The target time period is a continuous time period covering the target time. The target time is the moment when the first triggering condition or the second triggering condition is met. The vehicle-mounted sensor data for the target time period is uploaded to the cloud, and the roadside is notified to upload the roadside sensor data for the target time period to the cloud. The cloud then reconstructs a 3D scene based on the vehicle-mounted sensor data and the roadside sensor data for the target time period, and annotates the scene semantic information of the reconstructed scene samples represented by dynamic occupancy grids and traffic participant trajectory sets.

10. The method for acquiring scene perception training data according to claim 9, characterized in that, Determine whether the first triggering condition is met, including: Collect natural language speech from the user, perform speech recognition on the natural language speech, and obtain the recognized text; The fine-tuned natural language understanding model is invoked, and combined with the scene capture trigger word set, it is determined whether the recognized text is an instruction that triggers scene capture; If the identified text is an instruction to trigger scene capture, then the first triggering condition is determined to be met.

11. The method for acquiring scene-aware training data according to claim 9, characterized in that, Determine whether the second triggering condition is met, including: Based on the acquired vehicle-side sensor data, a current scene representation vector is obtained, wherein the current scene representation vector represents the current dynamic traffic environment and the motion state of traffic participants; Calculate the similarity between the current scene representation vector and the scene representation vector of each high-value scene template in the preset high-value scene template set; If the maximum similarity among the calculated similarities is greater than the preset similarity threshold, then the second triggering condition is satisfied.

12. The method for acquiring scene perception training data according to claim 10, characterized in that, Also includes: When an update instruction containing keywords related to scene capture is received from the cloud, the keywords are extracted from the update instruction; The set of trigger words for scene capture is updated based on the keywords.

13. A method for acquiring scene-aware training data, characterized in that, Applied to the roadside, the method includes: Real-time acquisition of roadside sensor data, and real-time determination of whether the second triggering condition is met, wherein the second triggering condition is that the current scene matches a preset high-value scene template; If the second triggering condition is met, then the roadside sensor data for the target time period is extracted from the acquired roadside sensor data. The target time period is a continuous time period covering the target time, and the target time is the time when the second triggering condition is met. The roadside sensor data for the target time period is uploaded to the cloud, and the vehicle is notified to upload its own sensor data for the target time period to the cloud. The cloud then reconstructs a 3D scene based on the vehicle-side sensor data and the roadside sensor data for the target time period, and annotates the scene semantic information of the reconstructed scene samples represented by dynamic occupancy grids and traffic participant trajectory sets.

14. A cloud server, characterized in that, It includes at least one processor and a memory connected to the processor, wherein: The memory is used to store computer programs; The processor is used to execute the computer program to enable the cloud server to implement the scene perception training data acquisition method as described in any one of claims 1 to 8.

15. An autonomous vehicle, characterized in that, It includes at least one processor and a memory connected to the processor, wherein: The memory is used to store computer programs; The processor is used to execute the computer program to enable the autonomous vehicle to implement the scene perception training data acquisition method as described in any one of claims 9 to 12.

16. A roadside device, characterized in that, It includes at least one processor and a memory connected to the processor, wherein: The memory is used to store computer programs; The processor is used to execute the computer program to enable the roadside device to implement the scene perception training data acquisition method as described in claim 13.

17. An autonomous driving intelligent agent system, characterized in that, include: Cloud-based intelligent agents, vehicle-side intelligent agents, and roadside intelligent agents; The cloud-based intelligent agent is used to execute the scene perception training data acquisition method as described in any one of claims 1 to 8; The vehicle-side intelligent agent is used to execute the scene perception training data acquisition method as described in any one of claims 9 to 12; The roadside agent is used to execute the scene perception training data acquisition method as described in claim 13.

Images