A method for screening a host vehicle responsibility dangerous scene for automatic driving algorithm training

Abstract

The application discloses a kind of main vehicle responsibility dangerous scene screening methods for automatic driving algorithm training.The history high-risk traffic interaction data including dangerous scene text record, position information, scene risk attribute label and responsibility attribution label are obtained;According to the position information, the road satellite image is obtained and the road environment description information is generated, which is integrated with the dangerous scene text record into the standardized scene text prompt word, and the vertical domain fine-tuning sample is constructed according to this, and the basic large language model is fine-tuned, and the dangerous scene risk responsibility attribution model is obtained;The visual language analysis is carried out to the candidate dangerous scene video data, and the structured scene description text is generated and input into the model, and the scene risk attribute, responsibility attribution and prediction confidence are obtained;The candidate dangerous scene of main vehicle responsibility is screened, and the main vehicle responsibility dangerous scene dataset for automatic driving algorithm training is constructed.

Description

Technical Field

[0001] This invention belongs to the field of interdisciplinary technology of artificial intelligence large model and traffic safety, specifically involving a method for screening hazardous scenarios where the vehicle is primarily responsible for the accident during autonomous driving algorithm training. Background Technology

[0002] As autonomous driving systems increasingly operate in complex road environments, the data required for algorithm training is no longer limited to high-frequency scenarios such as regular following and constant-speed cruising. It needs to cover safety-critical interactions such as merging, cutting in, obstruction, intersection conflicts, and non-motorized vehicle mixing. These scenarios have a low probability of occurring in natural driving data, and the costs of collecting, de-identifying, labeling, and reproducing them are high, leading to training data tending to concentrate on normal traffic conditions. Perception, prediction, planning, or control algorithms trained on a limited number of long-tailed samples may lack sufficient specificity when facing low-frequency but high-consequence traffic interactions.

[0003] Existing technology CN120317141A discloses a method for generating autonomous driving scenarios. Based on driving scenario materials in the form of text, images, or videos, it uses a multimodal large model to generate driving scenario description information, further generating road network structures and driving scenario script files. Then, a traffic simulation device and an autonomous driving simulator control traffic facilities, road obstacles, and traffic participants to obtain the autonomous driving scenario. This method can reduce the workload of manually writing simulation scripts and facilitates the expansion of the sources of autonomous driving test scenarios.

[0004] However, the primary goal of the aforementioned technologies is to convert scene materials into runnable simulation scenarios, with the output focusing on road networks, scripts, and the movement of traffic participants. Existing scene generation processes lack stable attribution results regarding whether the generated hazardous states are mainly caused by defects in the perception, prediction, planning, or control of the tested vehicle, or by abnormal actions of surrounding traffic participants. Therefore, many scenarios that can be successfully generated or have a high collision probability are not necessarily effective training scenarios capable of exposing defects in the vehicle's algorithm. For example, if surrounding vehicles suddenly reverse, cut in laterally within a very short distance, brake suddenly without reasonable cause, or actively pursue a collision, even if the vehicle follows normal driving logic in braking and evading, it may still enter a hazardous state due to insufficient reaction time. If scenarios are collected solely based on risk indicators such as collision, minimum distance, or collision time, it is easy to mix and store scenarios primarily caused by surrounding traffic participants with scenarios where the vehicle is primarily responsible, weakening the correspondence between training data and aspects of the vehicle's algorithm that need improvement.

[0005] On the other hand, hazardous scenario data typically includes structured state variables, natural language records, road location information, images or videos, and manual labels. Data fields, time bases, and descriptive granularities vary across different sources, and some records lack environmental information such as the number of lanes, road intersection patterns, or the distribution of surrounding buildings. While general multimodal models can generate natural language descriptions, their output format is open, and category names and expressions may change depending on prompts. When batch-screening scenarios, it is difficult to directly form stable and comparable scenario risk attributes and liability categories, and it is also inconvenient to verify low-confidence results based on output probabilities.

[0006] Existing adversarial scenario construction methods often prioritize increasing risk levels, generating candidate scenarios by resampling the speed, acceleration, lateral position, or lane-changing trajectories of surrounding traffic participants. When the risk objective is not effectively correlated with the rationality of the behavior, road traffic rules, and the responsible party, the training process may repeatedly reinforce conservative strategies such as "early deceleration, frequent braking, and avoiding entering the interaction area as much as possible." While such strategies can reduce some simulated collision indicators, they may sacrifice normal traffic efficiency and obscure the true need for improvement in the driver's decision-making boundaries. Therefore, how to distinguish between driver-responsible dangerous scenarios from natural driving scenarios and simulated adversarial scenarios, and generate training data with unified semantics, traceable labels, and credibility information, remains a technical problem that needs to be solved in the training of autonomous driving algorithms. Summary of the Invention

[0007] The purpose of this invention is to overcome the shortcomings of the existing technology and provide a method for screening hazardous scenarios where the vehicle is primarily responsible for autonomous driving algorithm training.

[0008] The objective of this invention can be achieved through the following technical solutions: This invention provides a method for screening hazardous scenarios where the vehicle is primarily responsible for the accident during autonomous driving algorithm training, comprising the following steps: Acquire historical high-risk traffic interaction data, which includes text records of dangerous scenarios, location information of dangerous scenarios, scenario risk attribute tags, and dangerous scenario responsibility attribution tags; Based on the location information of the dangerous scene, obtain the road satellite image corresponding to the road location of the dangerous scene, and perform visual language parsing on the road satellite image to obtain road environment description information; The dangerous scenario text records and the road environment description information are structurally integrated to generate standardized scenario text prompts corresponding to the dangerous scenario risk responsibility attribution task; Based on the standardized scenario text prompts, scenario risk attribute labels, and hazardous scenario responsibility attribution labels, vertical fine-tuning samples are constructed, and the basic large language model is fine-tuned using the vertical fine-tuning samples to obtain a hazardous scenario risk responsibility attribution model. Acquire candidate hazardous scene video data and perform visual language parsing on the candidate hazardous scene video data to obtain structured scene description text that represents the formation process of hazardous state, the interactive behavior of traffic participants and road environment; The structured scenario description text is input into the dangerous scenario risk responsibility attribution model, which then performs causal analysis of the formation of dangerous states and analysis of the rationality of road traffic rules and behaviors based on the structured scenario description text, and outputs scenario risk attribute determination results, dangerous scenario responsibility attribution results and corresponding prediction confidence scores. From the candidate hazardous scene video data, candidate hazardous scenes with primary vehicle responsibility are selected based on the hazardous scene responsibility attribution results. A primary vehicle responsibility hazardous scene dataset is constructed and used for training autonomous driving algorithms. The primary vehicle is the test vehicle corresponding to the autonomous driving algorithm to be trained.

[0009] Furthermore, the acquisition of historical high-risk traffic interaction data specifically includes: Acquire high-risk traffic interaction records collected by natural driving and / or high-risk traffic interaction records generated by the autonomous driving simulation platform, and extract globally unique scene numbers from the high-risk traffic interaction records; Based on the globally unique scene number, the main vehicle status information, surrounding traffic participants information, dangerous scene text records, scene risk attribute tags and dangerous scene responsibility attribution tags corresponding to the same high-risk traffic interaction scene are associated; Extract the scene occurrence time, scene location, weather information, route name, and GPS coordinate information from the high-risk traffic interaction records to form the location information of the dangerous scene; The host vehicle status information, surrounding traffic participant information, dangerous scene text records, dangerous scene location information, scene risk attribute labels, and dangerous scene responsibility attribution labels are stored according to the globally unique scene number to obtain the historical high-risk traffic interaction data.

[0010] Furthermore, the step of obtaining road satellite images corresponding to the road locations of the hazardous scenes based on the hazardous scene location information, and performing visual language parsing on the road satellite images to obtain road environment description information, specifically includes: Based on the GPS coordinate information in the dangerous scene location information, the map application interface is called to obtain road satellite images within a preset spatial range with the location corresponding to the GPS coordinate information as the center. The road satellite image is input into a visual language parsing model, which includes an image encoding unit, a multimodal feature alignment unit, and a text generation unit. The image encoding unit extracts road visual features from the road satellite image, including lane number features, road marking features, road intersection morphology features, and building distribution features near the road location corresponding to dangerous scenes. The road visual features are semantically aligned with preset road environment parsing prompts by the multimodal feature alignment unit to obtain road environment semantic features related to the attribution of risk responsibility in dangerous scenarios; The text generation unit transforms the semantic features of the road environment into road environment description information, which is used to supplement the missing or incomplete road infrastructure information and road surrounding environment information in the historical high-risk traffic interaction data.

[0011] Furthermore, the step of structurally integrating the hazardous scenario text records and the road environment description information to generate standardized scenario text prompts corresponding to the hazardous scenario risk responsibility attribution task specifically includes: The text records of the dangerous scenarios are parsed to extract the time of occurrence of the scenario, the location of the scenario, weather information, the status information of the main vehicle, the behavior information of surrounding traffic participants, the interaction process information of traffic participants, and the participation information of non-motorized vehicles or pedestrians. The road environment description information is matched with the field information extracted from the dangerous scene text record to obtain a multi-source scene feature set; According to the input requirements of the hazardous scenario risk responsibility attribution task, the multi-source scenario feature set is divided into system task information, scenario basic information, road environment information, traffic participant interaction behavior information, and non-motorized vehicle or pedestrian information. Based on the system task information, system prompts are generated to define the attribution roles and task objectives for risk responsibility in dangerous scenarios. Based on the scenario basic information, basic factual text containing scenario occurrence time, scenario location, and weather information is generated. Based on the road environment information, road environment text containing the number of lanes, road markings, road intersection patterns, and the distribution of surrounding buildings is generated. Based on the traffic participant interaction behavior information, interactive factual text containing the behavior of the main vehicle, the behavior of surrounding traffic participants, and their interaction status is generated. Based on the non-motorized vehicle or pedestrian information, supplementary text for the participating objects is generated. The system prompt words, the basic factual text, the road environment text, the interactive factual text, and the supplementary text of the participating object are concatenated in a preset order, and the concatenated text is semantically coherent to obtain the standardized scene text prompt words.

[0012] Furthermore, the step of constructing vertical fine-tuning samples based on the standardized scenario text prompts, the scenario risk attribute labels, and the hazardous scenario responsibility attribution labels, and then using these vertical fine-tuning samples to fine-tune the basic large language model to obtain the hazardous scenario risk responsibility attribution model, specifically includes: The task of attributing risk responsibility in dangerous scenarios is divided into a task of determining the risk attributes of scenarios and a task of attributing responsibility in dangerous scenarios. Target special tags are added to the word segmenter vocabulary corresponding to the basic large language model. The target special tags include a first set of special tags used to characterize the result of determining the risk attributes of scenarios and a second set of special tags used to characterize the result of attributing responsibility in dangerous scenarios. The standardized scene text prompts are concatenated with the corresponding scene risk attribute target special markers and dangerous scene responsibility attribution target special markers to obtain an autoregressive fine-tuning sequence, which enables the basic large language model to predict the corresponding scene risk attribute judgment results and dangerous scene responsibility attribution results based on the standardized scene text prompts. When fine-tuning the basic large language model in the vertical domain, the backbone network parameters of the basic large language model are frozen, and the input embedding layer, output embedding layer, and low-rank adaptive parameters are set to a trainable state. The parameters of the basic large language model are updated through low-rank adaptation, wherein the weight update satisfies: in, This represents the updated model weights. This represents the frozen pre-trained base weights. Indicates the weight update amount. and This represents the low-rank matrix in low-rank adaptive fine-tuning; During the vertical fine-tuning process, a local loss mask is applied to the autoregressive fine-tuning sequence so that the input text part corresponding to the standardized scene text prompt words does not participate in the loss calculation, and the fine-tuning loss is calculated only for the output part corresponding to the special marker of the scene risk attribute target and the special marker of the dangerous scene responsibility attribution target. Based on the fine-tuning loss, the input embedding layer, the output embedding layer, and the low-rank adaptive parameters are iteratively updated, and the risk responsibility attribution model for dangerous scenarios is obtained after the vertical fine-tuning is completed.

[0013] Furthermore, the first set of special markers includes <Property Damage Risk>, <Personal Injury Risk>, and <Fatal Risk>, while the second set of special markers includes <Primary Liability of the Main Vehicle>, <Primary Liability of the Non-Main Vehicle>, and <Primary Liability of the Non-Motorized Vehicle / Pedestrian>.

[0014] Furthermore, the joint probability of the target output portion in the autoregressive fine-tuning sequence is expressed as: in, This refers to the standardized scenario text prompts. This represents the output tag sequence, which consists of target-specific tags corresponding to the scenario risk attribute determination results and the hazardous scenario responsibility attribution results. Indicates the first in the output tag sequence A special marker for the target. This indicates the number of target-specific tags in the output tag sequence. This represents the model parameters that participate in vertical domain fine-tuning. This indicates the prediction of the first target given the standardized scene text prompts and the preceding target-specific markers. The probability of a target being specially marked.

[0015] Furthermore, the fine-tuning loss is formulated as follows: in, This represents the fine-tuning loss function; This indicates the number of labels in the autoregressive fine-tuning sequence; This represents the first step in the autoregressive fine-tuning sequence. One tag; Indicates the first The context tag preceding the tag; The representation model predicts the first based on context labels. The probability of each label; Indicates the first The loss mask value corresponding to the i-th label, when the i-th label... When a marker is a special marker for a scenario risk attribute target or a special marker for a hazardous scenario responsibility attribution target... When the first When a marker belongs to the input text portion corresponding to the standardized scene text prompt word .

[0016] Furthermore, the candidate dangerous scene video data includes high-risk traffic interaction scene videos extracted from natural driving data, and / or adversarial scene videos obtained by resampling the movement trajectories of surrounding traffic participants through an autonomous driving simulation platform; The motion trajectory resampling, while maintaining the road topology, the basic travel direction of traffic participants, and vehicle dynamics constraints, alters at least one of the speed, acceleration, lateral position, or lane-changing trajectory of surrounding traffic participants to generate the candidate hazardous scene video data.

[0017] Furthermore, the visual language parsing of the candidate hazardous scene video data yields structured scene description text representing the formation process of the hazardous state, the interactive behavior of traffic participants, and the road environment, specifically including: The candidate hazardous scene video data are extracted by video frame extraction or video segment division according to the time sequence of the formation of the hazardous state to obtain the scene time sequence image sequence. The scene time-series image sequence is input into the video visual language parsing model, which includes a video image encoding unit, an object detection unit, a spatiotemporal trajectory tracking unit, a multimodal feature alignment unit, and a text generation unit. The video image encoding unit extracts temporal visual features from the scene temporal image sequence, and the target detection unit identifies traffic participants and road scene objects in the scene temporal image sequence. The traffic participants include main vehicles, non-main vehicles, non-motorized vehicles and pedestrians, and the road scene objects include lanes, road markings, road boundaries, merging areas, intersection areas and traffic control facilities. The spatiotemporal trajectory tracking unit performs cross-frame association of the traffic participants in continuous video frames to obtain the spatiotemporal pose change information and interaction behavior information of each traffic participant during the formation of a dangerous state. The interaction behavior information includes driving direction, lane occupancy status, lane changing behavior, braking behavior, acceleration behavior, steering behavior, and collision contact status. The spatiotemporal pose change information, the interaction behavior information, and the scene semantic information corresponding to the road scene object are semantically aligned through the multimodal feature alignment unit to obtain dynamic physical state features for characterizing the formation process of dangerous states. The text generation unit converts the dynamic physical state features into the structured scene description text, which includes a road scene description, a traffic participant description, an interactive behavior description, a hazardous state formation process description, and a road environment description. The structured scenario description text is input into the hazardous scenario risk responsibility attribution model. The probability distributions of the special labels of each scenario risk attribute target and each hazardous scenario responsibility attribution target special label in the autoregressive output layer of the hazardous scenario risk responsibility attribution model are extracted. The category corresponding to the scenario risk attribute target special label with the highest probability is taken as the scenario risk attribute judgment result. The category corresponding to the hazardous scenario responsibility attribution target special label with the highest probability is taken as the hazardous scenario responsibility attribution result. The probability corresponding to the hazardous scenario responsibility attribution result is taken as the prediction confidence.

[0018] Compared with the prior art, the present invention has the following advantages: (1) Existing high-risk traffic interaction records have inconsistent field sources and description granularities, and some records lack road environment information that can affect liability attribution. This invention obtains road satellite images based on scene location information, supplements the number of lanes, road markings, road intersection patterns, and the distribution of surrounding buildings through visual language parsing, and then organizes them with dangerous scene text records into standardized scene text prompts. This can reduce semantic shifts caused by differences in data sources for the same scene, keep the behavioral facts, road conditions, and labels in the training samples consistent, and improve the adaptability of subsequent models to cross-regional scenes.

[0019] (2) When general-purpose large language models are used for risk responsibility judgment, they tend to generate explanatory texts of varying lengths, and the training gradient may be occupied by a large number of fact restatement labels, making it difficult to stabilize the discrimination boundary of a few responsibility categories. This invention maps the task of scene risk attribute judgment and the task of attributing responsibility for dangerous scenes to target-specific labels respectively. Based on the frozen backbone network, the input embedding layer, output embedding layer and low-rank adaptive parameters are trained, and a local loss mask is used to calculate the fine-tuning loss only for the target-specific labels. This combination allows the model's parameter updates to focus on the differences between risk categories and responsibility categories, rather than learning to repeatedly generate input facts; while reducing the computational overhead of full fine-tuning, it can also compress the output space and reduce the interference caused by category name drift and redundant text on batch screening. For high-risk data with uneven distribution, the clear competitive relationship formed by the target label probabilities can also alleviate the phenomenon of the model continuously biasing towards high-frequency categories.

[0020] (3) Existing hazardous scene generation or trajectory resampling technologies typically rely on collision probability, minimum distance, or collision time as the main basis, which can easily misinterpret unavoidable collisions caused by abnormal behavior of surrounding traffic participants as defects in the master vehicle's algorithm. This invention first uses video visual language parsing to obtain the formation process of hazardous states, the behavior of traffic participants, and the road environment. Then, the hazardous scene risk responsibility attribution model combines causal relationships, road traffic rules, and behavioral rationality to output responsibility attribution results and prediction confidence, and selects scenarios where the master vehicle is primarily responsible. The resulting dataset is no longer just a set of "high-risk" scenarios, but a set of training samples that can point to the boundaries of the master vehicle's perception, prediction, planning, or control decisions. For scenarios dominated by surrounding participants, such as sudden reversal or active collisions, it can prevent them from repeatedly driving the master vehicle to form unnecessary conservative actions during training. For scenarios where the master vehicle fails to yield, has delayed lane change judgment, or has improper braking timing, it can retain its complete interaction process for targeted training. Therefore, without simply pursuing a higher collision rate, it can also improve the ability of training data to reveal defects in the master vehicle's algorithm, while taking into account both safety and normal traffic efficiency after training.

[0021] (4) Candidate hazardous scenarios can come from natural driving data or adversarial scenarios generated under road topology, basic traffic direction and vehicle dynamics constraints. Both types of sources are processed by the same structured description and responsibility attribution process, which can expand the coverage of low-frequency interactions while maintaining the credibility of real-world scenarios.

[0022] (5) This invention associates and stores the screening results with candidate videos, traffic participant trajectories, road environment information, risk attributes, liability categories, and prediction confidence levels, which can directly form the scene data units required for training autonomous driving algorithms. The prediction confidence level can also be used to set automatic entry thresholds or trigger manual review, thereby reducing the possibility of amplifying erroneous labels after low-confidence samples enter the training set. Attached Figure Description

[0023] Figure 1 This is a schematic diagram of the overall process of the method for screening hazardous scenarios where the vehicle is primarily responsible for autonomous driving algorithm training according to the present invention. Figure 2 To verify the scenario risk attributes and attribution confusion matrix of the present invention model and the comparative model under the data; Figure 3 This is a schematic diagram of the scenario risk attributes and attribution confusion matrix of the present invention model and the comparative model under independent test data; Figure 4 A schematic diagram illustrating the relationship between different prediction confidence thresholds and the prediction accuracy of scenario risk attributes and liability attribution in hazardous scenarios; Figure 5Example diagram for screening hazardous scenarios where the primary vehicle is primarily responsible in a highway merging scenario. Detailed Implementation

[0024] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0025] Example 1: like Figure 1 As shown in the illustration, this embodiment further explains a method for screening hazardous scenarios where the primary vehicle is primarily responsible for training autonomous driving algorithms. This method is executed collaboratively by a data processing server, a model fine-tuning server, a video parsing server, and a scene data storage device. These servers can be set up separately or implemented by different functional modules within the same computing device. The primary vehicle is the test vehicle corresponding to the autonomous driving algorithm to be trained; non-primary vehicles are other motor vehicles that interact with the primary vehicle in traffic; and non-motorized vehicles or pedestrians are another type of traffic participant. The data processing server is used to generate vertical-domain fine-tuning samples, the model fine-tuning server is used to obtain a risk responsibility attribution model for hazardous scenarios, the video parsing server is used to process candidate hazardous scenario video data, and the scene data storage device is used to store the screened hazardous scenarios where the primary vehicle is primarily responsible and their associated information.

[0026] In this embodiment, historical high-risk traffic interaction data can come from a natural driving data acquisition system or an autonomous driving simulation platform, and the two can also be used in combination. The natural driving data acquisition system includes an onboard camera, a vehicle state acquisition interface, a positioning device, and a road test data recording device. Its advantage lies in its ability to preserve the interaction distribution formed by the combined effects of real drivers, real road structures, and real traffic flow, making it particularly suitable for providing scenarios that are difficult to enumerate through rules, such as unconventional yielding, sudden cutting in, road occlusion, and multi-party interactions. The autonomous driving simulation platform can repeatedly generate specified interaction types under road topology, traffic rules, and vehicle dynamics constraints. Its advantage lies in its ability to control the initial position, speed, and behavioral parameters of each traffic participant, facilitating the supplementation of long-tail scenarios in real data that are scarce but highly valuable for autonomous driving safety training. When the two are combined, the behavioral distribution of real scenarios can be determined first using natural driving data, and then parameter perturbation scenarios can be generated around this behavioral distribution in the simulation platform, thus balancing data authenticity and coverage.

[0027] Each high-risk traffic interaction record is assigned a globally unique scene number. This globally unique scene number can be formed by combining the data source identifier, collection date, road area identifier, vehicle identifier, and scene sequence number, or it can be generated by a hash function based on these fields. This number is used to associate the main vehicle status information, surrounding traffic participant information, hazardous scene text records, hazardous scene location information, scene risk attribute labels, and hazardous scene responsibility attribution labels within the same scene. The main vehicle status information can at least include the main vehicle's position, speed, longitudinal acceleration, lateral acceleration, heading angle, steering angle, braking status, throttle status, and autonomous driving function status; surrounding traffic participant information can include object category, object number, relative position, relative speed, lane, and direction of movement. The advantage of using globally unique scene numbers for association is that it avoids cross-scene mismatches between training samples, even if text, images, trajectories, and annotations are stored in different data tables.

[0028] Hazardous scenario text records can be generated from at least one of the following: road test records, simulation logs, accident or hazard records, and manual review records. To ensure a definite data entry point for subsequent structured processing, hazardous scenario text records should include at least the scenario occurrence time, scenario location, weather information, vehicle status, behavior of surrounding traffic participants, interaction process of traffic participants, and participation of non-motorized vehicles or pedestrians; where possible, visibility, road surface adhesion status, lighting conditions, traffic flow density, vehicle control mode, and driver takeover status may also be included. For missing fields, a missing identifier is retained in the data table instead of using unfounded speculative values, thereby reducing the risk of incorrectly filling in missing facts as confirmed facts during the text generation stage.

[0029] Scene risk attribute labels are used to represent the level of damage that a dangerous situation may cause or has already caused. This embodiment divides them into <Property Damage Risk>, <Personal Injury Risk>, and <Fatal Risk>. For data from real collision records, labels can be determined based on the actual damage results in the records; for data from simulation platforms, labels can be determined based on the simulation collision results, vehicle speed changes, collision object categories, and the damage level output by the simulation platform. Hazardous scene liability attribution labels are used to characterize the primary responsible party for the formation of the dangerous situation, including <Primary Responsibility of the Main Vehicle>, <Primary Responsibility of the Non-Main Vehicle>, and <Primary Responsibility of the Non-Motorized Vehicle / Pedestrian>. Labels can be provided by labelers with traffic safety analysis experience, combining road traffic rules, the sequential relationship between the actions of the main vehicle and surrounding traffic participants, and the causes of the dangerous situation; when multiple labelers are involved, the final labels can be generated through majority voting or expert review. This labeling method ensures that the model learns not just general scene semantics, but the correspondence between scene risk levels and responsible parties.

[0030] For natural driving data, high-risk scenarios can be initially screened based on vehicle status and target trajectory. One possible approach is to calculate the estimated collision time between the main vehicle and the object in front of it. in, Indicates time The estimated collision time, This indicates the relative distance between the main vehicle and the interacting object along the potential collision direction. Represents relative velocity, when two objects approach each other. , This represents a preset stability constant to avoid a denominator of zero. Any of the following conditions can be used as high-risk triggers: the estimated collision time is less than a preset time threshold, the main vehicle's braking deceleration exceeds a preset braking threshold, the minimum lateral clearance is less than a preset distance threshold, or a collision contact is detected. The estimated collision time is suitable for identifying longitudinal rear-end collision risks, while the minimum lateral clearance is more suitable for identifying merging, lane-changing, and lateral intrusion risks. Using multiple conditions in combination can reduce missed detections caused by relying on a single indicator.

[0031] The location information of a hazardous scene is formed by the time of the scene occurrence, the location of the scene, the route name, and GPS coordinates. For records with short-term GPS coordinate drift, map matching can be performed using adjacent positioning points, road centerlines, and route names to determine the road location corresponding to the hazardous scene. Then, the map application programming interface (API) is called to acquire satellite images of the road within a preset spatial range centered on that road location. In a specific implementation, the preset spatial range can be a rectangular area centered on the GPS coordinates with sides ranging from 100 to 500 meters. The image resolution can be 512×512 pixels, 1024×1024 pixels, or other resolutions supported by the map API. For larger scenes such as highway merging areas and complex intersections, the capture range can be expanded. Using a smaller range preserves lane and lane marking details, while using a larger range is more conducive to identifying road connections and the distribution of surrounding buildings. Therefore, the image range can be adaptively selected according to the route type.

[0032] A visual language parsing model is developed using road satellite imagery. This model comprises an image encoding unit, a multimodal feature alignment unit, and a text generation unit. The image encoding unit can employ a visual Transformer, a Swing Transformer, a convolutional neural network, or a CLIP image encoder. The visual Transformer is suitable for acquiring global road topology, while the Swing Transformer and convolutional neural networks offer good multi-scale representation capabilities for local lane boundaries and road marking textures. The CLIP image encoder facilitates the direct mapping of road visual features to a shared semantic space between the image and text. Alternatively, the local texture features extracted by the convolutional neural network can be concatenated with the global features extracted by the visual Transformer and then input into the multimodal feature alignment unit, thus accommodating both fine-grained road elements and a larger range of road structures.

[0033] The multimodal feature alignment unit can employ cross-attention networks, Q-Former query transformation structures, or embedding space alignment structures formed through image-text comparison learning. Cross-attention networks can select task-relevant image regions based on predefined road environment parsing prompts, making them suitable for targeted extraction of lane numbers and road intersection patterns. Q-Former compresses visual features with a limited number of query vectors, reducing the input length of subsequent text generation units. Image-text embedding space alignment structures have lower computational cost, making them suitable for scenarios requiring batch processing of large numbers of satellite images. The text generation unit can use a Transformer decoder or be implemented by a large language model with text generation capabilities.

[0034] The road visual features extracted by the image coding unit include lane number features, pavement marking features, road intersection morphology features, and building distribution features near the road location corresponding to the hazardous scene. Lane number features can indicate the number of lanes in the direction of travel of the hazardous scene, the number of oncoming lanes, and whether there are ramp lanes, acceleration / deceleration lanes, or emergency lanes. Pavement marking features can indicate solid lines, dashed lines, guide lines, stop lines, pedestrian crossings, lane dividers, and turn arrows. Road intersection morphology features can indicate crossroads, T-junctions, roundabouts, ramp merging points, and diverging or merging zones. Building distribution features can indicate the density of buildings on both sides of the road, the distance between buildings and road boundaries, the presence of continuous building occlusion, and the distribution of buildings at intersection corners. Building distribution features are not used to identify specific buildings, but rather to supplement surrounding environmental information that may affect sight distance, occlusion relationships, and the complexity of the road environment.

[0035] Pre-defined road environment analysis prompts constrain the output content and field order. For example, the model is required to output "number of lanes - road markings - road intersection patterns - distribution of nearby buildings - uncertain information" in sequence. When visual evidence for a certain item is insufficient, the text generation unit outputs "cannot be determined from the current satellite image" without forcibly specifying a category. This approach maintains a fixed structure for road environment description information and reduces the generation of content unrelated to satellite images by the visual language model due to open generation. When multiple satellite images of the same scene at different zoom levels are obtained, local road descriptions and global road connectivity descriptions can be generated separately, and then merged according to a globally unique scene number. This method is particularly suitable for scenarios that require both marking recognition and ramp connection recognition.

[0036] Subsequently, field parsing is performed on the text records of hazardous scenes. Field parsing can employ a pre-defined field dictionary and rule expressions, or it can use a text model trained on information extraction. The rule-based approach provides stable output and is suitable for structured or semi-structured logs with fixed field names; the text model can handle synonyms and ellipsis in natural language descriptions and is suitable for human-generated records from complex sources. The two approaches can be used in combination: first, explicit fields are extracted using rules, and then the text model supplements any unmatched content. Road environment description information is matched with field information extracted from hazardous scene text records using globally unique scene IDs to form a multi-source scene feature set. When there is a conflict between the text record and the satellite image parsing results, both sources and their confidence information can be retained, and the conflict status can be written into standardized scene text prompts, enabling subsequent models to distinguish between the original recorded facts and the image inference results.

[0037] The multi-source scene feature set is divided into system task information, scene basic information, road environment information, traffic participant interaction behavior information, and non-motorized vehicle or pedestrian information. System task information is used to define the model role and output category; scene basic information includes at least time, location, and weather; road environment information includes the number of lanes, road markings, road intersection patterns, and the distribution of surrounding buildings; traffic participant interaction behavior information includes the behavior of the main vehicle, the behavior of surrounding traffic participants, relative motion states, and interaction results; non-motorized vehicle or pedestrian information is written as "none" when the corresponding object does not exist in the scene. The input structure formed by the above division can map scene records of different sources and lengths to the same information slots, avoiding unstable judgments made by the basic large language model based solely on field arrangement or text length.

[0038] In one implementation, information components are directly concatenated using a fixed prompt template. Fixed templates offer advantages such as fast processing speed and controllable input structure. In another implementation, the concatenated field key-value pairs are input to a pre-trained text generation model and semantically coherently processed, transforming them into complete narrative text. This method reduces interference from field abbreviations, symbols, and discrete key-value pairs on the understanding of the underlying large language model. Semantic coherence processing only changes the form of expression and does not add facts not present in the original fields. An exemplary standardized scene text prompt can sequentially record: the model's task requirement as a risk responsibility attribution expert for hazardous scenes; the time, location, and weather of the scene; the number of lanes, road markings, road intersection patterns, and the distribution of surrounding buildings; the initial positions, speeds, lane-changing or braking behaviors, and relative distance changes of the primary and secondary vehicles; the participation of non-motorized vehicles or pedestrians; and finally, the model is required to output the scene risk attributes and the target-specific markers corresponding to the hazardous scene responsibility attribution.

[0039] The basic large language model adopts a Transformer-based autoregressive language model, including an input embedding layer, a multi-layer self-attention network, a feedforward network, and an output embedding layer. In one specific implementation, the Qwen series model with billions of parameters can be used; in another implementation, the LLaMA series or other open-source basic large language models capable of autoregressive label prediction can be used. Models with smaller parameter sizes have the advantages of fast inference speed and low deployment resource requirements, while models with larger parameter sizes have a stronger ability to represent complex multi-agent causal relationships. The model selection is not limited to specific product names, as long as its word segmenter vocabulary and network parameters can be expanded and fine-tuned as in this embodiment.

[0040] The task of attributing responsibility for risks in hazardous scenarios is divided into a task of determining the risk attributes of a scenario and a task of attributing responsibility for hazardous scenarios. A first set of special tags and a second set of special tags are added to the word segmenter's vocabulary. The first set of special tags includes "Property Loss Risk," "Personal Injury Risk," and "Fatal Risk," while the second set includes "Primary Responsibility of the Main Vehicle," "Primary Responsibility of the Non-Main Vehicle," and "Primary Responsibility of the Non-Motorized Vehicle / Pedestrian." The input and output embeddings of the newly added special tags can be initialized using random distribution or by averaging the embeddings of semantically related common word tags. Random initialization is simple to implement and suitable for situations with a large number of training samples; semantic average initialization ensures that the newly added tags are located in embedding regions close to their category meaning in the early stages of fine-tuning, which usually shortens the convergence time.

[0041] Standardized scenario text prompts are concatenated with their corresponding scenario risk attribute target special tags and dangerous scenario responsibility attribution target special tags in a preset order to form an autoregressive fine-tuning sequence. Preferably, the scenario risk attribute target special tags are output first, followed by the dangerous scenario responsibility attribution target special tags, so that responsibility attribution prediction can utilize the generated risk attribute tags as prior context. In another implementation, the order of the two types of tags can be swapped, or a fixed separator tag can be added between the two tags. As long as the same order is used in the training and inference phases, the consistency of output parsing can be maintained. The joint probability of the target output part is expressed as: in, This indicates standardized scenario text prompts. This represents the output tag sequence, which consists of target-specific tags corresponding to the scenario risk attribute determination results and the hazardous scenario responsibility attribution results. Indicates the first in the output tag sequence A special marker for the target. Indicates the number of special markers on the target. This represents the model parameters involved in vertical fine-tuning.

[0042] During vertical fine-tuning, the backbone network parameters of the basic large language model are frozen, and the input embedding layer, output embedding layer, and low-rank adaptive parameters are set to a trainable state. The low-rank adaptive parameters can be set in at least one of the query projection, key projection, value projection, and output projection in the self-attention layer, or further set in the linear mapping layer of the feedforward network. The weight update is expressed as: in, This represents the updated model weights. This represents the frozen pre-trained base weights. Indicates the weight update amount. and This represents a low-rank matrix. If the input dimension of the original weight matrix is... Output dimension is Then it can make , ,in Less than and In one implementation, the low-rank dimension You can choose 8 or 16; choose the smaller value when the training data is small and you need to reduce the risk of overfitting. When there are many scene types and complex interaction relationships, the size can be appropriately increased. Because the backbone network remains frozen, this approach allows for the concentration of fewer trainable parameters for learning the classification boundaries between risk attributes and responsible parties while preserving basic language capabilities.

[0043] To prevent the model from using its primary training capabilities to paraphrase the input text, a local loss mask is applied to the autoregressive fine-tuning sequence. The loss mask value for the input markers corresponding to the standardized scene text prompts is set to 0, while the loss mask values ​​for the special markers corresponding to the scene risk attribute target and the special markers for the dangerous scene responsibility attribution target are set to 1. The fine-tuning loss is expressed as: in, This indicates the fine-tuning loss function. This indicates the number of labels in the autoregressive fine-tuning sequence. Indicates the first One tag, Indicates the first The context tag preceding the tag, The model predicts the first... The probability of each label. Indicates the first The loss mask value corresponds to each label. This loss ensures that the gradient is generated only by the target-specific label, reducing the interference of variations in the length of the input accident fact text on the training gradient, and making the loss scale comparable for samples of different lengths in the target classification part.

[0044] Training data can be divided into training, validation, and test sets based on scene IDs to prevent different video clips from the same original scene from entering different data subsets. To test cross-regional generalization ability, the data can also be divided according to road area, data collection agency, or simulation map, ensuring the test set originates differently from the training set. Uniform class sampling can be used during training to ensure a relatively balanced probability of occurrence for the three risk attributes and three responsible parties in each training batch; compared to directly copying minority class samples, establishing sampling queues by category reduces overfitting caused by repeated samples. The optimizer can be AdamW, and the learning rate of the low-rank adaptive parameter can be set to... to The input embedding layer and the output embedding layer use the same or lower learning rate; the number of training epochs is determined based on the validation set loss, and training is stopped early when the validation set loss no longer decreases for several consecutive evaluation epochs.

[0045] Video data for candidate hazardous scenarios can be extracted from natural driving data or generated through an autonomous driving simulation platform. For natural driving data, preset time windows before and after the triggering time can be extracted according to the aforementioned high-risk triggering conditions, such as extracting video and trajectory data from 5 seconds before to 3 seconds after the triggering time. A longer preceding time window is beneficial for observing the behavioral evolution before the formation of a hazardous state, while a shorter following time window can preserve the conflict resolution or collision result while avoiding the introduction of irrelevant driving processes.

[0046] For simulation platform data, while preserving road topology, basic traffic participant directions, and vehicle dynamics constraints, at least one of the following parameters for surrounding traffic participants—velocity, acceleration, lateral position, or lane-changing trajectories—is resampled. One possible form of trajectory perturbation is: in, Indicates the first surrounding traffic participants at time The original state vector, which may include longitudinal position, lateral position, velocity, and heading angle, This represents the state disturbance quantity obtained from sampling. This represents the resampled state vector. After resampling, it checks whether the vehicle is within a drivable area, whether its speed is non-negative, whether its acceleration and jerk are within the allowable range of the corresponding vehicle model, and whether the trajectory remains temporally continuous. Trajectories that do not meet the constraints are discarded and resampled. By changing the merging timing, deceleration, or lateral intrusion of non-primary vehicles, scenarios with different levels of danger can be generated without altering the road structure. Compared to completely random traffic behavior generation, this method preserves the behavioral basis of the original real trajectory, and the generated scenarios typically have higher interpretability.

[0047] Candidate hazardous scene videos are processed by extracting video frames or dividing them into segments according to the chronological order in which the hazardous state occurs, forming a temporal image sequence of the scene. Frame extraction can be performed at fixed time intervals or using an event-adaptive method. Fixed-interval extraction is simple to implement and facilitates batch processing; the event-adaptive method increases the frame extraction frequency when the relative distance decreases rapidly, the vehicle begins to change lanes, or the braking state changes, thus preserving key actions before and after the hazardous state occurs without significantly increasing the total number of frames. For videos with long durations, they can be divided into multiple video segments according to time windows, and then each segment can be encoded separately.

[0048] A video visual language parsing model is used to input scene temporal image sequences. The video image encoding unit can employ TimeSformer, VideoMAE, ViViT, a 3D convolutional network, or a visual Transformer with a temporal attention layer. TimeSformer and ViViT can explicitly model cross-frame attention, making them suitable for recognizing lane changes and merging processes; VideoMAE obtains strong temporal representation capabilities through video mask pre-training, making it suitable for scenarios with limited labeled videos; the 3D convolutional network has a compact structure, suitable for deployment environments with limited computing power. The object detection unit can employ YOLO, Faster R-CNN, or DETR, and the spatiotemporal trajectory tracking unit can employ Kalman filtering and Hungarian matching, DeepSORT, ByteTrack, or a Transformer-based multi-object tracking network. All different implementations are used to obtain consistent traffic participant IDs across frames, and the invention is not limited to specific network names.

[0049] The target detection unit identifies main vehicles, non-main vehicles, non-motorized vehicles, pedestrians, as well as lanes, road markings, road boundaries, merging areas, intersections, and traffic control facilities. The spatiotemporal trajectory tracking unit performs cross-frame correlation based on target bounding boxes, visual features, and motion prediction results in consecutive frames to obtain the position sequence of each object. Based on the position sequence, video timestamps, and camera calibration parameters, it can determine or estimate the speed changes, acceleration changes, driving direction, lane occupancy status, lane changing behavior, braking behavior, acceleration behavior, steering behavior, and collision contact status of traffic participants. When precise camera calibration parameters are unavailable, relative displacement and scale changes in image coordinates can be used to represent relative motion trends. This method has the advantage of good applicability to ordinary driving recording videos. Using calibration parameters to transform to a road coordinate system can generate more clearly defined spatiotemporal pose information, suitable for simulation data or high-precision vehicle acquisition.

[0050] The multimodal feature alignment unit semantically aligns spatiotemporal pose change information, interaction behavior information, and scene semantic information corresponding to road scene objects. To avoid the video visual language parsing model only describing the appearance of a single frame, the parsing prompts can be explicitly required to output in the order of "before the formation of a dangerous state - the stage of rapid risk increase - the stage of conflict or collision - the stage of outcome," and each traffic participant must maintain the same object number. Based on this, the text generation unit generates structured scene description text, which includes road scene description, traffic participant description, interaction behavior description, description of the dangerous state formation process, and road environment description. The structured scene description text can be in key-value pair format or in natural language format with a fixed paragraph order; the key-value pair format facilitates program verification and field retrieval, while the natural language format is closer to the pre-training corpus of the basic large language model, and therefore usually has advantages in causal semantic understanding.

[0051] A risk responsibility attribution model for hazardous scenarios is developed based on structured scenario description text input. The model outputs scenario risk attribute determination and hazardous scenario responsibility attribution results based on information such as the sequence of actions of the main vehicle and surrounding traffic participants, road priority relationships, whether necessary observation or avoidance was implemented, and whether the hazardous situation was directly caused by the proactive actions of a particular entity. The model does not require generating a complete accident report; instead, it uses specific target markers to form standardized category outputs, thereby reducing the impact of differences in natural language expression on result interpretation.

[0052] For the attribution of responsibility in hazardous scenarios, the prediction confidence can be defined as the maximum prediction probability in the second set of special labels: in, This indicates the prediction confidence level corresponding to the attribution results for responsibility in hazardous scenarios. Represents the second set of special symbols. This represents the structured scenario description text. The scenario risk attribute assessment results and the hazardous scenario responsibility attribution results are respectively taken from the category with the highest predicted probability in the corresponding special label set. The prediction confidence level can be directly used from the model's output probability, or it can be used after temperature calibration using the validation set; direct probability calculation is simple, while temperature calibration can improve the consistency between different confidence level intervals and the actual accuracy.

[0053] From the candidate hazardous scene video data, scenarios with a responsibility attribution result of "primary vehicle primarily responsible" are selected. When actually building the dataset, a pre-set confidence threshold can be further set. Only retain The samples can be set to a reliability threshold of 0.7 to 0.9. A higher threshold improves the reliability of the responsibility labels on automatically entered samples, while a lower threshold expands the candidate coverage. For samples below the threshold but with high risk attributes, they can be added to the manual review queue instead of being discarded. The selection criteria can be expressed as: in, Indicates scene filtering identifier, This is an indicator function that takes a value of 1 if the condition is true and 0 otherwise. Indicates the attribution of responsibility in dangerous scenarios. This indicates a pre-set confidence threshold. If no confidence threshold is set, the selection is based solely on the attribution result of the hazardous scenario as <primary vehicle primarily responsible>, which is still an optional implementation method in this embodiment.

[0054] When adding hazardous scenarios where the primary vehicle is primarily responsible to the database, at least the following should be associated and stored: candidate hazardous scenario video data, primary vehicle trajectory, trajectories of surrounding traffic participants, road environment information, structured scenario description text, scenario risk attribute determination results, hazardous scenario responsibility attribution results, and prediction confidence levels. Data sources, simulation parameters, model versions, and manual review status can also be stored for subsequent traceability. For a large number of similar samples generated from the same original scenario with slight perturbations, deduplication can be performed based on trajectory distance, video feature distance, or standardized scenario text similarity to prevent the training dataset from being overly dominated by a single scenario cluster. Furthermore, stratified sampling can be performed according to road type, interaction type, risk attributes, and primary vehicle error behavior, so that training batches include different primary vehicle responsibility modes such as failure to yield, insufficient following distance, illegal lane changing, cutting in at intersections, and failure to yield to non-motorized vehicles or pedestrians.

[0055] The primary vehicle-responsible hazardous scenario dataset can be used to train autonomous driving perception, prediction, planning, or end-to-end driving algorithms. When training trajectory prediction models, the historical trajectories of surrounding traffic participants in hazardous scenarios can be used as input, with their subsequent trajectories serving as supervision targets, improving the model's ability to predict strong interactions and sudden behaviors. When training planning models, the primary vehicle's original trajectory can be used as a sample of unwanted behaviors, with corrected trajectories that meet safety distance and yield requirements serving as the target trajectory. In reinforcement learning or closed-loop simulation training, primary vehicle-responsible hazardous scenarios can be repeatedly replayed, and safety rewards and rule constraints can be applied to the primary vehicle control strategy. Compared to using all traffic scenarios proportionally for training, focusing on training scenarios where the primary vehicle is primarily responsible allows a limited training budget to cover more situations where the autonomous driving algorithm's own decision-making is flawed.

[0056] As a specific scenario example, the main vehicle accelerates into the merging zone of a highway at approximately 27.9 m / s, while the non-main vehicle continues to travel in its original lane on the main road. The lateral distance between the main vehicle and the non-main vehicle continuously decreases, and the main vehicle reaches approximately -3.0 m / s at the end of the dangerous situation. 2 The video visual language parsing model identifies merging zones, situations where the primary vehicle accelerates to merge, non-prime vehicles maintain their original lanes, and lateral collisions between the two vehicles, generating structured scene description text. The hazardous scene risk responsibility attribution model outputs special markers for scene risk attributes and "prime vehicle primarily responsible"; when the predicted probability of "prime vehicle primarily responsible" is not lower than a preset confidence threshold, the scene is added to the primary vehicle primarily responsible hazardous scene dataset. This scene can be used to train planning algorithms to pre-assess the gap between vehicles on the primary road, reduce merging speed, or delay merging before merging, thus allowing the data filtering results to directly serve targeted improvements to the primary vehicle's decision-making deficiencies.

[0057] It should be noted that the model structure, parameter range, data source, and thresholds described above are all feasible examples. Those skilled in the art can choose one or more implementation methods based on computing resources, road type, and training objectives. The method described in this embodiment can be implemented as long as standardized scene text prompts are constructed based on historical high-risk traffic interaction data, a risk responsibility attribution model for dangerous scenes is obtained through target-specific markers and local loss masks, and a primary vehicle-responsible dangerous scene dangerous scene is selected from candidate dangerous scene video data based on this model to construct an autonomous driving algorithm training dataset.

[0058] Example 2: This embodiment is used to verify the ability of the risk responsibility attribution model for hazardous scenarios to distinguish between scenario risk attributes and responsible parties. Data that has undergone field standardization and manual review is divided into training data, validation data, and independent test data. The training data is used for vertical domain fine-tuning, and the validation set is used to verify performance on the vertical domain fine-tuning dataset. To ensure fair comparison, the comparison models such as TabNet, XGBoost, CatBoost, and AdaBoost use the same structured scenario fact fields and the same label divisions as the model in this invention.

[0059] Figure 2 and Figure 3 In this diagram, N1, N2, and N3 correspond to <Property Damage Risk>, <Personal Injury Risk>, and <Fatal Risk>, respectively; L1, L2, and L3 correspond to <Primary Responsibility of the Main Vehicle>, <Primary Responsibility of the Non-Main Vehicle>, and <Primary Responsibility of the Non-Motorized Vehicle / Pedestrian>, respectively. When using the terms "Nature of Accident" and "Division of Responsibilities" from the experimental records, these are equivalent to "Scenario Risk Attribute" and "Attribution of Responsibility for Dangerous Scenarios" in this application, respectively. The confusion matrix is ​​normalized according to the true categories; therefore, the diagonal values ​​represent the proportion of the corresponding category that is correctly identified.

[0060] Depend on Figure 2 As can be seen from the validation data, the model of this invention achieves a diagonal proportion of 98.8% for the N1 category, 54.0% for the N2 category, and 39.3% for the N3 category; in the responsibility attribution task, the diagonal proportions for the L1, L2, and L3 categories are 89.7%, 89.4%, and 68.5%, respectively. In comparison, TabNet's diagonal proportions for the N2 and N3 categories are 25.4% and 20.2%, respectively, and XGBoost's diagonal proportion for the L2 category is 30.4%. The results show that after vertical fine-tuning using standardized scene text prompts and target-specific markers, the model can not only identify high-frequency categories but also has a more stable ability to distinguish between personnel injuries, fatal risks, and different responsible parties.

[0061] Figure 3 Results on independent test data are presented. The diagonal proportions of the model in this invention are 98.9%, 61.9%, and 42.0% for categories N1, N2, and N3, respectively, and 92.4%, 89.8%, and 53.8% for categories L1, L2, and L3, respectively. CatBoost's diagonal proportions for categories N2 and N3 are 12.1% and 17.7%, respectively, and AdaBoost's diagonal proportion for category L1 is 48.9%. Even when the independent test data differs from the training data in terms of road location, interaction objects, or data collection conditions, the model in this invention still maintains a relatively concentrated diagonal distribution, indicating that road environment completion and unified text structure can reduce the impact of different data sources on category representation.

[0062] The off-diagonal distribution also reveals that traditional models tend to categorize a few risk categories into N1, or exhibit a significant unidirectional bias between responsibility categories. While the model in this invention still suffers from confusion in the N3 and L3 categories, where the sample size is smaller and the facts are more complex, it does not completely collapse into a single high-frequency category. This is related to the fact that during vertical fine-tuning, the loss is calculated only for the target-specific labels, and multiple target labels compete within a unified output space.

[0063] Figure 4 The relationship between the prediction confidence threshold and the actual prediction accuracy is shown. As the threshold increases, the accuracy of scene risk attributes and attribution of responsibility for hazardous scenes generally increases on both validation and independent test data. This result indicates that the probability of a target-specific label can reflect the reliability of the output. In automated data construction, a higher threshold can be selected for direct data entry, or a lower threshold can be used to expand the candidate set and supplement it with manual review. Therefore, the number of scene selections and label reliability can be adjusted according to the training phase without changing the model structure.

[0064] The comparison results in this embodiment are mainly used to illustrate the feasibility of the technical solution and do not require that the metrics for all categories be the same on all datasets. In practical applications, model performance is also affected by label quality, scene distribution, and video clarity, which can be further improved by increasing low-frequency category samples, performing hard example verification, or adjusting the batch sampling ratio.

[0065] Example 3: This embodiment uses Figure 5 The highway merging scenario shown illustrates the selection and training methods for scenarios where the lead vehicle is primarily responsible for a dangerous situation. The scenario number is highway_merge_2_1_198, and the time step dt in the video or simulation playback is 0.10s. At time t shown in the illustration, the lead vehicle's ego is located in the merging area, with a longitudinal position x_ego of 284.882, a lateral position y_ego of -3.611, a velocity v_ego of 27.921 m / s, and a heading angle yaw_ego of 0.098 rad; the lead vehicle's current longitudinal acceleration is -3.0 m / s². 2 The steering change is -0.027. Car5 is located in the main lane that the main vehicle intends to enter, and Car1 is located in the adjacent area behind it.

[0066] The video visual language parsing model first identifies the main vehicle (ego), car5, car1, and lane boundaries, and establishes a consistent association of object numbers across consecutive frames. Based on the relationship between the main vehicle's trajectory and lane lines, the model identifies that the main vehicle is entering the main road from the merging lane; based on the change in the relative position of the main vehicle and car5, it determines that the lateral and longitudinal distances between the two vehicles are continuously decreasing; based on -3.0 m / s 2 The deceleration is used to identify when the main vehicle approaches in a dangerous situation and takes relatively forceful action. The road environment description also indicates that the location is a merging area of ​​a highway, and merging vehicles need to observe the vehicles on the main road and choose a merging time with sufficient clearance.

[0067] The structured scene description text generated accordingly can be expressed as follows: The lead vehicle (ego) enters the main road at a relatively high speed along the merging lane, while car5 remains within the main road lane; after the lead vehicle begins to merge, the relative distance between it and car5 rapidly decreases, and the lead vehicle only applies significant braking when approaching the point of conflict. The dangerous situation arises from the lead vehicle's delayed choice of merging gap and braking timing. This text preserves the facts of the scene, the sequence of actions, and the road conditions, without directly presupposing a liability conclusion during the video analysis stage.

[0068] After the structured scenario description text is input into the hazardous scenario risk responsibility attribution model, the model assigns the corresponding risk category in the special label of the scenario risk attribute target, and assigns "primary vehicle primarily responsible" as the highest probability category in the special label of the responsibility attribution target. The causal basis is that the primary vehicle, as a merging vehicle, failed to maintain a sufficient safe distance before entering the main road, and the late forced movement indicates a lag in risk prediction or planning decisions; car5 was in a normal driving state in its original lane, and its behavior contributed little to the formation of the hazardous state. When the predicted probability of "primary vehicle primarily responsible" reaches a preset confidence threshold, the scenario is included in the primary vehicle primarily responsible hazardous scenario dataset.

[0069] Upon input, the system saves the original playback video, vehicle trajectory at a 0.10s time step, road topology, key state variables, structured scene description text, risk attribute labels, responsibility attribution labels, and prediction confidence scores. During the autonomous driving algorithm training phase, the same road structure and surrounding vehicle trajectories are reconstructed in a simulation platform, and the algorithm to be trained is loaded into the host vehicle's ego. Training objectives may include identifying acceptable merging gaps in advance, adjusting speed before entering conflict zones, and avoiding forced braking of vehicles on the main road.

[0070] For the planning algorithm, this scenario can be used to adjust the merging timing and target speed; for the prediction algorithm, the focus can be on training the judgment of vehicles continuously occupying the target lane on the main road; for the control algorithm, the longitudinal control response can be optimized by utilizing the late braking process of the main vehicle. Because the scenario has been filtered through responsibility attribution, the training feedback can more directly point to the decision-making defects of the main vehicle algorithm, rather than requiring the main vehicle to bear the entire avoidance task for the unpredictable extreme abnormal behavior of surrounding vehicles.

[0071] If the model assigns similar probabilities to "primary vehicle responsibility" and "non-primary vehicle responsibility," or if the acceleration and deceleration behavior of car5 in the video cannot be reliably identified, the scene is marked for review. Reviewers can examine keyframes, trajectory curves, and the structured scene description generated by the model before deciding whether to include it in the database. This process ensures that automatic screening has clear credibility boundaries, preventing ambiguous scenes from directly participating in algorithm training with incorrect labels.

[0072] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0073] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in the present invention, and these modifications or substitutions should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A method for screening hazardous scenarios where the primary vehicle is primarily responsible for the driver's actions during autonomous driving algorithm training, characterized in that, Includes the following steps: Acquire historical high-risk traffic interaction data, which includes text records of dangerous scenarios, location information of dangerous scenarios, scenario risk attribute tags, and dangerous scenario responsibility attribution tags; Based on the location information of the dangerous scene, obtain the road satellite image corresponding to the road location of the dangerous scene, and perform visual language parsing on the road satellite image to obtain road environment description information; The dangerous scenario text records and the road environment description information are structurally integrated to generate standardized scenario text prompts corresponding to the dangerous scenario risk responsibility attribution task; Based on the standardized scenario text prompts, scenario risk attribute labels, and hazardous scenario responsibility attribution labels, vertical fine-tuning samples are constructed, and the basic large language model is fine-tuned using the vertical fine-tuning samples to obtain a hazardous scenario risk responsibility attribution model. Acquire candidate hazardous scene video data and perform visual language parsing on the candidate hazardous scene video data to obtain structured scene description text that represents the formation process of hazardous state, the interactive behavior of traffic participants and road environment; The structured scenario description text is input into the dangerous scenario risk responsibility attribution model, which then performs causal analysis of the formation of dangerous states and analysis of the rationality of road traffic rules and behaviors based on the structured scenario description text, and outputs scenario risk attribute determination results, dangerous scenario responsibility attribution results and corresponding prediction confidence scores. From the candidate hazardous scene video data, candidate hazardous scenes with primary vehicle responsibility are selected based on the hazardous scene responsibility attribution results. A primary vehicle responsibility hazardous scene dataset is constructed and used for training autonomous driving algorithms. The primary vehicle is the test vehicle corresponding to the autonomous driving algorithm to be trained.

2. The method for screening hazardous scenarios where the vehicle is primarily responsible for the accident, as described in claim 1, is characterized in that... The acquisition of historical high-risk traffic interaction data specifically includes: Acquire high-risk traffic interaction records collected by natural driving and / or high-risk traffic interaction records generated by the autonomous driving simulation platform, and extract globally unique scene numbers from the high-risk traffic interaction records; Based on the globally unique scene number, the main vehicle status information, surrounding traffic participants information, dangerous scene text records, scene risk attribute tags and dangerous scene responsibility attribution tags corresponding to the same high-risk traffic interaction scene are associated; Extract the scene occurrence time, scene location, weather information, route name, and GPS coordinate information from the high-risk traffic interaction records to form the location information of the dangerous scene; The host vehicle status information, surrounding traffic participant information, dangerous scene text records, dangerous scene location information, scene risk attribute labels, and dangerous scene responsibility attribution labels are stored according to the globally unique scene number to obtain the historical high-risk traffic interaction data.

3. The method for screening hazardous scenarios where the vehicle is primarily responsible for the accident, as described in claim 1, is characterized in that... The step of obtaining road satellite images corresponding to the dangerous scene locations based on the dangerous scene location information, and performing visual language parsing on the road satellite images to obtain road environment description information, specifically includes: Based on the GPS coordinate information in the dangerous scene location information, the map application interface is called to obtain road satellite images within a preset spatial range with the location corresponding to the GPS coordinate information as the center. The road satellite image is input into a visual language parsing model, which includes an image encoding unit, a multimodal feature alignment unit, and a text generation unit. The image encoding unit extracts road visual features from the road satellite image, including lane number features, road marking features, road intersection morphology features, and building distribution features near the road location corresponding to dangerous scenes. The road visual features are semantically aligned with preset road environment parsing prompts by the multimodal feature alignment unit to obtain road environment semantic features related to the attribution of risk responsibility in dangerous scenarios; The text generation unit transforms the semantic features of the road environment into road environment description information, which is used to supplement the missing or incomplete road infrastructure information and road surrounding environment information in the historical high-risk traffic interaction data.

4. The method for screening hazardous scenarios where the vehicle is primarily responsible for the accident, as described in claim 1, is characterized in that... The step of structurally integrating the hazardous scenario text records and the road environment description information to generate standardized scenario text prompts corresponding to the hazardous scenario risk responsibility attribution task specifically includes: The text records of the dangerous scenarios are parsed to extract the time of occurrence of the scenario, the location of the scenario, weather information, the status information of the main vehicle, the behavior information of surrounding traffic participants, the interaction process information of traffic participants, and the participation information of non-motorized vehicles or pedestrians. The road environment description information is matched with the field information extracted from the dangerous scene text record to obtain a multi-source scene feature set; According to the input requirements of the hazardous scenario risk responsibility attribution task, the multi-source scenario feature set is divided into system task information, scenario basic information, road environment information, traffic participant interaction behavior information, and non-motorized vehicle or pedestrian information. Based on the system task information, system prompts are generated to define the attribution roles and task objectives for risk responsibility in dangerous scenarios. Based on the scenario basic information, basic factual text containing scenario occurrence time, scenario location, and weather information is generated. Based on the road environment information, road environment text containing the number of lanes, road markings, road intersection patterns, and the distribution of surrounding buildings is generated. Based on the traffic participant interaction behavior information, interactive factual text containing the behavior of the main vehicle, the behavior of surrounding traffic participants, and their interaction status is generated. Based on the non-motorized vehicle or pedestrian information, supplementary text for the participating objects is generated. The system prompt words, the basic factual text, the road environment text, the interactive factual text, and the supplementary text of the participating object are concatenated in a preset order, and the concatenated text is semantically coherent to obtain the standardized scene text prompt words.

5. The method for screening hazardous scenarios where the vehicle is primarily responsible for the accident, as described in claim 1, is characterized in that... The process of constructing vertically tuned samples based on the standardized scenario text prompts, scenario risk attribute labels, and hazardous scenario responsibility attribution labels, and then using these samples to fine-tune the basic large language model to obtain a hazardous scenario risk responsibility attribution model, specifically includes: The task of attributing risk responsibility in dangerous scenarios is divided into a task of determining the risk attributes of scenarios and a task of attributing responsibility in dangerous scenarios. Target special tags are added to the word segmenter vocabulary corresponding to the basic large language model. The target special tags include a first set of special tags used to characterize the result of determining the risk attributes of scenarios and a second set of special tags used to characterize the result of attributing responsibility in dangerous scenarios. The standardized scene text prompts are concatenated with the corresponding scene risk attribute target special markers and dangerous scene responsibility attribution target special markers to obtain an autoregressive fine-tuning sequence, which enables the basic large language model to predict the corresponding scene risk attribute judgment results and dangerous scene responsibility attribution results based on the standardized scene text prompts. When fine-tuning the basic large language model in the vertical domain, the backbone network parameters of the basic large language model are frozen, and the input embedding layer, output embedding layer, and low-rank adaptive parameters are set to a trainable state. The parameters of the basic large language model are updated through low-rank adaptation, wherein the weight update satisfies: in, This represents the updated model weights. This represents the frozen pre-trained base weights. Indicates the weight update amount. and This represents the low-rank matrix in low-rank adaptive fine-tuning. During the vertical fine-tuning process, a local loss mask is applied to the autoregressive fine-tuning sequence so that the input text part corresponding to the standardized scene text prompt words does not participate in the loss calculation, and the fine-tuning loss is calculated only for the output part corresponding to the special marker of the scene risk attribute target and the special marker of the dangerous scene responsibility attribution target. Based on the fine-tuning loss, the input embedding layer, the output embedding layer, and the low-rank adaptive parameters are iteratively updated, and the risk responsibility attribution model for dangerous scenarios is obtained after the vertical fine-tuning is completed.

6. The method for screening hazardous scenarios where the vehicle is primarily responsible for autonomous driving algorithm training according to claim 5, characterized in that, The first set of special markers includes <Property Damage Risk>, <Personal Injury Risk>, and <Fatal Risk>, while the second set of special markers includes <Primary Liability of the Main Vehicle>, <Primary Liability of the Non-Main Vehicle>, and <Primary Liability of the Non-Motorized Vehicle / Pedestrian>.

7. The method for screening hazardous scenarios where the vehicle is primarily responsible for autonomous driving algorithm training according to claim 5, characterized in that, The joint probability of the target output portion in the autoregressive fine-tuning sequence is expressed as: in, This refers to the standardized scenario text prompts. This represents the output tag sequence, which consists of target-specific tags corresponding to the scenario risk attribute determination results and the hazardous scenario responsibility attribution results. Indicates the first in the output tag sequence A special marker for the target. This indicates the number of target-specific tags in the output tag sequence. This represents the model parameters that participate in vertical domain fine-tuning. This indicates the prediction of the first target given the standardized scene text prompts and the preceding target-specific markers. The probability of a target being specially marked.

8. The method for screening hazardous scenarios where the vehicle is primarily responsible for autonomous driving algorithm training according to claim 5, characterized in that, The fine-tuning loss is formulated as follows: in, This represents the fine-tuning loss function; This indicates the number of labels in the autoregressive fine-tuning sequence; This represents the first step in the autoregressive fine-tuning sequence. One tag; Indicates the first The context tag preceding the tag; The representation model predicts the first based on context labels. The probability of each label; Indicates the first The loss mask value corresponding to the i-th label, when the i-th label... When a marker is a special marker for a scenario risk attribute target or a special marker for a hazardous scenario responsibility attribution target... When the first When a marker belongs to the input text portion corresponding to the standardized scene text prompt word .

9. The method for screening hazardous scenarios where the vehicle is primarily responsible for autonomous driving algorithm training according to claim 1, characterized in that, The candidate dangerous scenario video data includes high-risk traffic interaction scenario videos extracted from natural driving data, and / or adversarial scenario videos obtained by resampling the movement trajectories of surrounding traffic participants through an autonomous driving simulation platform. The motion trajectory resampling, while maintaining the road topology, the basic travel direction of traffic participants, and vehicle dynamics constraints, alters at least one of the speed, acceleration, lateral position, or lane-changing trajectory of surrounding traffic participants to generate the candidate hazardous scene video data.

10. The method for screening hazardous scenarios where the vehicle is primarily responsible for autonomous driving algorithm training according to claim 1, characterized in that, The visual language parsing of the candidate hazardous scene video data yields structured scene description text representing the hazardous state formation process, traffic participant interaction behavior, and road environment, specifically including: The candidate hazardous scene video data are extracted by video frame extraction or video segment division according to the time sequence of the formation of the hazardous state to obtain the scene time sequence image sequence. The scene time-series image sequence is input into the video visual language parsing model, which includes a video image encoding unit, an object detection unit, a spatiotemporal trajectory tracking unit, a multimodal feature alignment unit, and a text generation unit. The video image encoding unit extracts temporal visual features from the scene temporal image sequence, and the target detection unit identifies traffic participants and road scene objects in the scene temporal image sequence. The traffic participants include main vehicles, non-main vehicles, non-motorized vehicles and pedestrians, and the road scene objects include lanes, road markings, road boundaries, merging areas, intersection areas and traffic control facilities. The spatiotemporal trajectory tracking unit performs cross-frame association of the traffic participants in continuous video frames to obtain the spatiotemporal pose change information and interaction behavior information of each traffic participant during the formation of a dangerous state. The interaction behavior information includes driving direction, lane occupancy status, lane changing behavior, braking behavior, acceleration behavior, steering behavior, and collision contact status. The spatiotemporal pose change information, the interaction behavior information, and the scene semantic information corresponding to the road scene object are semantically aligned through the multimodal feature alignment unit to obtain dynamic physical state features for characterizing the formation process of dangerous states. The text generation unit converts the dynamic physical state features into the structured scene description text, which includes a road scene description, a traffic participant description, an interactive behavior description, a hazardous state formation process description, and a road environment description. The structured scenario description text is input into the hazardous scenario risk responsibility attribution model. The probability distributions of the special labels of each scenario risk attribute target and each hazardous scenario responsibility attribution target special label in the autoregressive output layer of the hazardous scenario risk responsibility attribution model are extracted. The category corresponding to the scenario risk attribute target special label with the highest probability is taken as the scenario risk attribute judgment result. The category corresponding to the hazardous scenario responsibility attribution target special label with the highest probability is taken as the hazardous scenario responsibility attribution result. The probability corresponding to the hazardous scenario responsibility attribution result is taken as the prediction confidence.

Images