Data return method and system for extreme driving scene, and medium

Abstract

The invention relates to the technical field of driving data sensing processing, in particular to a data returning method and system for an extreme driving scene and a medium, and the data processing method for the extreme driving scene comprises the steps: obtaining and inputting an image frame; screening the input image frames, and only starting a scoring process for the image frames meeting a screening condition; in the scoring process, the image frames are scored in an inter-frame hop sampling or event triggering mode, and a multi-dimensional scoring result is output in real time; when the multi-dimensional scoring result exceeds a preset scoring threshold value, executing rule labeling on the image frame and the scene segment corresponding to the image frame, and generating a labeling file; and storing the annotation file and the corresponding vehicle end data packet according to a priority sequence, wherein the vehicle end data packet at least comprises image data, radar point cloud data and CAN data.

Description

A data backhaul method, system, and medium for extreme driving scenarios Technical Field

[0001] This invention relates to the field of driving data perception and processing technology, specifically to a data backhaul method, system, and medium for extreme driving scenarios. Background Technology

[0002] With the development of advanced autonomous driving technology, the vehicle's ability to perceive, understand, and respond to data in real time has become crucial to ensuring driving safety.

[0003] Taking intelligent driving trajectory planning as an example, it relies heavily on the perception of the external environment. When working, it collects driving environment information through on-board sensors and combines it with its own driving task. It uses perception recognition algorithms to process on-board sensor signals. After receiving the perception output, the decision planning algorithm performs information fusion, and then makes decisions and plans the corresponding trajectory. The motion control algorithm controls the vehicle platform to follow the planned trajectory. After receiving the motion control signal, the vehicle platform drives the vehicle to move.

[0004] In actual road operation, vehicles face complex and ever-changing traffic environments, including dynamic participants such as pedestrians, non-motorized vehicles, and sudden obstacles; complex road structures such as construction zones and laneless road sections; and changing weather and lighting conditions such as nighttime, strong sunlight, backlighting, rain, fog, and snow. To improve the system's adaptability to these complex situations, it is urgent to collect more representative, comprehensive, and long-tailed extreme driving scenario (corner case) data.

[0005] However, since current intelligent driving systems rely on a large amount of real driving data for training, extreme driving scenario data is often scarce and difficult to collect actively. Labeling is costly and time-consuming, and some extreme driving scenarios are difficult to be identified by humans in a timely manner. Summary of the Invention

[0006] To address the aforementioned technical problems, this invention provides a data backhaul method, system, and medium for extreme driving scenarios, which can solve at least one of the technical problems mentioned in the background section.

[0007] To achieve the above objectives, the present invention adopts the following technical solution.

[0008] The first aspect of this invention provides a data backhaul method for extreme driving scenarios, comprising: acquiring and inputting image frames; filtering the input image frames and initiating a scoring process only for image frames that meet the filtering conditions; in the scoring process, scoring the image frames by means of inter-frame skip sampling and / or event triggering, and outputting multi-dimensional scoring results in real time; when the multi-dimensional scoring results exceed a preset scoring threshold, performing rule annotation on the image frames and their corresponding scene segments and generating an annotation file; storing the annotation file and the corresponding vehicle-side data packets in priority order, wherein the vehicle-side data packets include at least image data, radar point cloud data, and CAN data.

[0009] As an optional implementation, before filtering the input image frames, the following steps are taken: setting filtering conditions, including setting confidence fluctuation conditions and vehicle status abnormality prompt conditions; when the input image frame meets the confidence fluctuation conditions and / or vehicle status abnormality prompt conditions, it indicates that the image frame meets the filtering conditions.

[0010] As an optional implementation, in the scoring process, the image frames are scored by inter-frame skip sampling and / or event triggering, and multi-dimensional scoring results are output in real time, including: obtaining a perceptual uncertainty score; obtaining a behavior trigger score; obtaining an environmental complexity score; obtaining a model consistency score; normalizing the perceptual uncertainty score, the behavior trigger score, the environmental complexity score, and the model consistency score respectively, and obtaining the multi-dimensional scoring results by weighted fusion.

[0011] As an optional implementation, obtaining the perceptual uncertainty score includes: obtaining the detection confidence standard deviation and confidence fluctuation threshold of the image frame; and generating a perceptual uncertainty score based on the ratio of the detection confidence standard deviation and the confidence fluctuation threshold.

[0012] As an optional implementation, obtaining the behavior trigger score includes: obtaining an adjustment coefficient and the number of triggers per unit time; and generating a behavior trigger score based on the adjustment coefficient and the number of triggers per unit time.

[0013] As an optional implementation, obtaining the environmental complexity score includes: obtaining scores for each scenario factor; weighting and averaging the scores for each scenario factor to generate an environmental complexity score.

[0014] As an optional implementation, obtaining the model consistency score includes: obtaining the maximum deviation distance and the divergence tolerance threshold; and generating a model consistency score based on the maximum deviation distance and the divergence tolerance threshold.

[0015] As an optional implementation, when the multidimensional scoring result exceeds a preset scoring threshold, performing rule annotation on the image frame and its corresponding scene fragment and generating an annotation file includes: taking the image frame that triggered the scoring as the center, obtaining image frames within a preset time window before and after it to form the scene fragment corresponding to the image frame; when the multidimensional scoring result exceeds the preset scoring threshold, performing rule annotation on the image frame and its corresponding scene fragment in the form of structured tags, wherein the content of the rule annotation includes at least static background information rule annotation, dynamic participant information rule annotation, and system status rule annotation.

[0016] As an optional implementation, when the multidimensional scoring result exceeds a preset scoring threshold, after performing rule annotation on the image frame and its corresponding scene fragment and generating an annotation file, the method further includes: performing context completion and confidence recording on the annotation file to obtain an updated annotation file.

[0017] As an optional implementation, storing the annotation files and corresponding sensing data in priority order includes: dynamically queuing the stored data according to priority and feeding it back to the cloud.

[0018] A second aspect of the present invention provides a data backhaul system for extreme driving scenarios, comprising: an input module, at least for acquiring and inputting image frames; a lightweight strategy module, at least for filtering the input image frames and initiating a scoring process only for image frames that meet the filtering conditions; a scoring module, at least for scoring the image frames in the scoring process by means of inter-frame skip sampling and / or event triggering, and outputting multi-dimensional scoring results in real time; an annotation module, at least for performing rule annotation on the image frames and their corresponding scene segments and generating an annotation file when the multi-dimensional scoring results exceed a preset scoring threshold; and a storage module, at least for storing the annotation file and the corresponding vehicle-side data packets in priority order, wherein the vehicle-side data packets include at least image data, radar point cloud data, and CAN data.

[0019] As an optional implementation, the lightweight strategy module further includes a filtering and judgment submodule, which is used to set filtering conditions, including setting confidence fluctuation conditions and vehicle status abnormality prompt conditions; when the input image frame meets the confidence fluctuation conditions and / or vehicle status abnormality prompt conditions, it indicates that the image frame meets the filtering conditions.

[0020] A third aspect of the present invention provides an electronic device, comprising: at least one processor; and at least one memory communicatively connected to the processor, wherein: the memory stores program instructions executable by the processor, and the processor, by invoking the program instructions, can perform the steps of the method described in the first aspect of the present invention.

[0021] A fourth aspect of the present invention provides a readable storage medium storing a computer program that is executed by a processor as described in the first aspect of the present invention. Attached Figure Description

[0022] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0023] Figure 1 is a flowchart illustrating a data backhaul method for extreme driving scenarios according to a specific embodiment of the present invention.

[0024] Figure 2 is a block diagram of a data backhaul system for extreme driving scenarios according to a specific embodiment of the present invention.

[0025] Figure 3 is a schematic diagram of the structure of an electronic device according to an embodiment of the present invention. Detailed Implementation

[0026] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application. Furthermore, it should be understood that the specific embodiments described herein are only for illustration and explanation of this application and are not intended to limit this application.

[0027] It should be noted that the order of description of the following embodiments is not intended to limit the preferred order of the embodiments of this application. Furthermore, the descriptions of each embodiment in the following embodiments have their own emphasis; for parts not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0028] As shown in Figure 1, the first aspect of the present invention provides a data backhaul method for extreme driving scenarios, including the following steps.

[0029] Step S100: Acquire and input image frames.

[0030] Specifically, the image frames mainly come from forward-looking, surround-view, or side-view cameras (usually RGB or IR images).

[0031] In this invention, image frames are the key source of visual signals, and are synchronously acquired and labeled with the following signals to form multimodal data: ① point cloud frames (from LiDAR or millimeter-wave radar, used for depth and obstacle detection); ② CAN signal frames (containing dynamic states such as vehicle speed, acceleration, steering angle, and braking signals); ③ GPS or IMU data (providing vehicle pose information). These data are synchronized on the timeline using timestamp alignment to form a complete set of multimodal perception frames for subsequent scene scoring and labeling.

[0032] An image frame specifically refers to a single image or video frame captured by an onboard camera at a fixed sampling period (e.g., 10Hz, 30Hz, etc.). Each image frame represents the vehicle's visual perception result at that point in time and is the basic input unit for onboard vision algorithms (such as object detection, semantic segmentation, optical flow analysis, etc.). It has temporal continuity (frame sequence) and spatial consistency (image resolution, field of view, etc.).

[0033] Visual perception is the primary source of corner cases. Corner cases often manifest first in the visual modality (e.g., backlighting, rain and fog obstruction, atypical obstacle shapes, abnormal traffic signs, etc.). Using image frames as the basic unit allows for the earliest capture of abnormal features. Furthermore, because image frames are highly synchronized with other modalities, their timestamps are usually the primary reference (master clock source) for system synchronization. Therefore, it is convenient to perform multimodal fusion analysis with other multimodal data.

[0034] For example: one image frame + point cloud within the corresponding time window + CAN status = one complete perception frame.

[0035] Furthermore, scoring on a frame-by-frame basis allows for a reasonable design of computation and storage granularity, and enables flexible adjustment of the sampling frequency, such as calculating once every 3 frames or every 1 second.

[0036] In this way, while maintaining real-time performance, the computational and bandwidth pressure can be significantly reduced.

[0037] Step S200: Filter the input image frames and start the scoring process only for image frames that meet the filtering conditions.

[0038] Specifically, this invention uses image frames as the basic unit for filtering data, but it is not limited to single static images. Instead, it can include multiple frame segments associated with their temporal neighborhood. This is because while a single frame can detect problems, training or playback analysis often requires temporal context (e.g., the appearance, occlusion, and disappearance of a target). Therefore, this invention automatically packages several frames before and after the trigger frame as a segment for uploading after detection. That is, although the filtering action is triggered by an image frame (e.g., detecting a frame with abnormal confidence fluctuations), the actual content saved and uploaded may be several frames before and after that frame (e.g., a time window), forming a short time segment.

[0039] This approach preserves complete context information, facilitating backend scenario reconstruction; saves data only during the triggering period, thereby reducing invalid data uploads; and achieves structural compatibility with training data, such as dynamic scene samples of 3 seconds and 5 seconds.

[0040] Step S300: In the scoring process, the image frame is scored by inter-frame skip sampling or event triggering, and multi-dimensional scoring results are output in real time.

[0041] Among them, for the two triggering methods mentioned above, periodic triggering is implemented through inter-frame skip sampling, which is used to score image frames at preset intervals or frame intervals during normal driving; event triggering is used to immediately score the corresponding image frame or segment when events such as fluctuations in perception confidence, abnormal vehicle status, or model prediction discrepancies are detected, thereby achieving rapid response to sudden extreme scenarios.

[0042] Specifically, scoring is performed at preset intervals of frames by skipping between frames, or scoring is performed at fixed time windows, so as to periodically check the scene status in a lightweight manner.

[0043] Specifically, the inter-frame skip sampling method is supplemented by event triggering. When a certain type of abnormal signal is detected, scoring can be started without waiting for the sampling period.

[0044] Specifically, inter-frame skip sampling is a periodically triggered method; event triggering is a method that is triggered immediately when an anomaly occurs. These are two different triggering mechanisms, and the scoring process can be started and scored using both methods.

[0045] Specifically, inter-frame skip sampling refers to a method that, in order to reduce computational power and latency, does not perform scoring and inference for every single image frame. Instead, it performs scoring once every N frames (e.g., every 3, 5, or 10 frames) or once within a fixed time window (e.g., every 1 second). In other words, inter-frame skip sampling uses a lightweight approach to periodically check the scene state, rather than performing a full computation. By using inter-frame skip sampling, computational power consumption can be reduced while still maintaining real-time performance at the second level.

[0046] Specifically, event-triggered mechanisms refer to immediately activating the scoring module upon detecting a certain type of abnormal signal, without waiting for the next sampling cycle. Typical events include a sudden drop in perception confidence; CAN signal displaying AEB, FCW, or manual intervention; the scene classification model outputting abnormal scene labels; sharp changes in vehicle trajectory; and divergence in multi-model prediction results. Event triggering complements the skip sampling mechanism, ensuring the system responds immediately at critical moments.

[0047] Preferably, the periodic scoring specifically involves performing a scoring check on the scene at fixed intervals or a fixed number of frames, rather than calculating every single frame. For autonomous driving scene scoring, the camera operates at 30 frames per second. However, the scoring system can process only 1-3 frames per second. This approach reduces computational power consumption while maintaining sufficient real-time performance without losing important scenes.

[0048] Step S400: When the multidimensional scoring result exceeds the preset scoring threshold, rule annotation is performed on the image frame and its corresponding scene segment and an annotation file is generated.

[0049] Specifically, when the multidimensional scoring result exceeds the preset scoring threshold, the image frames within the preset time window before and after the image frame that triggered the scoring are taken as the center to form a scene segment, and rule annotation is performed on the image frame and its corresponding scene segment to generate an annotation file.

[0050] Specifically, a scene segment, or simply segment, refers to a continuous image sequence centered on a trigger frame, containing several time-series frames (e.g., ±10 frames) before and after it. A segment not only includes multiple frames of images but can also synchronously include radar point cloud data, CAN bus signals, and GPS / IMU pose data within the same time window. Therefore, a segment is essentially a spatiotemporally continuous multimodal sensing window.

[0051] Specifically, the annotation file is a record file generated after rule-based annotation, used to store all the label content corresponding to the image frame. This file usually exists in the form of JSON, XML, TXT, binary PB (protobuf), etc., and contains the results of manual rule annotation for the image frame or segment.

[0052] The annotation file should include at least: a unique ID of the image frame (frame_id); a timestamp; image source information (camera_id, sensor_id); annotated objects (e.g., vehicles, pedestrians, obstacles, road elements); the location and shape of the annotated objects (e.g., bounding boxes, polygons); label categories (e.g., car / pedestrian / lane); attribute information (e.g., speed, occlusion, key event type); and the quality score and its source.

[0053] Step S500: Store the annotation file and the corresponding vehicle-side data packet in priority order. The vehicle-side data packet includes at least image data, radar point cloud data and CAN data.

[0054] Specifically, the annotation file corresponding to the image frame and the vehicle-side raw data collected synchronously with the image frame constitute a high-value data packet. The annotation file and vehicle-side data packet are prioritized according to factors such as scoring results, event type, scene rarity, and system task requirements, and are dynamically queued according to priority.

[0055] When the cache pool capacity is limited, high-priority data packets are stored or uploaded first, while low-priority data packets can be processed later or discarded, thus ensuring that scarce scenarios and critical events are saved first.

[0056] Here, the present invention, through the input of image frames, can continuously and automatically construct a high-value extreme driving sample set without relying on manual screening and intervention. This improves the problem that the data of extreme driving scenarios is always under-collected due to the existing collection mechanism, enhances the stability and generalization ability of the intelligent driving system under long-tail distribution, meets the needs of large-scale data closed-loop training, and improves the problem of the huge bandwidth and storage resources occupied by directly transmitting all the original perception data. Especially when the network conditions of the vehicle-end device are limited, the present invention can achieve high-frequency and controllable data backflow.

[0057] In one embodiment of the present invention, before filtering the input image frame, the following steps are taken: setting filtering conditions, including setting confidence fluctuation conditions and vehicle status abnormality prompt conditions; when the input image frame meets the confidence fluctuation conditions and / or vehicle status abnormality prompt conditions, it indicates that the image frame meets the filtering conditions.

[0058] The confidence fluctuation condition specifically refers to the situation where, in the target detection results of several consecutive frames, the detection confidence of the main target or key category target fluctuates significantly, and the fluctuation exceeds a preset threshold.

[0059] Significant fluctuations include, but are not limited to: the standard deviation of the target confidence score exceeding a threshold, the difference between adjacent frames of the target confidence score exceeding a threshold, or rare large jitter in the detection results.

[0060] When any of the above situations occur, the image frame is considered to meet the confidence fluctuation condition.

[0061] The conditions for abnormal vehicle status warnings specifically refer to control behaviors or system warning signals that are inconsistent with the normal autonomous driving state during vehicle operation, including but not limited to: AEB (Automatic Emergency Braking) triggering, FCW (Forward Collision Warning) triggering, LKA / ELK (Lane Keeping Assist / Lane Departure Correction) strong intervention, manual takeover events, system malfunction indicator lamp warnings, or abnormal fluctuations in vehicle control commands. When any of the above abnormal vehicle status warnings is detected, the corresponding image frame is considered to meet the conditions for abnormal vehicle status warnings.

[0062] In the initial screening stage, before the input image frames are screened, this invention uses confidence fluctuation conditions as trigger signals to quickly determine whether the image frames have potential abnormal features. After entering the scoring process, frequency division is performed based on a complete multi-dimensional scoring logic to achieve a refined evaluation of the scene value.

[0063] In one embodiment of the present invention, the scoring process, which scores the image frames by means of inter-frame skip sampling or event triggering, and outputs multi-dimensional scoring results in real time, includes: obtaining a perceptual uncertainty score S. uncertainty ; Obtain behavior trigger score S intervention Obtain the environmental complexity score S. complexity Obtain the model consistency score S divergence The perceived uncertainty score, the behavior trigger score, the environmental complexity score, and the model consistency score are normalized respectively, and multidimensional scoring results are obtained by weighted fusion.

[0064] Specifically, this invention integrates four dimensions of features—perception confidence fluctuations, behavioral trigger signals, environmental complexity factors, and multi-model consistency—in real time to quantify and score the current driving scenario.

[0065] The reason why the scoring process in this invention uses four dimensions of features—perceived confidence fluctuation, behavioral trigger signal, environmental complexity factor, and multi-model consistency—for quantitative scoring is that the above four features represent the current vehicle scenario from four directions: perceived reliability, abnormal system behavior, external environmental complexity, and model prediction consistency. They can comprehensively cover the main root causes of extreme scenarios and have the characteristics of real-time detection, quantifiable calculation, and low-cost acquisition by vehicle-side modules.

[0066] Specifically, the implementation of this invention is as follows: the method of obtaining multidimensional scoring results through weighted fusion includes setting weights w1, w2, w3, and w4 for each score, so that the multidimensional scoring result S total =w1×S uncertainty +w2×S intervention +w3×S complexity +w4×S divergence .

[0067] The perception system is the primary module of autonomous driving, and the stability of its output directly determines whether the system can recognize road targets. In real-world driving, when encountering backlighting, rain or fog, partial occlusion, changes in lighting, or unseen objects, the target detection confidence level typically fluctuates drastically. Confidence fluctuation is one of the most direct and sensitive indicators of challenging perception scenarios. Confidence fluctuation is the most common direct symptom of corner cases; it can be obtained in real-time from the detection network output without increasing computational load. It can also sensitively capture situations that cause model instability, such as changes in lighting or partial occlusion, and is independent of multimodal data (point clouds, etc.), exhibiting strong adaptability, as verified in extensive closed-loop training on real vehicles.

[0068] In one embodiment of the present invention, obtaining the perceptual uncertainty score includes: obtaining the detection confidence standard deviation σ of the image frame. p and confidence fluctuation threshold θ1; based on the detection confidence standard deviation σ p The ratio of the confidence fluctuation threshold θ1 to the perceived uncertainty score is used to generate a perceived uncertainty score.

[0069] Among them, the standard deviation of detection confidence represents the degree of dispersion of the confidence of target detection results in the same image frame.

[0070] The confidence fluctuation threshold is a threshold used to determine whether there is abnormal confidence fluctuation in the current image. It is a fixed or adaptive reference value used to compare with the detection confidence standard deviation.

[0071] Specifically, the generation of perceived uncertainty scores is achieved through S... uncertainty =min(1,σ) p / θ1) is implemented.

[0072] Generally speaking, when a vehicle exhibits behaviors such as AEB triggering, manual takeover, system degradation, or significant lateral control deviation, it indicates that the vehicle can no longer maintain a stable autonomous driving state in the current scenario. This represents the most typical and high-value extreme scenario type. These behavioral triggering events originate from the vehicle's control system, and the trigger signals have a strong causal relationship. They naturally possess timestamps and are recordable, do not rely on complex calculations, and are considered "high-risk events" with an extremely low probability of occurrence and extremely high value.

[0073] In one embodiment of the present invention, obtaining the behavior trigger score includes: obtaining an adjustment coefficient k1 and the number of triggers n per unit time; and generating a behavior trigger score based on the adjustment coefficient k1 and the number of triggers n per unit time.

[0074] The trigger count per unit time is used to characterize the number of control anomalies or behavioral trigger events generated within a fixed or sliding time window. To obtain the trigger count per unit time: a preset time window is set (e.g., 1 second, 3 seconds, 10 seconds, or other configurable duration). Within this time window, the number of behavioral trigger events generated by the vehicle control system is counted, including but not limited to manual takeover signals, automatic emergency braking triggers, degradation warnings, sudden deceleration events, and route replanning anomalies. When the time window ends or the sliding window is updated, the cumulative number of events is taken as the trigger count per unit time.

[0075] The adjustment coefficient is used to convert the number of triggers per unit time into a uniform scale of behavior trigger score. The adjustment coefficient is a pre-set weighting factor of the system, and its value is configured offline based on model training results, scene risk level, score dimensions, or the distribution of actual labeled data to ensure that the numerical range of the behavior trigger score is consistent with other dimensions of the score. The adjustment coefficient can be obtained through correlation analysis between historical trigger events and high-value data, empirical calibration, or model training, and is stored in the system configuration file for use in the real-time scoring process.

[0076] Among them, the behavior trigger score represents the normalized number of system control anomalies accumulated over a period of time. The behavior trigger score is proportional to the number of anomaly triggers per unit time. The more triggers, the higher the risk and the higher the behavior trigger score. The fewer triggers, the lower the risk and the lower the behavior trigger score.

[0077] Specifically, the generation of behavior trigger scores is achieved through S intervention =1-exp(-k1×n) is implemented.

[0078] Many long-tail problems arise from complex environments, such as nighttime, rain and fog, backlighting, tunnel entrances and exits, missing lane markings, and road surface glare. These factors can reduce perception performance and lead to planning deviations. Changes in the external environment are the most critical exogenous variables affecting the performance of autonomous driving; environmental labels (such as weather and illumination) can be obtained in real time by lightweight models or rules; environmental complexity factors are highly correlated with accident risk; and they can be used to supplement the "slow-changing scenario difficulty" problem that cannot be identified in confidence fluctuations.

[0079] In one embodiment of the present invention, obtaining the environmental complexity score includes: obtaining the scores s of each scene factor. iThe scores of each scenario factor are weighted and averaged to generate an environment complexity score.

[0080] The process involves scene recognition of the input image frames and related sensor data to obtain various environmental factors such as weather, lighting, road type, and occlusion level. For each environmental factor, a score is obtained based on a pre-set scoring table or scene classification model.

[0081] The score for each scenario factor is a normalized environmental complexity score, used to represent the degree of influence of that factor on the complexity of the driving scenario. The score for each scenario factor is obtained through image classification, semantic segmentation, multi-sensor fusion, or rule detection methods, and determined by looking up a table based on its risk level.

[0082] Specifically, the environmental complexity score represents the labels output by the scene classification model, such as night, rain, backlight, and missing structures. Its generation is achieved through S... complexity =(s1+s2+…+s n ) / N implementation.

[0083] Structural deficiencies refer to the absence, damage, or unclear identification of key road structures such as lanes, roads, and traffic infrastructure in visual, radar, or fusion results, making it impossible for vehicles to accurately understand road boundaries and driving intentions. Night, rain, backlight, and structural deficiencies are environmental feature labels output by scene classification models or semantic segmentation models after processing image frames, used to represent the current driving environment's lighting, weather, and road structure status. These labels can coexist without conflict, belonging to different dimensions of environmental factors: for example, night + rain (occurring simultaneously), rain + backlight (entirely possible, for example, backlighting from oncoming headlights in heavy rain). Industry standards (such as autonomous driving data closed-loop routes) also consider scenarios with multiple overlapping factors to be more dangerous, representing typical corner case triggering conditions.

[0084] If different models or different sensors show significant differences in location, category, or trajectory predictions, it indicates that the current scene is difficult or there is a potential perceptual bias. Model consistency is an effective means of measuring perceptual robustness; it can capture abnormal model behavior and detect problems even if the confidence level does not fluctuate; in conjunction with multimodal fusion, it is particularly suitable for difficult scenarios such as "inconsistency between point cloud and image" and can also discover structural types that the model has not seen before, such as unlabeled construction sites and objects with special shapes.

[0085] In one embodiment of the present invention, obtaining the model consistency score includes: obtaining the maximum deviation distance D and the divergence tolerance threshold θ2; and generating a model consistency score based on the maximum deviation distance D and the divergence tolerance threshold θ2.

[0086] Specifically, the maximum deviation distance D can be obtained through the following steps: pairing the detection results of each model (e.g., Hungarian matching based on IoU); calculating the positional difference (IoU difference or center point normalized distance), class probability difference, and target number difference for the pairing results, and obtaining the single target difference d by linearly combining them according to weights. i The maximum difference at the frame level or the highest quantile is used as the comprehensive difference D, and a penalty term (e.g., d=1) is applied to the unpaired results.

[0087] The divergence tolerance threshold θ2 can be obtained by percentileing the statistical distribution of D on a normal scenario validation set (e.g., 95th percentile), or by using an intuitive threshold based on IoU, or by an adaptive approach based on scenario complexity.

[0088] Specifically, the model consistency score is used to compare positional or class differences among predictions from multiple models, and it is generated through S... divergence =min(1, D / θ2) to achieve this.

[0089] Multi-model refers to two or more independent perception models running simultaneously on the vehicle, used for parallel inference on the same image frame or sensor data to compare the consistency of their outputs. These models can be: models with different structures, such as Model A: YOLO series (for detection); Model B: Transformers detector; Model C: BEV-based 3D perception model; models for different tasks (producing outputs that need alignment), such as object detection models (output bounding boxes); lane detection models (output lane keypoints); semantic segmentation models (output pixel masks); and a main model + shadow model, such as the main model: the currently installed mass-produced model; and the shadow model: running silently in the background as a comparison model, but not affecting control.

[0090] In one embodiment of the present invention, when the multidimensional scoring result exceeds a preset scoring threshold, performing rule annotation on the image frame and its corresponding scene fragment and generating an annotation file includes: when the multidimensional scoring result exceeds the preset scoring threshold, performing rule annotation on the image frame and its corresponding scene fragment in the form of structured tags, wherein the content of the rule annotation includes at least static background information rule annotation, dynamic participant information rule annotation and system status rule annotation.

[0091] Specifically, the correspondence between the annotation file structure and the rule annotation is as follows.

[0092] After the annotation process is triggered, structured annotations are generated for image frames and their corresponding scene fragments that meet the preset rules, and written to an annotation file. The annotation file is encapsulated in JSON format and packaged with the corresponding vehicle-side raw data into a high-value data package. To facilitate backend processing and training, the annotation fields are organized according to three categories of annotation results.

[0093] Static background information rule annotation: describes elements or scene attributes in the scene that do not change rapidly over time, including fields such as scene_context (e.g., weather, illumination, road_type), road geometry / infrastructure labels, map reference ID, etc.

[0094] Dynamic participant information rule annotation: The structured label of each dynamic target is stored in the element list objects (each target is a structured label unit), including id, class, position, velocity, bounding box, confidence, occlusion, track_id, etc.; these fields are generated by the rule annotation module (in combination with detection / tracking / multi-sensor fusion).

[0095] System state / events: Represented by the system_state and corner_case_flags fields, these fields record the state and events (such as AEB trigger, manual_takeover, trajectory_spike, etc.) of the vehicle control module and higher-level policies within this frame / segment. This information is written by the vehicle-side CAN / control log and event triggers.

[0096] The annotation file contains timestamp, ego_pose (pose information), and scene_score (detailed scores for each dimension and total score), enabling cloud-based scene reconstruction, clustering, and training based on time and pose. The annotation file is sent via the vehicle-side upload queue in the form of a vehicle-side data packet plus the annotation file. The transmission protocol can use encapsulation methods such as ROS2 msg or TCP + JSON.

[0097] In one application scenario of this invention, all annotation files are generated locally using JSON field format and embedded in the uploaded data packet, including: timestamp UTC: millisecond timestamp; ego_pose: vehicle coordinates and orientation (e.g., latitude and longitude, heading angle, speed); scene_context: weather, illumination, road_type; objects: id, class, position, velocity, bounding box, confidence; system_state: current control state identifier, such as whether AEB / LKA is activated; corner_case_flags: such as low_confidence, manual_takeover, trajectory_spike, unseen_object, etc.; scene_score: detailed scores for each dimension + total score.

[0098] All tags are transmitted to the upload queue in the form of "vehicle-side data packet + tag structure" and then transmitted to the cloud system through the protocol layer (such as ROS2msg or TCP JSON encapsulation).

[0099] The structured label is a tag data unit that describes each target or scene element in an image frame in a segmented and standardized manner according to a preset tagging system. This label has a fixed format and field definitions to ensure consistency of data from different sources.

[0100] In one embodiment of the present invention, when the multidimensional scoring result exceeds a preset scoring threshold, after performing rule annotation on the image frame and generating an annotation file, the method further includes: performing context completion and confidence recording on the annotation file to obtain an updated annotation file.

[0101] Specifically, context completion is used to supplement and correct the consistency of the initial annotation content by using the context information of the current frame (including neighboring frames, multimodal data and system status) to solve problems such as unstable single-frame recognition, target occlusion, and semantic jumps, thereby improving the completeness and temporal continuity of the annotation.

[0102] Specifically, confidence records are used to attach perceived reliability information to each labeled item, enabling the cloud to perform operations such as sample quality control, low-confidence verification, and weighted model training, thereby further improving the effectiveness of training data and the robustness of the model.

[0103] In one embodiment of the present invention, storing the annotation file and the corresponding sensing data in priority order includes: dynamically queuing the stored data according to priority and feeding it back to the cloud; uploading in batches when the network is stable and delaying the reporting when the network is down.

[0104] This invention supports lightweight uploading and priority control, improves data loop efficiency, enables proactive identification of extreme scenarios without relying on manual backtracking, and enhances robust training capabilities.

[0105] As shown in Figure 2, a second aspect of the present invention provides a data processing system for extreme driving scenarios, comprising: an input module, at least for acquiring and inputting image frames; a lightweight strategy module, at least for filtering the input image frames and initiating a scoring process only for image frames that meet the filtering conditions; a scoring module, at least for scoring the image frames in the scoring process by means of inter-frame skip sampling or event triggering, and outputting multi-dimensional scoring results in real time; an annotation module, at least for performing rule annotation on the image frames and their corresponding scene segments and generating annotation files when the multi-dimensional scoring results exceed a preset scoring threshold, specifically, performing asynchronous inference and cache management on the scoring module; and a storage module, at least for storing the annotation files and corresponding vehicle-side data packets in priority order, wherein the vehicle-side data packets include at least image data, radar point cloud data, and CAN data.

[0106] In the scoring module, to reduce the computational burden on the main perception and control process, the present invention adopts an asynchronous reasoning and cache management mechanism.

[0107] For asynchronous inference: when an image frame or time segment that meets the scoring trigger condition arrives, the scoring task is not executed synchronously on the main thread, but is placed in an asynchronous task queue and inferred by an independent thread or computing unit; asynchronous inference allows the scoring module to calculate multi-dimensional scoring results in an event-driven or batch processing manner, thereby avoiding blocking the main control and perception threads and improving processing efficiency in high frame rate scenarios; after the inference is completed, the results are written back to the local cache for subsequent annotation and data backflow.

[0108] For cache management: Input frames, score outputs, and intermediate features for asynchronous inference are temporarily stored; the cache can be managed according to time order, frame number, or total score priority, with high-scoring frames or high-value segments being retained first, and low-scoring frames being processed later or discarded; cache management also supports subsequent processing operations such as context completion, confidence recording, and annotation file generation; high-value annotation files and corresponding original vehicle-side data packets can be dynamically queued according to cache priority to achieve efficient uploading and storage.

[0109] Through asynchronous inference and cache management, this invention can complete multi-dimensional scoring calculation, annotation generation, and priority return of high-value data without increasing the main thread latency, effectively alleviating computing power pressure and improving the system's ability to capture and manage extreme scenarios.

[0110] In one embodiment of the present invention, the lightweight strategy module further includes a filtering and judgment submodule, which is at least used to set filtering conditions, including setting confidence fluctuation conditions and vehicle status abnormality prompt conditions; when the input image frame meets the confidence fluctuation conditions and / or vehicle status abnormality prompt conditions, it indicates that the image frame meets the filtering conditions.

[0111] This invention relates to a scoring mechanism for extreme driving scenarios that features multimodal perception fusion, multi-model consistency analysis, and automatic behavior detection. It can reduce computational overhead through hierarchical processing and simultaneously address the drawbacks of high labeling costs, long processing times, and the difficulty in timely identification of some extreme driving scenarios by humans. The mechanism can automatically identify, filter, and label extreme driving scenarios on the vehicle side, improving data feedback efficiency. It can continuously and automatically build a high-value extreme driving sample set without relying on manual screening and intervention, thereby enhancing the stability and generalization ability of intelligent driving systems under long-tail distributions.

[0112] Furthermore, this invention can be adapted to natural driving data acquisition mechanisms, supports input from various sensors such as cameras, radar, and CAN bus, and integrates semantic segmentation, deep learning, and behavior detection algorithms to effectively improve the ability to acquire and manage extreme data.

[0113] As shown in FIG3, a third aspect of the present invention provides an electronic device, comprising: at least one processor; and at least one memory communicatively connected to the processor, wherein: the memory stores program instructions executable by the processor, and the processor invokes the program instructions to perform the steps of the method as described in any of the above embodiments.

[0114] A fourth aspect of the present invention discloses a readable storage medium storing a computer program, which is executed by a processor as described in any of the above embodiments.

[0115] Computer-readable storage media can include: any entity or device capable of carrying computer programs, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, read-only memory, random access memory, and software distribution media, etc. Computer programs include computer program code. Computer program code can be in the form of source code, object code, executable files, or some intermediate form, etc. Computer-readable storage media can include: any entity or device capable of carrying computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, read-only memory, random access memory, and software distribution media, etc.

[0116] Any process or method description in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or more executable instructions for implementing a particular logical function or process, and the scope of the preferred embodiments of the invention includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functions involved, as will be understood by those skilled in the art to which embodiments of the invention pertain.

[0117] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus or device (such as a computer-based system, a system including a processing module or other system that can fetch and execute instructions from, an instruction execution system, apparatus or device).

[0118] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A data backhaul method for extreme driving scenarios, characterized in that, include: Acquire and input image frames; The input image frames are filtered, and the scoring process is initiated only for image frames that meet the filtering criteria; In the scoring process, the image frames are scored by inter-frame skip sampling and / or event triggering, and multi-dimensional scoring results are output in real time. When the multi-dimensional scoring results exceed a preset scoring threshold, rule annotation is performed on the image frames and their corresponding scene segments to generate annotation files. The annotation files and the corresponding vehicle-side data packets are stored in priority order, and the vehicle-side data packets include at least image data, radar point cloud data, and CAN data.

2. The data backhaul method for extreme driving scenarios according to claim 1, characterized in that, Before filtering the input image frames, the following steps are taken: setting filtering conditions, including setting confidence fluctuation conditions and / or vehicle status abnormality prompt conditions; when the input image frame meets the confidence fluctuation conditions and / or vehicle status abnormality prompt conditions, it indicates that the image frame meets the filtering conditions.

3. The data backhaul method for extreme driving scenarios according to claim 1, characterized in that, In the scoring process, the image frames are scored by inter-frame skip sampling and / or event triggering, and multi-dimensional scoring results are output in real time, including: obtaining a perceptual uncertainty score; obtaining a behavior trigger score; obtaining an environmental complexity score; obtaining a model consistency score; normalizing the perceptual uncertainty score, the behavior trigger score, the environmental complexity score, and the model consistency score respectively, and obtaining the multi-dimensional scoring results by weighted fusion.

4. The data backhaul method for extreme driving scenarios according to claim 3, characterized in that, The process of obtaining the perceptual uncertainty score includes: obtaining the detection confidence standard deviation and confidence fluctuation threshold of the image frame; and generating a perceptual uncertainty score based on the ratio of the detection confidence standard deviation and the confidence fluctuation threshold.

5. The data backhaul method for extreme driving scenarios according to claim 3, characterized in that, The process of obtaining the behavior trigger score includes: obtaining an adjustment coefficient and the number of triggers per unit time; and generating a behavior trigger score based on the adjustment coefficient and the number of triggers per unit time.

6. The data backhaul method for extreme driving scenarios according to claim 3, characterized in that, The process of obtaining the environmental complexity score includes: obtaining the scores of each scenario factor; weighting and averaging the scores of each scenario factor to generate the environmental complexity score.

7. The data backhaul method for extreme driving scenarios according to claim 3, characterized in that, The process of obtaining the model consistency score includes: obtaining the maximum deviation distance and the divergence tolerance threshold; and generating a model consistency score based on the maximum deviation distance and the divergence tolerance threshold.

8. The data backhaul method for extreme driving scenarios according to claim 1, characterized in that, When the multidimensional scoring result exceeds a preset scoring threshold, rule annotation is performed on the image frame and its corresponding scene fragment to generate an annotation file. This includes: taking the image frame that triggered the scoring as the center, obtaining image frames within a preset time window before and after it to form the scene fragment corresponding to the image frame; when the multidimensional scoring result exceeds the preset scoring threshold, rule annotation is performed on the image frame and its corresponding scene fragment in the form of structured tags. The content of the rule annotation includes at least static background information rule annotation, dynamic participant information rule annotation, and system status rule annotation.

9. The data backhaul method for extreme driving scenarios according to claim 1, characterized in that, When the multidimensional scoring result exceeds the preset scoring threshold, after performing rule annotation on the image frame and its corresponding scene fragment and generating an annotation file, the method further includes: performing context completion and confidence recording on the annotation file to obtain an updated annotation file.

10. The data backhaul method for extreme driving scenarios according to claim 1, characterized in that, The step of storing the labeled files and corresponding sensing data in priority order includes: dynamically queuing the stored data according to priority and feeding it back to the cloud.

11. A data backhaul system for extreme driving scenarios, characterized in that, include: The input module is used at least to acquire and input image frames; The lightweight strategy module is used at least to filter the input image frames, and only initiates the scoring process for image frames that meet the filtering conditions; The scoring module is used, at least in the scoring process, to score the image frame by means of inter-frame skip sampling or event triggering, and to output multi-dimensional scoring results in real time. The annotation module is at least used to perform rule annotation on the image frame and its corresponding scene fragments and generate an annotation file when the multidimensional scoring result exceeds a preset scoring threshold. The storage module is used to store the annotation file and the corresponding vehicle-side data packet in priority order, wherein the vehicle-side data packet includes at least image data, radar point cloud data and CAN data.

12. The data backhaul system for extreme driving scenarios according to claim 11, characterized in that, The lightweight strategy module also includes a filtering and judgment submodule, which is used to set filtering conditions, including setting confidence fluctuation conditions and vehicle status abnormality prompt conditions; when the input image frame meets the confidence fluctuation conditions and / or vehicle status abnormality prompt conditions, it means that the image frame meets the filtering conditions.

13. An electronic device, characterized in that, include: At least one processor; And at least one memory communicatively connected to the processor, wherein: the memory stores program instructions executable by the processor, and the processor invokes the program instructions to perform the steps of the data backhaul method for extreme driving scenarios as described in any one of claims 1-10.

14. A readable storage medium storing a computer program, characterized in that, The computer program is executed by a processor using the steps of the data backhaul method for extreme driving scenarios as described in any one of claims 1-10.

Images