A method and device for confirming differences in vehicle-side shadow mode driving

Abstract

This disclosure provides a method and apparatus for confirming discrepancies in vehicle-side shadow mode driving. It receives multimodal data such as video, trajectory, and vehicle status uploaded from the vehicle via a cloud-based visual language model. It performs secondary confirmation on initial discrepancy events and outputs valid discrepancy judgments, discrepancy causes, and scene labels, thereby filtering out invalid discrepancies and retaining high-quality samples with genuine algorithmic optimization value. This allows the vehicle to perform only lightweight initial discrepancy detection and data uploading, while the complex multimodal understanding, long-term time-series analysis, validity confirmation, and root cause analysis are handled by the cloud. This reduces the computational burden on the vehicle, improves the accuracy of discrepancy data collection, enables automated explanation of discrepancy causes and scene labeling, and enhances the iteration efficiency and reliability of autonomous driving algorithms for complex and long-tail scenarios.

Description

Technical Field

[0001] This disclosure relates to the field of autonomous driving data processing technology, and more specifically, to a method and apparatus for confirming differences in vehicle-side shadow mode driving. Background Technology

[0002] With the development of autonomous driving technology, autonomous driving algorithms typically rely on a large amount of real-world road scenario data for training, validation, and iterative optimization. To identify potential problems in autonomous driving algorithms without affecting normal vehicle driving control, shadow mode has gradually become an important technical means in the data closed loop of autonomous driving systems. Shadow mode refers to a scenario where, while the vehicle is actually controlled by a human driver, the on-board autonomous driving algorithm runs in parallel in the background, outputting its perception, prediction, planning, or control decision results in the current scenario. Subsequently, by comparing the predicted or planned trajectory output by the autonomous driving algorithm with the actual driving behavior of the human driver, it identifies whether there are significant differences between the two. When the differences meet certain conditions, data collection is triggered for subsequent algorithm analysis and optimization.

[0003] Existing shadow mode solutions typically rely primarily on vehicle-side computing resources to perform difference detection. For example, the vehicle can acquire real-time data from vehicle sensors, vehicle status data, and control signals from the human driver such as steering wheel, accelerator, and brake, while simultaneously obtaining short-term planned trajectories output by autonomous driving algorithms. Then, the vehicle-side processing unit performs a geometric comparison between the human driving trajectory and the algorithm-predicted trajectory. When the distance, angle, or deviation between the two trajectories exceeds a preset threshold, the scenario is identified as a difference scenario, triggering the recording or uploading of relevant data.

[0004] However, due to limited computing resources on the vehicle side, only short-term, localized, and rule-based trajectory comparisons can typically be completed, making it difficult to support long-term inference and deep scene understanding in complex scenarios. When a vehicle is in a complex road environment, a simple trajectory deviation does not necessarily indicate a defect in the autonomous driving algorithm. For example, human drivers may produce a driving trajectory different from the algorithm's planned trajectory due to distraction, personal driving habits, or unnecessary operations. If data feedback is triggered solely based on the degree of trajectory deviation, invalid differences may be misjudged as valuable differences, leading to the collection of a large amount of redundant data and reducing data closure efficiency. Summary of the Invention

[0005] This disclosure provides at least one method and apparatus for confirming vehicle-side shadow mode driving differences. It receives multimodal data such as video, trajectory, and vehicle status uploaded from the vehicle via a cloud-based visual language model. It performs secondary confirmation on initial difference events and outputs valid difference judgments, difference causes, and scene labels, thereby filtering out invalid differences and retaining high-quality samples with genuine algorithmic optimization value. This allows the vehicle to perform only lightweight initial difference detection and data uploading, while the complex multimodal understanding, long-term time-series analysis, validity confirmation, and root cause analysis are handled by the cloud. This reduces the computational burden on the vehicle, improves the accuracy of difference data collection, enables automated explanation of difference causes and scene labeling, and enhances the iteration efficiency and reliability of autonomous driving algorithms for complex and long-tail scenarios.

[0006] This disclosure provides a method for confirming differences in vehicle-side shadow mode driving, including: Acquire vehicle-side driving-related data during vehicle operation, including at least environmental perception data, vehicle status data, human driving data, and algorithm decision data output by autonomous driving algorithms. Based on the human driving data and the algorithm decision data, the human driving trajectory and the algorithm predicted trajectory are determined respectively; The human driving trajectory and the algorithm-predicted trajectory are compared for differences. When a preset difference trigger condition is met, the corresponding preliminary difference event is determined. Based on the preliminary difference event, event data corresponding to the preliminary difference event is extracted from the vehicle-side driving association data, and the event data is uploaded to the cloud analysis platform; The cloud-based analysis platform constructs a visual language model based on the event data as multimodal analysis input, and uses the cloud-based visual language model to confirm the validity of the preliminary difference events and analyze the reasons for the differences, thereby obtaining the difference confirmation results; Based on the difference confirmation results, the event data is filtered and labeled to obtain difference sample data for autonomous driving algorithm optimization.

[0007] In one optional implementation, the human driving trajectory and the algorithm-predicted trajectory are determined based on the human driving data and the algorithm decision data, respectively, specifically including: Based on the human driving data and vehicle dynamics model, the human driving trajectory of the vehicle in the target prediction time domain is determined. Based on the algorithm decision data, the algorithm prediction trajectory output by the autonomous driving algorithm in the same target prediction time domain is obtained; The human driving trajectory and the algorithm-predicted trajectory are converted to the same time reference and the same vehicle coordinate system or road coordinate system to obtain comparable trajectory data.

[0008] In one optional implementation, the human driving trajectory and the algorithm-predicted trajectory are compared for differences. When a preset difference trigger condition is met, a corresponding preliminary difference event is determined, specifically including: Calculate the trajectory deviation index between the human driving trajectory and the algorithm-predicted trajectory; The trajectory deviation index is compared with a preset difference threshold. When the trajectory deviation index exceeds the preset difference threshold, or when the trajectory deviation index continuously meets the deviation condition within a preset duration, the initial difference event is determined to have occurred. The trajectory deviation index includes at least one of the following: average trajectory distance, maximum lateral offset distance, trajectory endpoint deviation, trajectory curvature difference, and trajectory Hausdorff distance.

[0009] In one optional implementation, based on the preliminary difference event, event data corresponding to the preliminary difference event is extracted from the vehicle-side driving association data, and the event data is uploaded to a cloud analysis platform, specifically including: Determine the trigger time corresponding to the initial difference event; Based on the triggering time, the event time window is determined according to the preset forward time length and the preset backward time length; The environmental perception data, vehicle status data, human driving data, algorithm decision data, human driving trajectory, and algorithm predicted trajectory are extracted within the event time window. The captured data is associated with and encapsulated with event identifiers, timestamp information, vehicle location information, and difference trigger information to obtain an event data packet; After the event data packets are compressed and de-identified, they are uploaded to the cloud analysis platform.

[0010] In one optional implementation, the cloud-based analytics platform constructs a multimodal analysis input for the visual language model based on the event data, specifically including: Extract video frame sequences and trajectory data from the event data; The human driving trajectory, the algorithm-predicted trajectory, and the cloud reference trajectory are mapped to the video frame sequence or the corresponding bird's-eye view scene map to form scene input data with trajectory labels; Based on the scene input data, trajectory difference information, and preset analysis task text, a multimodal analysis input for the visual language model is constructed; The preset analysis task text is used to instruct the visual language model to determine whether the preliminary difference event is a valid difference that the autonomous driving algorithm needs to learn, and to output the reason for the difference and the scene label.

[0011] In one optional implementation, the validity of the preliminary difference event and the cause of the difference are confirmed by a cloud-based visual language model to obtain the difference confirmation result, specifically including: The cloud-based visual language model is used to perform multimodal understanding of environmental targets, road structures, traffic participants, vehicle trajectories, and driving behaviors in the event data. Based on the multimodal understanding results, determine whether the difference between the human driving trajectory and the algorithm-predicted trajectory is caused by at least one of the following reasons: effective driving intention, potential risk avoidance, abnormal behavior of traffic participants, missed perception, insufficient prediction, or unreasonable planning. When it is determined that the difference is related to the defects of the autonomous driving algorithm or the learning scenario of the algorithm, the valid difference result is output; If the discrepancy is determined to be caused by random operation by a human driver, driver distraction, unnecessary lane changes, abnormal human intervention, or behavior with no learning value, an invalid discrepancy result is output.

[0012] In one optional implementation, based on the difference confirmation results, the event data is filtered and labeled to obtain difference sample data for optimizing autonomous driving algorithms, specifically including: When the difference confirmation result indicates that the preliminary difference event is a valid difference, the event data is associated with the corresponding root cause analysis information, scene label information and trajectory difference information and stored in the optimization database. When the difference confirmation result indicates that the preliminary difference event is an invalid difference, the event data is filtered, stored with reduced weight, or classified into the invalid difference database. Based on the differential sample data in the optimized database, a dataset is formed for training, testing, or replaying autonomous driving algorithms.

[0013] In one optional implementation, after receiving the event data, the cloud analytics platform further performs cloud trajectory reconstruction processing, which includes: Based on the historical environmental perception data, vehicle status data, and trajectory data in the event data, the scene context information corresponding to the preliminary difference event is determined; A cloud-based trajectory prediction model is used to perform long-term time-series trajectory prediction on the scene context information to generate a cloud-based reference trajectory. The cloud-based reference trajectory, the human driving trajectory, and the algorithm-predicted trajectory are input into the cloud-based visual language model to assist the cloud-based visual language model in determining whether the algorithm-predicted trajectory has perception, prediction, or planning defects.

[0014] In one optional implementation, before the event data is uploaded to the cloud analytics platform, the following steps are further included: Determine whether the preliminary difference event meets the upload conditions; The upload conditions include: the trajectory deviation reaches a preset upload threshold, the event location belongs to the target road area, the event data integrity meets the preset integrity requirements, the network status meets the upload requirements, and the event type does not belong to at least one of the preset filtering types. When the upload conditions are met, the event data is uploaded to the cloud analysis platform; If the upload conditions are not met, the event data may be cached locally, uploaded with a delay, or discarded.

[0015] This disclosure also provides a vehicle-side shadow mode driving difference confirmation device, including: The data acquisition module is used to acquire vehicle-side driving-related data during vehicle operation. The vehicle-side driving-related data includes at least environmental perception data, vehicle status data, human driving data, and algorithm decision data output by the autonomous driving algorithm. The shadow mode processing module is used to determine the human driving trajectory and the algorithm predicted trajectory based on the human driving data and the algorithm decision data, respectively. The difference judgment module is used to compare the human driving trajectory and the algorithm-predicted trajectory, and determine the corresponding preliminary difference event when the preset difference triggering conditions are met. The vehicle-side communication module is used to extract event data corresponding to the preliminary difference event from the vehicle-side driving association data based on the preliminary difference event, and upload the event data to the cloud analysis platform; The cloud-based analysis module is used to perform multimodal analysis input of the visual language model constructed by the cloud-based analysis platform based on the event data, and to perform validity confirmation and difference cause analysis on the preliminary difference event through the cloud-based visual language model to obtain the difference confirmation result; The result generation module is used to filter and label the event data based on the difference confirmation results to obtain difference sample data for autonomous driving algorithm optimization.

[0016] This disclosure also provides an electronic device, including: a processor, a memory, and a bus. The memory stores machine-readable instructions executable by the processor. When the electronic device is running, the processor communicates with the memory via the bus. When the machine-readable instructions are executed by the processor, the steps of the above-described vehicle-side shadow mode driving difference confirmation method, or any possible implementation of the above-described vehicle-side shadow mode driving difference confirmation method, are performed.

[0017] This disclosure also provides a computer-readable storage medium storing a computer program that, when executed by a processor, performs the steps of the above-described vehicle-side shadow mode driving difference confirmation method, or any possible implementation of the above-described vehicle-side shadow mode driving difference confirmation method.

[0018] This disclosure also provides a computer program product, including a computer program / instruction, which, when executed by a processor, implements the above-described vehicle-side shadow mode driving difference confirmation method, or any possible implementation of the above-described vehicle-side shadow mode driving difference confirmation method.

[0019] This disclosure provides a method and apparatus for confirming differences in vehicle-side shadow mode driving. It receives multimodal data such as video, trajectory, and vehicle status uploaded from the vehicle via a cloud-based visual language model. It performs secondary confirmation on initial difference events and outputs valid difference judgments, difference causes, and scene labels, thereby filtering out invalid differences and retaining high-quality samples with genuine algorithmic optimization value. This allows the vehicle to perform only lightweight initial difference detection and data uploading, while the complex multimodal understanding, long-term time-series analysis, validity confirmation, and root cause analysis are handled by the cloud. This reduces the computational burden on the vehicle, improves the accuracy of difference data collection, enables automated explanation of difference causes and scene labeling, and enhances the iteration efficiency and reliability of autonomous driving algorithms for complex and long-tail scenarios.

[0020] To make the above-mentioned objects, features and advantages of this disclosure more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings. Attached Figure Description

[0021] To more clearly illustrate the technical solutions of the embodiments of this disclosure, the accompanying drawings used in the embodiments will be briefly described below. These drawings are incorporated in and constitute a part of this specification. They illustrate embodiments conforming to this disclosure and, together with the specification, serve to explain the technical solutions of this disclosure. It should be understood that the following drawings only show some embodiments of this disclosure and should not be considered as limiting the scope. Those skilled in the art can obtain other related drawings based on these drawings without creative effort.

[0022] Figure 1 A flowchart of a method for confirming differences in vehicle-side shadow mode driving provided by an embodiment of this disclosure is shown; Figure 2 A flowchart of another vehicle-side shadow mode driving difference confirmation method provided by an embodiment of this disclosure is shown; Figure 3A schematic diagram of a vehicle-side shadow mode driving difference confirmation device provided in an embodiment of this disclosure is shown; Figure 4 A schematic diagram of an electronic device provided in an embodiment of the present disclosure is shown. Detailed Implementation

[0023] To make the objectives, technical solutions, and advantages of the embodiments of this disclosure clearer, the technical solutions of the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this disclosure, and not all of them. The components of the embodiments of this disclosure described and shown in the accompanying drawings can generally be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of this disclosure provided in the accompanying drawings is not intended to limit the scope of the claimed disclosure, but merely represents selected embodiments of this disclosure. All other embodiments obtained by those skilled in the art based on the embodiments of this disclosure without inventive effort are within the scope of protection of this disclosure.

[0024] It should be noted that similar labels and letters in the following figures indicate similar items. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures.

[0025] In this document, the term "and / or" merely describes a relationship, indicating that three relationships can exist. For example, A and / or B can represent three cases: A alone, A and B simultaneously, and B alone. Furthermore, the term "at least one" in this document means any combination of at least two of any one or more elements. For example, including at least one of A, B, and C can mean including any one or more elements selected from the set consisting of A, B, and C.

[0026] Research has revealed that existing shadow mode solutions, due to limited on-vehicle computing resources, typically only enable short-term, localized, and rule-based trajectory comparisons, making it difficult to support long-term inference and deep scene understanding in complex scenarios. When vehicles are in complex road environments, simple trajectory deviations do not necessarily indicate a flaw in the autonomous driving algorithm. For example, human drivers may exhibit trajectories different from the algorithm's planned trajectory due to distraction, personal driving habits, or unnecessary operations. If data feedback is triggered solely based on the degree of trajectory deviation, invalid differences may be misjudged as valuable differences, leading to the collection of large amounts of redundant data and reducing data loop efficiency.

[0027] Based on the above research, this disclosure provides a method and apparatus for confirming vehicle-side shadow mode driving differences. It receives multimodal data such as video, trajectory, and vehicle status uploaded from the vehicle via a cloud-based visual language model, performs secondary confirmation on initial difference events, and outputs valid difference judgments, difference causes, and scene labels. This filters out invalid differences and retains high-quality samples with genuine algorithm optimization value. The vehicle only performs lightweight initial difference detection and data uploading, while the complex multimodal understanding, long-term time-series analysis, validity confirmation, and root cause analysis are handled by the cloud. This reduces the computational burden on the vehicle, improves the accuracy of difference data collection, enables automated explanation of difference causes and scene labeling, and enhances the iteration efficiency and reliability of autonomous driving algorithms for complex and long-tail scenarios.

[0028] To facilitate understanding of this embodiment, a detailed description of the vehicle-side shadow mode driving difference confirmation method disclosed in this disclosure embodiment will be provided first. The executing entity of the vehicle-side shadow mode driving difference confirmation method provided in this disclosure embodiment is generally a computer device with a certain computing capability. This computer device may include, for example, a terminal device, a server, or other processing devices. The terminal device may be a user equipment (UE), mobile device, user terminal, terminal, cellular phone, cordless phone, personal digital assistant (PDA), handheld device, computing device, in-vehicle device, wearable device, etc. In some possible implementations, the vehicle-side shadow mode driving difference confirmation method can be implemented by a processor calling computer-readable instructions stored in memory.

[0029] See Figure 1 The diagram shows a flowchart of a method for confirming differences in vehicle-side shadow mode driving according to an embodiment of this disclosure. The method includes steps S101 to S106, wherein: S101. Obtain vehicle-side driving-related data during vehicle operation. The vehicle-side driving-related data includes at least environmental perception data, vehicle status data, human driving data, and algorithm decision data output by the autonomous driving algorithm.

[0030] In practical implementation, the vehicle-side subsystem continuously acquires vehicle-side driving-related data during vehicle operation. This data is used to characterize the vehicle's external environment, its own motion state, the actual operating behavior of the human driver, and the driving decision results generated by the autonomous driving algorithm in shadow mode. This provides foundational data for subsequent comparisons of human driving trajectories with algorithm-predicted trajectories, preliminary identification of discrepancies, and in-depth confirmation in the cloud.

[0031] Specifically, the vehicle terminal system may include a data acquisition module, which is communicatively connected to a vehicle environment perception device, a vehicle status acquisition device, a vehicle control signal acquisition device, and an autonomous driving algorithm module. The vehicle environment perception device may include at least one of an onboard camera, a surround-view camera, LiDAR, millimeter-wave radar, and ultrasonic radar; the vehicle status acquisition device may include at least one of a vehicle positioning unit, an inertial measurement unit, a vehicle speed sensor, an acceleration sensor, and a yaw rate sensor; the vehicle control signal acquisition device may acquire the actual control signals from the human driver via the vehicle CAN bus, Ethernet bus, or the domain controller's internal communication interface; and the autonomous driving algorithm module is used to operate in parallel in shadow mode without taking over actual vehicle control and outputs corresponding algorithm decision data.

[0032] Here, environmental perception data is used to reflect the external traffic environment during vehicle operation. In specific implementations, environmental perception data may include video streams or image frame sequences collected by front-view cameras, side-view cameras, rear-view cameras, or surround-view cameras, as well as point cloud data collected by lidar, target distance and relative speed data collected by millimeter-wave radar, and near-range obstacle data collected by ultrasonic radar.

[0033] Here, vehicle status data is used to reflect the vehicle's own motion state and position information during driving. In specific implementations, vehicle status data may include at least one of the following: vehicle speed, longitudinal acceleration, lateral acceleration, yaw rate, heading angle, gear status, braking status, steering status, GPS coordinates, road positioning information, and vehicle attitude information.

[0034] Here, human driving data is used to reflect the operational behavior of human drivers during actual vehicle operation. In specific implementations, human driving data may include at least one of the following: steering wheel angle, steering wheel angular velocity, accelerator pedal opening, brake pedal pressure, brake pedal travel, turn signal status, gear shifting operations, and driving mode switching operations. Since the vehicle is usually still under the actual control of a human driver in shadow mode, human driving data can be used to characterize the actual driving choices made by the human driver in the current road scenario.

[0035] Here, algorithm decision data is used to reflect the background decision results generated by the autonomous driving algorithm in the current driving scenario during shadow mode operation. In specific implementation, the autonomous driving algorithm module can output at least one of the following in the background based on the vehicle's current environmental perception data and vehicle state data: perception results, target tracking results, behavior prediction results, planned trajectory point set, target speed, target acceleration, target lane, lane change decision, avoidance decision, and braking decision.

[0036] During the data acquisition process, the data acquisition module can synchronously record environmental perception data, vehicle status data, human driving data, and algorithm decision data according to a unified timestamp. For example, the data acquisition module can use the system clock of the vehicle domain controller, the positioning and timing signal, or the vehicle network synchronization clock as a unified time reference to add timestamps to data from different sources and establish the correspondence between data frames, vehicle status sampling points, driving control signal sampling points, and algorithm decision results.

[0037] Furthermore, the data acquisition module can also preprocess the acquired vehicle-side driving-related data. Preprocessing can include at least one of the following: data format conversion, time synchronization, coordinate transformation, outlier removal, data compression, and cache management. For example, camera video streams can be cached according to a preset encoding format, target information detected by sensors can be converted to a vehicle coordinate system or a road coordinate system, vehicle CAN bus data can be parsed into physical quantities that can be used for subsequent calculations, and outlier sampled values ​​that are clearly outside the reasonable range can be removed.

[0038] In one specific example, while the vehicle is driving on a normal road, the vehicle-mounted subsystem continuously collects video streams from the forward-facing camera, vehicle speed, vehicle acceleration, GPS coordinates, steering wheel angle, accelerator pedal opening, brake pedal pressure, and a set of trajectory points planned for the next few seconds by the autonomous driving algorithm. This data is written to the vehicle-mounted cache with timestamps and stored in a rolling cache format for the most recent period. When a significant difference is subsequently detected between the human driving trajectory and the algorithm-predicted trajectory, the vehicle-mounted subsystem can extract relevant data from the cache within a time window before and after the difference event, forming an event data package for uploading to the cloud analysis platform.

[0039] S102. Based on the human driving data and the algorithm decision data, determine the human driving trajectory and the algorithm predicted trajectory respectively.

[0040] In practical implementation, after acquiring vehicle-side driving-related data during vehicle operation, the vehicle-to-vehicle subsystem determines the human driving trajectory and the algorithm-predicted trajectory based on human driving data and algorithm decision data, respectively. The human driving trajectory represents the actual or expected movement path of the vehicle under the current actual operation by a human driver, while the algorithm-predicted trajectory represents the planned or predicted driving path output by the autonomous driving algorithm in shadow mode for the same driving scenario.

[0041] Specifically, human driving data can include at least one of the following: steering wheel angle, steering wheel angular velocity, accelerator pedal opening, brake pedal pressure, brake pedal travel, gear shift status, turn signal status, and driving mode status. The vehicle-to-vehicle terminal system can determine the human driving trajectory corresponding to the current moment and a future target time range based on the above human driving data, combined with vehicle state data and a vehicle dynamics model.

[0042] For example, the steering wheel angle can be used to determine the front wheel angle or yaw rate of a vehicle, and the accelerator pedal opening and brake pedal pressure can be used to determine the longitudinal acceleration of a vehicle. By combining the vehicle's current speed, current position, heading angle, and yaw rate, multiple trajectory points of the vehicle within a preset time range in the future can be calculated, thus forming a human driving trajectory.

[0043] In another implementation, the human driving trajectory can also be determined in reverse based on the actual driving state of the vehicle. Specifically, the vehicle terminal system can obtain the actual position sequence of the vehicle at continuous moments during continuous driving based on vehicle state data such as positioning information, inertial measurement information, vehicle speed, heading angle, and yaw rate, and use this actual position sequence as the human driving trajectory.

[0044] In practical implementation, the vehicle-to-vehicle (V2V) terminal system can pre-define the target prediction time domain. This target prediction time domain can be several seconds after the current moment, or it can be a continuous time window centered on the current moment. For example, at the current sampling moment, the V2V terminal system can calculate the vehicle's expected movement path within the next three to five seconds based on the human driver's current control input and the vehicle's current motion state; or, after a differential event is triggered, it can form a corresponding human driving trajectory based on the vehicle's actual driving position within a certain time window before and after the event.

[0045] Here, for the algorithm-predicted trajectory, the vehicle-to-vehicle subsystem can obtain it from the algorithm decision data output by the autonomous driving algorithm module. The algorithm decision data may include a set of future trajectory points generated by the autonomous driving algorithm based on the current perception results, prediction results, and planning strategies. The set of future trajectory points may include multiple trajectory points arranged in chronological order, and each trajectory point may include at least one of the following: position coordinates, heading angle, target velocity, target acceleration, curvature, and timestamp.

[0046] The vehicle terminal system can use the future trajectory point set as the algorithm's predicted trajectory, or it can perform interpolation, smoothing, resampling, and other processing on the future trajectory point set to obtain the algorithm's predicted trajectory for comparison with human driving trajectories.

[0047] To ensure comparability between human driving trajectories and algorithm-predicted trajectories, the vehicle-to-vehicle terminal system can perform time synchronization and coordinate unification processing for the two types of trajectories. Specifically, based on a unified timestamp, trajectory points in the human driving trajectory can be aligned to the same or similar time nodes with those in the algorithm-predicted trajectory; for trajectory data with different sampling frequencies, trajectory point sequences with consistent time intervals can be generated through interpolation, resampling, or trajectory fitting.

[0048] At the same time, both types of trajectories can be uniformly converted to vehicle coordinate system, road coordinate system, map coordinate system, or local coordinate system with the current position of the vehicle as the origin, so as to avoid trajectory deviation calculation distortion caused by inconsistency of coordinate systems.

[0049] Furthermore, the vehicle-to-vehicle terminal system can also perform validity checks on human driving trajectories and algorithm-predicted trajectories. Validity checks can include at least one of the following: trajectory point integrity checks, timestamp continuity checks, trajectory point outlier checks, trajectory curvature rationality checks, and speed change rationality checks.

[0050] For example, when the sudden change in position corresponding to a certain trajectory point is significantly beyond the range of the vehicle's physical movement capability, or when trajectory data is missing within a certain time period, the trajectory point can be removed, interpolated, or the comparison weight of that time period can be reduced.

[0051] In one specific example, when the vehicle is driving in shadow mode, the autonomous driving algorithm module outputs a set of planned trajectory points for the next three to five seconds every preset period. Simultaneously, the vehicle-to-vehicle (V2V) subsystem collects the driver's steering wheel angle, accelerator pedal opening, and brake pedal pressure via the CAN bus. Combining this with the vehicle's current speed, heading angle, and yaw rate, it calculates the human driving trajectory within the same time range based on the vehicle's kinematics model. Subsequently, the V2V subsystem transforms both the human driving trajectory and the algorithm-predicted trajectory to a local coordinate system with the current vehicle position as the origin, and resamples the trajectory points at equal time intervals, thus obtaining two directly comparable trajectory sequences.

[0052] In another specific example, after the vehicle completes a certain distance of actual driving, the vehicle-to-vehicle subsystem can form the actual driving trajectory based on the vehicle's positioning and attitude data within the event time window, and use this actual driving trajectory as the human driving trajectory. Simultaneously, the planned trajectories output by the autonomous driving algorithm at various sampling moments within the same time window are spliced ​​or selected to form the algorithm's predicted trajectory. This method is particularly suitable for retrospective analysis of pre-triggered discrepancy events, ensuring that subsequent event data uploaded to the cloud simultaneously includes both the actual human driving results and the algorithm's predictions or planning results at that time.

[0053] S103. Compare the human driving trajectory and the algorithm-predicted trajectory to determine the corresponding preliminary difference event when the preset difference triggering condition is met.

[0054] In practical implementation, after determining the human driving trajectory and the algorithm-predicted trajectory, the vehicle-to-vehicle terminal system compares the differences between the two trajectories. When the comparison result meets the preset difference triggering conditions, a corresponding preliminary difference event is determined. The preliminary difference event indicates that there is a significant inconsistency between the prediction or planning result output by the autonomous driving algorithm in shadow mode and the actual driving behavior of the human driver, requiring further collection of relevant scene data and submission to the cloud analysis platform for in-depth confirmation.

[0055] Specifically, the vehicle-to-vehicle terminal system can transform human driving trajectories and algorithm-predicted trajectories to the same coordinate system and time reference, and align trajectory points according to the same time intervals or trajectory length intervals. For two trajectories with different sampling frequencies, different numbers of trajectory points, or different trajectory output periods, interpolation, resampling, trajectory fitting, or trajectory segment matching can be used to create comparable trajectory point pairs at multiple corresponding times or path locations. This avoids misjudgments caused by differences in trajectory sampling and improves the accuracy of difference comparison.

[0056] In practice, trajectory difference indices can be calculated based on the aligned trajectory point pairs. These indices may include at least one of the following: average distance, maximum distance, lateral offset distance, longitudinal offset distance, trajectory endpoint deviation, heading angle difference, velocity difference, acceleration difference, curvature difference, and trajectory shape similarity between the human driving trajectory and the algorithm-predicted trajectory.

[0057] For example, the lateral offset between each corresponding trajectory point can be calculated separately, and the average or maximum value of multiple lateral offsets can be used as the trajectory lateral deviation index; alternatively, the distance between the endpoints of two trajectories at the end of the target prediction time domain can be calculated and used as the trajectory endpoint deviation index.

[0058] Furthermore, difference comparison is not limited to a single distance indicator; it can also be based on a combination of trajectory features across multiple dimensions. For example, in highway scenarios, the weights of longitudinal speed differences, target lane differences, and lane-changing intention differences can be increased; in urban road scenarios, the weights of lateral deviation, deceleration behavior, avoidance behavior, and heading angle changes can be increased; and in low-speed parking or congestion scenarios, the weights of near-distance obstacle avoidance trajectories, braking timing, and acceleration changes can be increased.

[0059] In one specific implementation, the vehicle terminal system can compare the trajectory difference index with a preset difference threshold. When the average distance between the human driving trajectory and the algorithm-predicted trajectory is greater than a first preset threshold, or the maximum lateral offset distance between the two trajectories is greater than a second preset threshold, or the endpoint deviation between the two trajectories is greater than a third preset threshold, it can be determined that the preset difference triggering condition is met, and the corresponding preliminary difference event is identified.

[0060] It should be noted that the above preset thresholds can be pre-configured based on vehicle type, speed range, road grade, driving scenario type, or autonomous driving algorithm verification requirements, or can be dynamically adjusted based on historical data statistics.

[0061] In another specific implementation, to avoid false triggering due to sensor noise, short-term jitter, or instantaneous control fluctuations, the vehicle terminal system can be configured with continuous judgment conditions. Specifically, when the trajectory difference index exceeds the corresponding preset threshold, a preliminary difference event is not immediately triggered. Instead, it is determined whether the trajectory difference index continues to exceed the threshold within a preset duration, or whether the difference condition is met for multiple consecutive sampling periods.

[0062] In this way, a preliminary difference event is only determined to have occurred when the trajectory difference meets the persistence requirement. This reduces the uploading of invalid events caused by occasional noise or transient deviations, and improves the stability of the initial screening at the vehicle end.

[0063] Furthermore, the preset difference triggering conditions can also include behavioral semantic level triggering conditions. Specifically, the vehicle-to-vehicle terminal system can identify differences in driving behavior based on human driving trajectory and algorithm-predicted trajectory. For example, human driving trajectory may show deceleration and avoidance while algorithm-predicted trajectory may show maintaining speed; human driving trajectory may show lane changing and detour while algorithm-predicted trajectory may show maintaining the original lane; or human driving trajectory may show early braking while algorithm-predicted trajectory may show delayed braking or no braking.

[0064] Here, when the aforementioned behavioral semantic differences meet the preset conditions, even if the geometric distance between the two trajectories has not yet reached a large threshold, it can still be identified as a preliminary difference event to avoid missing difference scenarios with potential security value.

[0065] Furthermore, the vehicle-to-vehicle terminal system can also constrain the difference triggering results by combining vehicle status data. For example, when the vehicle is in low-speed maneuvering, manual takeover, active turn signal activation, temporary parking, reversing, or abnormal driving mode, the triggering priority of the corresponding preliminary difference event can be reduced, or such events can be marked as events to be confirmed; when the vehicle is in normal driving state, and there are traffic participants, road structure changes, or risk targets ahead, the difference triggering sensitivity can be increased.

[0066] In one specific example, when the vehicle is driven under human driver control, the vehicle-to-vehicle subsystem determines the human driving trajectory and the algorithm-predicted trajectory output by the autonomous driving algorithm within the next five seconds, and calculates the lateral offset distance between the two at each corresponding moment. When the average lateral offset distance exceeds a preset lateral offset threshold, and this state persists for more than a preset time length, the vehicle-to-vehicle subsystem determines that there is a trajectory difference near that moment and generates a preliminary difference event. This preliminary difference event may include the event trigger time, event trigger location, trigger indicator type, trajectory offset value, and the corresponding human driving trajectory segment and algorithm-predicted trajectory segment.

[0067] In another specific example, when there is a slow-moving vehicle or a pedestrian who appears to be obscured ahead, the human driver slows down in advance and makes slight lateral adjustments to the driving trajectory. However, the autonomous driving algorithm, in shadow mode, outputs a trajectory that maintains the current lane and current speed. Even if the geometric shift between the two trajectories is not significant in a short period, the vehicle-to-vehicle subsystem can identify a preliminary discrepancy event based on the clear difference between the deceleration behavior corresponding to the human driving trajectory and the speed-maintaining behavior corresponding to the algorithm-predicted trajectory. This event can then be uploaded to a cloud-based analysis platform, where a cloud-based visual language model can further determine whether the discrepancy is valid.

[0068] Here, after identifying the initial difference event, the vehicle terminal system can generate an event identifier for the initial difference event and record the trigger information related to the event. The trigger information may include at least one of the following: trigger time, trigger location, trigger threshold, trajectory difference index, vehicle speed, road type, algorithm version, human driving trajectory summary, algorithm predicted trajectory summary, and trigger cause type.

[0069] S104. Based on the preliminary difference event, extract the event data corresponding to the preliminary difference event from the vehicle-side driving association data, and upload the event data to the cloud analysis platform.

[0070] In practical implementation, after identifying a preliminary discrepancy event, the vehicle terminal system extracts the corresponding event data from the continuously collected and cached vehicle-side driving-related data based on the event, and uploads the event data to the cloud analysis platform. The event data comprehensively characterizes the vehicle's environment, operating status, human driving behavior, and autonomous driving algorithm output before and after the preliminary discrepancy event, enabling the cloud analysis platform to confirm the validity of the preliminary discrepancy event and analyze the causes of the discrepancy based on a more complete context.

[0071] Specifically, when a preliminary difference event is identified, the vehicle-side terminal system can record the event trigger time, event trigger location, trigger indicator type, and trajectory difference indicator value corresponding to that event. The event trigger time can be the moment when the trajectory difference indicator first meets the preset difference trigger conditions, the moment when the trajectory difference indicator reaches its peak, or the moment when the trigger is confirmed after the trajectory difference has continuously met the preset conditions. The vehicle-side terminal system can use the event trigger time as a benchmark to determine the corresponding event time window and extract the vehicle-side driving-related data within that event time window from the vehicle-side cache.

[0072] In practice, an event time window can include a forward time period before the event trigger time and a backward time period after the event trigger time. For example, data from several seconds before and several seconds after the event trigger time can be extracted to form event data covering the generation, development, and termination of the difference. The forward and backward time periods can be configured based on vehicle speed, road type, difference type, or data upload strategy.

[0073] In high-speed driving scenarios, the forward time period can be appropriately extended to retain scenario information on how drivers can avoid distant risks in advance; in low-speed urban scenarios, the backward time period can be appropriately extended to record the impact of different events on subsequent driving behavior.

[0074] Event data can include at least environmental perception data, vehicle state data, human driving data, and algorithm decision data within the event time window. Environmental perception data can include video streams, image frame sequences, point cloud data, radar target lists, or obstacle detection data, used to characterize the state of the road structure, traffic participants, obstacles, traffic signs and markings, and potential risk targets around the vehicle before and after the event. Vehicle state data can include vehicle speed, acceleration, yaw rate, heading angle, positioning information, and vehicle attitude information, used to characterize the vehicle's motion state before and after the event. Human driving data can include steering wheel angle, accelerator pedal opening, brake pedal pressure, turn signal status, and gear shifting status, used to characterize the actual operational behavior of the human driver during the event. Algorithm decision data can include the perception results, prediction results, planned trajectory, target speed, target acceleration, and driving decision commands output by the autonomous driving algorithm, used to characterize the background decision results of the autonomous driving algorithm in the same scenario.

[0075] Furthermore, event data can also include human driving trajectories and algorithm-predicted trajectories corresponding to the initial discrepancies. The vehicle-to-vehicle (V2V) subsystem can correlate human driving trajectory segments and algorithm-predicted trajectory segments within the event time window with environmental perception data, enabling the cloud-based analytics platform to overlay and display both types of trajectories in video frames or scene maps. Trajectory data can include trajectory point coordinates, trajectory point timestamps, trajectory point velocities, trajectory point accelerations, trajectory point heading angles, trajectory curvature, and trajectory source identifiers.

[0076] In one alternative implementation, the vehicle terminal system can structurally encapsulate the event data to generate an event data packet. The event data packet may include event metadata, environmental perception data segments, vehicle status data segments, human driving data segments, algorithm decision data segments, trajectory data segments, and verification information segments.

[0077] The event metadata may include event identifier, vehicle identifier, algorithm version, trigger time, trigger location, road type, weather information, trigger indicator type, and trigger indicator value; the verification information segment may include data integrity verification value, data length information, data version information, and encryption identifier information.

[0078] Furthermore, before uploading to the cloud analytics platform, the vehicle terminal system can preprocess the event data. Preprocessing may include at least one of the following: data compression, data anonymization, data filtering, data quality verification, and data classification.

[0079] For example, the vehicle terminal system can encode and compress video stream data to reduce the amount of data uploaded; it can de-identify vehicle identification, personnel information, license plate information, or sensitive geographical information to meet data security requirements; it can mark the quality of event data with serious frame loss, abnormal timestamps, or missing key fields; and it can set the upload priority of event data according to the degree of trajectory difference, potential risk level, or road scene complexity.

[0080] During the upload process, the vehicle-mounted subsystem can upload event data to the cloud analysis platform via the onboard communication module. The onboard communication module supports cellular networks, vehicle-to-everything (V2X) networks, wireless LANs, or other available vehicle-to-cloud communication links. Under good network conditions, the vehicle-mounted subsystem can upload event data in real-time or near real-time. When network conditions are unfavorable, the subsystem can temporarily store the event data in local storage and resume uploads or perform batch uploads once the network is restored. This avoids data loss due to network fluctuations and ensures the integrity of the returned differential samples.

[0081] Furthermore, the vehicle-to-vehicle terminal system can also determine whether to upload event data based on the upload strategy. The upload strategy may include at least one of the following: event difference level, event risk level, data integrity, vehicle's current network status, cloud platform load status, vehicle storage space status, and preset sampling ratio.

[0082] For example, for preliminary discrepancy events with a high degree of trajectory difference that involve braking and avoidance or potential collision risks, a higher upload priority can be set; for events with a low degree of difference or suspected to be caused by arbitrary human operation, local de-priority caching or delayed upload can be implemented.

[0083] In one specific example, when the vehicle-to-vehicle (V2V) subsystem detects that the lateral deviation between the human driving trajectory and the algorithm-predicted trajectory continuously exceeds a preset threshold, it determines that a preliminary discrepancy event has occurred. Centering on the moment this discrepancy triggers, it extracts video streams, vehicle speed, vehicle acceleration, GPS coordinates, steering wheel angle, accelerator pedal opening, brake pedal pressure, the planned trajectory point set output by the autonomous driving algorithm, and fragments of the human driving trajectory from the local rolling buffer for the five seconds before and after the event. Subsequently, the V2V subsystem associates and encapsulates the above data with event trigger information, vehicle identification, algorithm version, and timestamp information, compresses it, and uploads it to the cloud analysis platform via the vehicle network.

[0084] In another specific example, when the vehicle terminal system determines that the human driver has slowed down in advance to avoid a collision, and the autonomous driving algorithm still outputs the algorithm's predicted trajectory of maintaining speed in shadow mode, the vehicle terminal system can mark the event as a preliminary difference event of suspected risk avoidance, and prioritize extracting data such as front camera video, target detection results, brake pedal pressure changes, vehicle longitudinal acceleration changes, algorithm target speed and planned trajectory, and upload them to the cloud analysis platform.

[0085] In this way, the cloud-based visual language model can combine the visual scene and changes in driving behavior before and after the event to further determine whether the human driver's deceleration is due to an obstructed target in front, abnormal behavior of traffic participants, or other effective risk factors.

[0086] As one possible implementation, before uploading event data to the cloud analysis platform, it can be determined whether the preliminary difference event meets the upload conditions. The upload conditions include: the degree of trajectory deviation reaches a preset upload threshold, the event location belongs to the target road area, the integrity of the event data meets the preset integrity requirements, the network status meets the upload requirements, and the event type does not belong to at least one of the preset filtering types. When the upload conditions are met, the event data is uploaded to the cloud analysis platform. When the upload conditions are not met, the event data is locally cached, uploaded with a delay, or discarded.

[0087] S105. The cloud-based analysis platform constructs a multimodal analysis input for the visual language model based on the event data, and uses the cloud-based visual language model to confirm the validity of the preliminary difference event and analyze the reasons for the difference, thereby obtaining the difference confirmation result.

[0088] In practice, after receiving event data uploaded by the vehicle subsystem, the cloud-based analytics platform constructs a multimodal analysis input for a visual language model based on the event data. The cloud-based visual language model then verifies the validity of the initial discrepancies and analyzes the causes of these discrepancies, yielding a discrepancy confirmation result. This result characterizes whether the initial discrepancy event is a valid discrepancy with value for optimizing autonomous driving algorithms and further explains the contextual, driving behavior, or algorithmic defects that caused the discrepancy.

[0089] Specifically, the cloud-based analytics platform can first parse and verify the received event data. Event data can include event metadata, environmental perception data, vehicle status data, human driving data, algorithm decision data, human driving trajectories, and algorithm-predicted trajectories. The cloud-based analytics platform can perform data integrity and time consistency checks on the event data based on event identifiers, timestamp information, data version information, and integrity verification information. After successful verification, different types of data are aligned according to event time windows to obtain unified scene data corresponding to the initially differing events. For cases with minor frame gaps, inconsistent sampling intervals, or inconsistent trajectory point numbers, the cloud-based analytics platform can preprocess the data through interpolation, resampling, keyframe extraction, or data completion to ensure the integrity and consistency of subsequent multimodal analysis inputs.

[0090] In practical implementation, the cloud-based analytics platform can extract environmental image information, trajectory information, vehicle status information, and driving behavior information from event data. Environmental image information can include video frame sequences, key image frames, bird's-eye view scene maps, or target detection results within the event time window; trajectory information can include human driving trajectories, algorithm-predicted trajectories, and the trajectory difference indicators between the two; vehicle status information can include vehicle speed, acceleration, yaw rate, heading angle, and positioning information; driving behavior information can include changes in steering wheel angle, accelerator pedal position, brake pedal position, turn signal status, and changes in driver control behavior.

[0091] Furthermore, the cloud-based analytics platform can construct multimodal analysis inputs suitable for visual language model processing based on event data. These multimodal analysis inputs can include image inputs, trajectory inputs, and text prompt inputs. The image input can be key video frames within the event time window, continuous video clips, or scene sequences synthesized from multiple images; the trajectory input can be human driving trajectories and algorithm-predicted trajectories superimposed on image frames, bird's-eye view maps, or road coordinate maps; and the text prompt input can include the event occurrence time, vehicle speed, triggering reason, trajectory deviation indicators, a summary of human driving behavior, a summary of algorithm decision, and a pre-defined analysis task description.

[0092] In one alternative implementation, the cloud-based analytics platform can overlay human driving trajectories and algorithm-predicted trajectories onto corresponding video frames or scene images using different labeling methods. For example, the human driving trajectory can be labeled as the first trajectory, the algorithm-predicted trajectory as the second trajectory, and the trajectory direction, key moment locations, locations of discrepancies, and target vehicle locations can be annotated in the image.

[0093] Furthermore, when constructing the text prompt input section, the cloud-based analytics platform can generate analysis prompts according to a preset template. These prompts can be used to instruct the cloud-based visual language model to perform specific driving difference confirmation tasks. For example, the prompts might include: Based on the video scene, vehicle status information, human driving trajectory, and algorithm-predicted trajectory within the current event time window, determine whether the significant inconsistency between the two trajectories near the time of the difference's occurrence constitutes a valid difference that the autonomous driving algorithm needs to learn; if it is a valid difference, explain the main reason for the difference and provide the corresponding scene label; if it is not a valid difference, explain why it is an invalid difference.

[0094] In another alternative implementation, the cloud analytics platform can also reconstruct cloud trajectories based on event data and use the reconstructed cloud reference trajectory as part of the multimodal analysis input. Specifically, the cloud analytics platform can utilize a trajectory prediction model with higher computing power in the cloud, combined with historical environmental perception data within the event time window, the movement status of traffic participants, road topology information, and vehicle status data, to generate a cloud reference trajectory corresponding to the current scenario. The cloud reference trajectory can serve as an auxiliary judgment criterion, used to compare with human driving trajectories and algorithm-predicted trajectories, thereby determining whether the algorithm-predicted trajectory significantly deviates from a more reasonable driving path, or whether the human driving trajectory is merely random manual operation.

[0095] When reasoning using a cloud-based visual language model, the model can jointly understand environmental images, trajectory overlay information, and textual prompts within an event time window. Specifically, the cloud-based visual language model can identify scene elements such as road structure, lane lines, traffic signals, vehicles ahead, vehicles to the side, pedestrians, non-motorized vehicles, obstacles, and occluded areas. By combining the differences between human driving trajectories and algorithm-predicted trajectories, it can analyze whether the human driver made a reasonable operation based on specific risk objectives or traffic rules.

[0096] At the same time, the cloud-based visual language model can also combine vehicle speed, braking changes, steering changes and algorithm output trajectory to determine whether the algorithm has problems such as perception omissions, insufficient predictions, planning deviations, insufficient risk assessments or unreasonable behavioral decisions.

[0097] In one specific implementation, if the cloud-based visual language model analysis detects that the human driving trajectory exhibits premature deceleration, avoidance, or lane-changing behavior compared to the algorithm-predicted trajectory, and the environmental image contains obscured pedestrians, sudden deceleration of the vehicle in front, adjacent vehicles cutting in, road construction, static obstacles, or other potential risk targets, then the preliminary difference event can be determined as a valid difference. In this case, the difference confirmation result may include a valid difference identifier, root cause analysis information in natural language form, and corresponding scene labels.

[0098] For example, root cause analysis information can be characterized as follows: human drivers slow down in advance to avoid obscured targets or abnormal traffic participants, while autonomous driving algorithms do not fully recognize this risk, and therefore this difference has value for algorithm optimization.

[0099] In another specific implementation, if the cloud-based visual language model analysis finds that although there is a geometric deviation between the human driving trajectory and the algorithm-predicted trajectory, there are no obvious risk targets or traffic rule constraints in the environmental image, and the human driving data shows short-term arbitrary turning, unnecessary braking, unplanned lane changes, or other operations unrelated to road risks, then the preliminary difference event can be determined as an invalid difference.

[0100] At this point, the difference confirmation result can include an invalid difference identifier and an explanation of the invalid reason. For example, the difference may be caused by the driver's personal habits, short-term operation fluctuations, distracted operation, or unnecessary human intervention, and should not be used as a high-value sample for training or optimizing autonomous driving algorithms.

[0101] Furthermore, the discrepancy confirmation results can be output in a structured format so that the cloud analytics platform can automatically store and distribute them. The discrepancy confirmation results may include at least one of the following: confirmation result field, root cause analysis field, scene label field, algorithm defect type field, confidence level field, and suggested handling field.

[0102] The confirmation result field indicates whether the preliminary difference event is a valid difference, an invalid difference, or a difference requiring manual review; the root cause analysis field describes the cause of the difference in natural language; the scene label field can include labels such as target occlusion, defensive driving, pedestrian crossing, vehicle cutting in, road construction, remote risk, perception miss, insufficient prediction, and planning deviation; the algorithm defect type field can indicate whether the difference is more likely to correspond to a problem in the perception module, prediction module, planning module, or control module; the confidence field indicates the degree of trust of the visual language model in the confirmation result; and the suggested processing field indicates whether to include it in the training set, whether to include it in the manual review queue, or whether to discard it.

[0103] In one alternative implementation, to improve the reliability of the difference confirmation results, the cloud-based analysis platform can also set a result verification strategy. The result verification strategy may include at least one of the following: model confidence threshold judgment, cross-judgment of multiple visual language models, rule verification, manual sampling, or comparison with historical similar cases.

[0104] For example, when the confidence level of the cloud-based visual language model output is lower than the preset confidence threshold, the preliminary difference event can be marked as awaiting manual review; when multiple models output consistent and valid difference conclusions for the same event, the credibility level of the event can be improved; when the root cause analysis output by the visual language model is significantly contradictory to the trajectory difference index or vehicle status change, a secondary analysis can be triggered.

[0105] In one specific example, the event data uploaded by the vehicle shows that near the moment the event was triggered, the human driving trajectory slightly shifted to the left and was accompanied by braking and deceleration, while the algorithm-predicted trajectory remained in the original lane and continued straight. The cloud analytics platform inputs the forward-looking camera video frames, the human driving trajectory, the algorithm-predicted trajectory, changes in vehicle speed, and changes in brake pedal pressure into the cloud-based visual language model.

[0106] A cloud-based visual language model identified a pedestrian jutting out from the right side of a bus ahead and potentially crossing the road. This event was then identified as a valid difference, and root cause analysis information and scene labels such as target occlusion, pedestrian crossing, and defensive driving were generated. This result suggests that autonomous driving algorithms may have insufficient ability to identify the risk of occluded pedestrians.

[0107] In another specific example, the event data uploaded by the vehicle shows that the human driving trajectory deviated slightly laterally for a short period of time, while the algorithm predicted that the trajectory would remain centered in the lane. The cloud-based visual language model, based on video frames and vehicle status information, determines that in this scenario, there are no obstacles ahead, no pedestrians crossing, no adjacent vehicles cutting in, and no abnormal lane markings or traffic rule constraints. Simultaneously, the human driving data only shows a short-term, slight steering wheel movement. In this case, the cloud-based visual language model can output this event as an invalid discrepancy and label the invalidity as a trajectory deviation caused by short-term driver operation fluctuations or driving habits, thus preventing such low-value data from entering the algorithm's training set.

[0108] S106. Based on the difference confirmation results, the event data is filtered and labeled to obtain difference sample data for autonomous driving algorithm optimization.

[0109] In practice, after obtaining the discrepancy confirmation results, the cloud-based analytics platform filters and labels the event data based on these results to obtain discrepancy sample data for optimizing autonomous driving algorithms. This discrepancy sample data serves as input data for training, testing, simulation playback, defect localization, or model iteration of autonomous driving algorithms, enabling the algorithms to optimize for problems encountered in real-world road scenarios.

[0110] Specifically, the difference confirmation result can include at least one of the following: confirmation result field, root cause analysis field, scene label field, algorithm defect type field, confidence level field, and suggested processing field. The cloud analytics platform can determine whether the event data has algorithm optimization value based on the confirmation result field. When the confirmation result field indicates that the initial difference event is a valid difference, the cloud analytics platform identifies the event data as candidate difference sample data; when the confirmation result field indicates that the initial difference event is an invalid difference, the cloud analytics platform can filter, downweight, or classify the event data into an invalid difference database; when the confirmation result field indicates that the initial difference event is a difference awaiting review, the cloud analytics platform can transfer the event data to a manual review queue or a secondary model analysis process.

[0111] In practice, for event data that is confirmed as having a valid difference, the cloud analytics platform can retain the corresponding environmental perception data, vehicle status data, human driving data, algorithm decision data, human driving trajectory, algorithm predicted trajectory, as well as the cloud reference trajectory, trajectory difference index, and scene understanding results generated during the cloud analytics process.

[0112] Furthermore, the cloud-based analytics platform can add natural language descriptions to the event data based on the root cause analysis field in the discrepancy confirmation results. These descriptions can be used to explain the main reasons for the differences between human driving trajectories and algorithm-predicted trajectories. For example, a human driver might brake in advance to avoid an obstructed pedestrian, while the algorithm-predicted trajectory does not reflect this risk avoidance; or a human driver might change lanes in advance to avoid a neighboring vehicle cutting in, while the algorithm-predicted trajectory remains in the original lane.

[0113] Furthermore, the cloud-based analytics platform can add structured scene tags to event data based on the scene tag field in the difference confirmation results. Structured scene tags can include at least one of the following: target occlusion, pedestrian crossing, non-motorized vehicle interference, adjacent vehicle cutting in, sudden braking of the vehicle in front, road construction, lane line abnormalities, potential risks at a distance, defensive driving, missed detection, insufficient prediction, planning deviation, unreasonable braking timing, and unreasonable lane change decision.

[0114] In one alternative implementation, the cloud-based analytics platform can further categorize discrepancy sample data into different algorithm optimization categories based on an algorithm defect type field. For example, when the discrepancy confirmation result characterization algorithm fails to identify occluded pedestrians or obstacles, the corresponding discrepancy sample data can be labeled as a perception miss sample; when the discrepancy confirmation result characterization algorithm fails to accurately predict the movement trends of surrounding vehicles or pedestrians, the corresponding discrepancy sample data can be labeled as an underprediction sample; and when the discrepancy confirmation result characterization algorithm has detected a risk target but the planned trajectory fails to make reasonable avoidance or deceleration, the corresponding discrepancy sample data can be labeled as a planning decision deviation sample.

[0115] Furthermore, the cloud-based analytics platform can use a confidence level field to manage differential sample data in a tiered manner. Specifically, when the confidence level of the differential confirmation result is higher than the first preset confidence threshold, the corresponding event data can be directly stored in the optimization database as high-confidence differential sample data; when the confidence level of the differential confirmation result is between the first and second preset confidence thresholds, the corresponding event data can be used as differential sample data to be reviewed and assigned to manual annotation or expert review processes; when the confidence level of the differential confirmation result is lower than the second preset confidence threshold, the corresponding event data can be downweighted or temporarily excluded from the training set.

[0116] During the actual storage process, the cloud-based analytics platform can store the discrepancy sample data into an optimization database. This optimization database can be indexed using fields such as event identifier, vehicle identifier, algorithm version, road type, scene label, defect type, occurrence time, occurrence location, and confidence level.

[0117] In one alternative implementation, the cloud analytics platform is not required to directly delete event data deemed invalid. Instead, it can label the invalidity as a reason and store it with low priority, depending on actual needs. For example, invalid differences can be labeled as driver random operation, driver distraction, unnecessary manual lane changes, short-term steering wheel vibration, normal road disturbance, sensor malfunction, or incomplete data.

[0118] Furthermore, the cloud-based analytics platform can iteratively optimize vehicle-side triggering strategies or cloud-based filtering strategies based on the statistical results between valid and invalid difference samples. For example, when the proportion of invalid differences corresponding to a certain type of vehicle-side triggering condition is high, the threshold corresponding to that triggering condition can be increased or auxiliary judgment conditions can be added; when the proportion of valid differences in a certain type of long-tail scenario is high but the number of triggers is low, the triggering threshold for the corresponding scenario can be lowered or the upload priority can be increased.

[0119] See Figure 2 The diagram shows a flowchart of another vehicle-side shadow mode driving difference confirmation method provided in this embodiment of the present disclosure. The method includes steps S201-S203 and is applied to, for example... Figure 1 After step S106, where: S201. Based on the scene labels and root cause analysis information corresponding to the differential sample data, determine the corresponding algorithm defect type.

[0120] S202. Based on the algorithm defect type, the differential sample data is distributed to the corresponding perception model training process, prediction model training process, planning model optimization process, or simulation playback verification process.

[0121] S203. After completing the algorithm optimization, the optimized autonomous driving algorithm is validated by regression analysis using the scene data corresponding to the difference sample data.

[0122] Specifically, the scene labels may include at least one of the following: target occlusion, pedestrian crossing, non-motorized vehicle interference, adjacent vehicle cutting in, sudden braking of the vehicle in front, road construction, abnormal lane markings, potential risks at a distance, defensive driving, complex intersections, unprotected left turns, merging and merging, and congested following. The root cause analysis information may include natural language descriptions of causes generated by a cloud-based visual language model, or it may include causal elements obtained through structured parsing, such as the type of risk target, the location of the risk target, the time of risk occurrence, the driving behavior taken by the human driver, the abnormal decision results output by the autonomous driving algorithm, and the differences between the human driving trajectory and the algorithm's predicted trajectory. Based on the above scene labels and root cause analysis information, the cloud analysis platform can determine the main source of the algorithmic problem reflected in the discrepancy sample data.

[0123] In practical implementation, the cloud-based analytics platform can pre-establish a mapping relationship between scene tags, root cause analysis elements, and algorithm defect types. These algorithm defect types can include at least one of the following: perception defects, prediction defects, planning defects, control defects, decision-making strategy defects, map positioning defects, and data quality defects.

[0124] For example, when the scene labels include target occlusion, pedestrian crossing, and obstacle missed detection, and the root cause analysis information indicates that the algorithm failed to identify potential risk targets ahead, the corresponding algorithm defect type can be identified as a perception defect; when the scene labels include adjacent vehicles cutting in, non-motorized vehicles crossing, and vehicles braking suddenly in front, and the root cause analysis information indicates that the algorithm failed to accurately predict the movement trends of traffic participants, the corresponding algorithm defect type can be identified as a prediction defect; when the root cause analysis information indicates that the algorithm has perceived risk targets but still generates unreasonable driving paths, braking timings, or lane-changing strategies, the corresponding algorithm defect type can be identified as a planning defect or a decision-making strategy defect.

[0125] Furthermore, the cloud-based analytics platform can identify one or more algorithm defect types for the same discrepancy sample data. When a discrepancy sample data involves both perception misses and insufficient planning and avoidance, it can be simultaneously labeled as a perception defect and a planning defect, with primary and secondary defect types assigned. For example, in a scenario where a pedestrian is obstructed ahead, if the autonomous driving algorithm fails to detect the pedestrian, the primary defect type can be a perception defect; if the algorithm detects the pedestrian but fails to slow down or change lanes to avoid them in advance, the primary defect type can be a planning defect.

[0126] After determining the type of algorithm defect, the cloud-based analysis platform can distribute the differential sample data to the corresponding algorithm optimization process based on the defect type. Specifically, when the algorithm defect type is a perception defect, the corresponding differential sample data can be distributed to the perception model training process. The perception model training process can utilize video frames, image frames, point cloud data, target detection annotation information, occlusion area information, and scene labels from the differential sample data to train or fine-tune target detection models, semantic segmentation models, instance segmentation models, lane line detection models, or multi-sensor fusion perception models, thereby improving the autonomous driving algorithm's ability to identify risky targets, occluded targets, weak targets, or complex road structures.

[0127] When the algorithm defect type is a prediction defect, the cloud-based analysis platform can distribute the corresponding discrepancy sample data to the prediction model training process. This training process can utilize the traffic participant trajectories, target historical states, road topology relationships, vehicle interaction relationships, human driving behavior, and scene labels from the discrepancy sample data to train or validate traffic participant behavior prediction models, intent recognition models, trajectory prediction models, or interaction prediction models. For example, for scenarios such as adjacent vehicles cutting in, pedestrians crossing, and non-motorized vehicles crossing, the ability of the prediction model to model the potential movement intentions and interactive behaviors of traffic participants can be enhanced based on the discrepancy sample data.

[0128] When the algorithm defect type is a planning defect or a decision-making strategy defect, the cloud analysis platform can distribute the corresponding difference sample data to the planning model optimization process. The planning model optimization process can utilize the human driving trajectory, algorithm predicted trajectory, cloud reference trajectory, vehicle status data, scene labels, and root cause analysis information in the difference sample data to optimize the path planning strategy, speed planning strategy, risk cost function, behavioral decision rules, avoidance strategy, or defensive driving strategy.

[0129] For example, in scenarios where a human driver brakes in advance but the algorithm fails to decelerate in time, the difference sample data can be used to adjust the risk assessment weights, braking trigger conditions, or safe distance strategies in the planning model; in scenarios where a human driver changes lanes in advance to avoid a collision but the algorithm stays in the original lane, the planning model's path selection strategy for low-speed targets ahead, construction areas, or potential obstruction risks can be optimized.

[0130] In one alternative implementation, the cloud-based analytics platform can also distribute some of the discrepancy sample data to the simulation playback verification process. The simulation playback verification process can reconstruct a real road scene based on the scene data corresponding to the discrepancy sample data, and reproduce the states of traffic participants, vehicle states, road structure, and environmental perception inputs before and after the event in the simulation environment.

[0131] Furthermore, when distributing differential sample data to different optimization processes, the cloud-based analytics platform can determine the distribution strategy based on sample priority, risk level, confidence level, and scenario scarcity. For example, differential sample data involving potential collision risks, pedestrian crossings, occluded targets, or abnormal behaviors on highways can be given a higher priority and prioritized for training and simulation verification processes. Differential sample data with lower confidence levels but potentially involving long-tail scenarios can first undergo manual review, and then the decision on whether to proceed to the training process can be based on the review results. For a large number of existing common scenario samples, they can be included in the training set according to the sampling ratio to avoid imbalanced training data distribution.

[0132] After algorithm optimization, the optimized autonomous driving algorithm can be validated using the scene data corresponding to the differential sample data. Specifically, the original environmental perception data, vehicle state data, traffic participant state data, road structure data, and event time window information from the differential sample data can be input into the optimized autonomous driving algorithm. This allows the optimized algorithm to re-output perception results, prediction results, planned trajectories, or decision results under scene conditions that are the same as or similar to the original events. Subsequently, the optimized algorithm output can be compared with the original algorithm output, the human driving trajectory, and the cloud reference trajectory to determine whether the algorithm's shortcomings have been improved.

[0133] In one specific implementation, regression validation can include at least one of perception result validation, prediction result validation, and planning result validation. For perception defect samples, it can be verified whether the optimized perception model can correctly detect pedestrians, vehicles, non-motorized vehicles, obstacles, or lane lines that were missed in the original scene; for prediction defect samples, it can be verified whether the optimized prediction model can more accurately predict the movement direction, speed changes, or lane-cutting intentions of traffic participants; for planning defect samples, it can be verified whether the optimized planning model can generate driving paths and speed curves that are closer to reasonable human driving trajectories or cloud reference trajectories.

[0134] Furthermore, the cloud-based analytics platform can generate sample closed-loop states based on regression validation results. These sample closed-loop states can include at least one of the following: unprocessed, distributed, training, pending validation, validated, validated but not passed, and requiring manual review. When the optimized autonomous driving algorithm outputs results in the corresponding scenario data that meet preset validation conditions, the discrepancy sample data can be marked as validated successfully; when the optimized autonomous driving algorithm still has the same or similar defects, the discrepancy sample data can be marked as validated but not passed and redistributed to the corresponding algorithm optimization process.

[0135] This disclosure provides a method and apparatus for confirming differences in vehicle-side shadow mode driving. It receives multimodal data such as video, trajectory, and vehicle status uploaded from the vehicle via a cloud-based visual language model. It performs secondary confirmation on initial difference events and outputs valid difference judgments, difference causes, and scene labels, thereby filtering out invalid differences and retaining high-quality samples with genuine algorithmic optimization value. This allows the vehicle to perform only lightweight initial difference detection and data uploading, while the complex multimodal understanding, long-term time-series analysis, validity confirmation, and root cause analysis are handled by the cloud. This reduces the computational burden on the vehicle, improves the accuracy of difference data collection, enables automated explanation of difference causes and scene labeling, and enhances the iteration efficiency and reliability of autonomous driving algorithms for complex and long-tail scenarios.

[0136] Those skilled in the art will understand that, in the above-described method of the specific implementation, the order in which each step is written does not imply a strict execution order and does not constitute any limitation on the implementation process. The specific execution order of each step should be determined by its function and possible internal logic.

[0137] Based on the same inventive concept, this disclosure also provides a vehicle-side shadow mode driving difference confirmation device corresponding to the vehicle-side shadow mode driving difference confirmation method. Since the principle of the device in this disclosure for solving the problem is similar to the above-mentioned vehicle-side shadow mode driving difference confirmation method in this disclosure, the implementation of the device can refer to the implementation of the method, and the repeated parts will not be described again.

[0138] Please see Figure 3 , Figure 3 This is a schematic diagram of a vehicle-side shadow mode driving difference confirmation device provided in an embodiment of this disclosure. Figure 3 As shown in the figure, the vehicle-side shadow mode driving difference confirmation device 300 provided in this embodiment includes: The data acquisition module 310 is used to acquire vehicle-side driving-related data during vehicle operation. The vehicle-side driving-related data includes at least environmental perception data, vehicle status data, human driving data, and algorithm decision data output by the autonomous driving algorithm.

[0139] The shadow mode processing module 320 is used to determine the human driving trajectory and the algorithm predicted trajectory based on the human driving data and the algorithm decision data, respectively.

[0140] The difference judgment module 330 is used to compare the human driving trajectory and the algorithm-predicted trajectory, and determine the corresponding preliminary difference event when the preset difference triggering conditions are met.

[0141] The vehicle-side communication module 340 is used to extract event data corresponding to the preliminary difference event from the vehicle-side driving association data based on the preliminary difference event, and upload the event data to the cloud analysis platform.

[0142] The cloud analysis module 350 is used to perform multimodal analysis input for constructing a visual language model based on the event data by the cloud analysis platform, and to perform validity confirmation and cause analysis of the preliminary difference event through the cloud visual language model to obtain the difference confirmation result.

[0143] The result generation module 360 ​​is used to filter and label the event data based on the difference confirmation results to obtain difference sample data for autonomous driving algorithm optimization.

[0144] The processing flow of each module in the device and the interaction flow between each module can be referred to the relevant descriptions in the above method embodiments, and will not be detailed here.

[0145] Corresponding to Figure 1 and Figure 2 In the method for confirming differences in vehicle-side shadow mode driving, this disclosure also provides an electronic device 400, such as... Figure 4 The diagram shown is a structural schematic of an electronic device 400 provided in an embodiment of this disclosure, including: Processor 41, memory 42, and bus 43; memory 42 is used to store execution instructions, including main memory 421 and external memory 422; the main memory 421, also called internal memory, is used to temporarily store the computational data in processor 41, as well as the data exchanged with external memory 422 such as hard disk. Processor 41 exchanges data with external memory 422 through main memory 421. When the electronic device 400 is running, processor 41 and memory 42 communicate through bus 43, enabling processor 41 to execute... Figure 1 and Figure 2 The steps of the vehicle-side shadow mode driving difference confirmation method.

[0146] This disclosure also provides a computer-readable storage medium storing a computer program. When executed by a processor, the computer program performs the steps of the vehicle-side shadow mode driving difference confirmation method described in the above method embodiments. The storage medium can be a volatile or non-volatile computer-readable storage medium.

[0147] This disclosure also provides a computer program product, which includes computer instructions. When the computer instructions are executed by a processor, they can perform the steps of the vehicle-side shadow mode driving difference confirmation method described in the above method embodiments. For details, please refer to the above method embodiments, which will not be repeated here.

[0148] The aforementioned computer program product can be implemented through hardware, software, or a combination thereof. In one optional embodiment, the computer program product is specifically embodied in a computer storage medium; in another optional embodiment, the computer program product is specifically embodied in a software product, such as a software development kit (SDK), etc.

[0149] Finally, it should be noted that the above-described embodiments are merely specific implementations of this disclosure, used to illustrate the technical solutions of this disclosure, and not to limit it. The protection scope of this disclosure is not limited thereto. Although this disclosure has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art can still modify or easily conceive of changes to the technical solutions described in the foregoing embodiments, or make equivalent substitutions for some of the technical features, within the scope of the technology disclosed in this disclosure. Such modifications, changes, or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this disclosure, and should all be covered within the protection scope of this disclosure. Therefore, the protection scope of this disclosure should be determined by the protection scope of the claims.

Claims

1. A method for confirming differences in vehicle-side shadow mode driving, characterized in that, include: Acquire vehicle-side driving-related data during vehicle operation, including at least environmental perception data, vehicle status data, human driving data, and algorithm decision data output by autonomous driving algorithms. Based on the human driving data and the algorithm decision data, the human driving trajectory and the algorithm predicted trajectory are determined respectively; The human driving trajectory and the algorithm-predicted trajectory are compared for differences. When a preset difference trigger condition is met, the corresponding preliminary difference event is determined. Based on the preliminary difference event, event data corresponding to the preliminary difference event is extracted from the vehicle-side driving association data, and the event data is uploaded to the cloud analysis platform; The cloud-based analysis platform constructs a visual language model based on the event data as multimodal analysis input, and uses the cloud-based visual language model to confirm the validity of the preliminary difference events and analyze the reasons for the differences, thereby obtaining the difference confirmation results; Based on the difference confirmation results, the event data is filtered and labeled to obtain difference sample data for autonomous driving algorithm optimization.

2. The method according to claim 1, characterized in that, Based on the human driving data and the algorithm decision data, the human driving trajectory and the algorithm-predicted trajectory are determined respectively, specifically including: Based on the human driving data and vehicle dynamics model, the human driving trajectory of the vehicle in the target prediction time domain is determined. Based on the algorithm decision data, the algorithm prediction trajectory output by the autonomous driving algorithm in the same target prediction time domain is obtained; The human driving trajectory and the algorithm-predicted trajectory are converted to the same time reference and the same vehicle coordinate system or road coordinate system to obtain comparable trajectory data.

3. The method according to claim 1, characterized in that, The human driving trajectory and the algorithm-predicted trajectory are compared for differences. When a preset difference trigger condition is met, the corresponding preliminary difference event is determined, specifically including: Calculate the trajectory deviation index between the human driving trajectory and the algorithm-predicted trajectory; The trajectory deviation index is compared with a preset difference threshold. When the trajectory deviation index exceeds the preset difference threshold, or when the trajectory deviation index continuously meets the deviation condition within a preset duration, the initial difference event is determined to have occurred. The trajectory deviation index includes at least one of the following: average trajectory distance, maximum lateral offset distance, trajectory endpoint deviation, trajectory curvature difference, and trajectory Hausdorff distance.

4. The method according to claim 1, characterized in that, Based on the initial difference event, event data corresponding to the initial difference event is extracted from the vehicle-side driving association data, and the event data is uploaded to the cloud analysis platform, specifically including: Determine the trigger time corresponding to the initial difference event; Based on the triggering time, the event time window is determined according to the preset forward time length and the preset backward time length; The environmental perception data, vehicle status data, human driving data, algorithm decision data, human driving trajectory, and algorithm predicted trajectory are extracted within the event time window. The captured data is associated with and encapsulated with event identifiers, timestamp information, vehicle location information, and difference trigger information to obtain an event data packet; After the event data packets are compressed and de-identified, they are uploaded to the cloud analysis platform.

5. The method according to claim 1, characterized in that, The multimodal analysis input for constructing a visual language model based on the event data by the cloud-based analytics platform specifically includes: Extract video frame sequences and trajectory data from the event data; The human driving trajectory, the algorithm-predicted trajectory, and the cloud reference trajectory are mapped to the video frame sequence or the corresponding bird's-eye view scene map to form scene input data with trajectory labels; Based on the scene input data, trajectory difference information, and preset analysis task text, a multimodal analysis input for the visual language model is constructed; The preset analysis task text is used to instruct the visual language model to determine whether the preliminary difference event is a valid difference that the autonomous driving algorithm needs to learn, and to output the reason for the difference and the scene label.

6. The method according to claim 1, characterized in that, The validity of the preliminary discrepancy events and the causes of the discrepancies are confirmed and analyzed using a cloud-based visual language model, resulting in the discrepancy confirmation results, specifically including: The cloud-based visual language model is used to perform multimodal understanding of environmental targets, road structures, traffic participants, vehicle trajectories, and driving behaviors in the event data. Based on the multimodal understanding results, determine whether the difference between the human driving trajectory and the algorithm-predicted trajectory is caused by at least one of the following reasons: effective driving intention, potential risk avoidance, abnormal behavior of traffic participants, missed perception, insufficient prediction, or unreasonable planning. When it is determined that the difference is related to the defects of the autonomous driving algorithm or the learning scenario of the algorithm, the valid difference result is output; If the discrepancy is determined to be caused by random operation by a human driver, driver distraction, unnecessary lane changes, abnormal human intervention, or behavior with no learning value, an invalid discrepancy result is output.

7. The method according to claim 1, characterized in that, Based on the difference confirmation results, the event data is filtered and labeled to obtain difference sample data for autonomous driving algorithm optimization, specifically including: When the difference confirmation result indicates that the preliminary difference event is a valid difference, the event data is associated with the corresponding root cause analysis information, scene label information and trajectory difference information and stored in the optimization database. When the difference confirmation result indicates that the preliminary difference event is an invalid difference, the event data is filtered, stored with reduced weight, or classified into the invalid difference database. Based on the differential sample data in the optimized database, a dataset is formed for training, testing, or replaying autonomous driving algorithms.

8. The method according to claim 1, characterized in that, After receiving the event data, the cloud analytics platform also performs cloud trajectory reconstruction processing, which includes: Based on the historical environmental perception data, vehicle status data, and trajectory data in the event data, the scene context information corresponding to the preliminary difference event is determined; A cloud-based trajectory prediction model is used to perform long-term time-series trajectory prediction on the scene context information to generate a cloud-based reference trajectory. The cloud-based reference trajectory, the human driving trajectory, and the algorithm-predicted trajectory are input into the cloud-based visual language model to assist the cloud-based visual language model in determining whether the algorithm-predicted trajectory has perception, prediction, or planning defects.

9. The method according to claim 1, characterized in that, Before the event data is uploaded to the cloud analytics platform, it also includes: Determine whether the preliminary difference event meets the upload conditions; The upload conditions include: the trajectory deviation reaches a preset upload threshold, the event location belongs to the target road area, the event data integrity meets the preset integrity requirements, the network status meets the upload requirements, and the event type does not belong to at least one of the preset filtering types. When the upload conditions are met, the event data is uploaded to the cloud analysis platform; If the upload conditions are not met, the event data may be cached locally, uploaded with a delay, or discarded.

10. A vehicle-mounted shadow mode driving difference confirmation device, characterized in that, include: The data acquisition module is used to acquire vehicle-side driving-related data during vehicle operation. The vehicle-side driving-related data includes at least environmental perception data, vehicle status data, human driving data, and algorithm decision data output by the autonomous driving algorithm. The shadow mode processing module is used to determine the human driving trajectory and the algorithm predicted trajectory based on the human driving data and the algorithm decision data, respectively. The difference judgment module is used to compare the human driving trajectory and the algorithm-predicted trajectory, and determine the corresponding preliminary difference event when the preset difference triggering conditions are met. The vehicle-side communication module is used to extract event data corresponding to the preliminary difference event from the vehicle-side driving association data based on the preliminary difference event, and upload the event data to the cloud analysis platform; The cloud-based analysis module is used to perform multimodal analysis input of the visual language model constructed by the cloud-based analysis platform based on the event data, and to perform validity confirmation and difference cause analysis on the preliminary difference event through the cloud-based visual language model to obtain the difference confirmation result; The result generation module is used to filter and label the event data based on the difference confirmation results to obtain difference sample data for autonomous driving algorithm optimization.

Images