A method for constructing a motion trajectory based on a visual large model

Abstract

The application discloses a motion trajectory construction method based on a visual large model, and relates to the technical field of computer vision, which comprises the following steps: a target tracking algorithm based on Kalman filtering is used to generate smooth time sequence synchronization trajectories from original laser radar point clouds and original video streams; a hybrid method combining the minimum description length principle and visual language model semantic verification is applied to divide the trajectories into a series of candidate kinematic segments which are complete in kinematics and semantics; visual language models are used to preliminarily label the segments, and initial labeled trajectories are generated; the initial labeled trajectories are refined for logic consistency until the refined labeled trajectories are obtained; and the refined trajectories are formatted into a standard structured semantic trajectory representation string. The application bridges the semantic gap between low-dimensional physical observations and high-dimensional driving intentions, and provides an ideal input for downstream complex scene understanding and prediction tasks.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer vision, and more particularly, to a motion trajectory construction method based on a visual large model. BACKGROUND

[0002] In the research and development and deployment of autonomous driving systems, reliable perception and understanding of dynamic and complex traffic environments is a fundamental challenge. Traditional perception pipelines rely on fusing multi-modal data from sensors such as lidar and cameras to construct geometric models of the world, and on this basis to perform object detection and state estimation. However, this paradigm suffers from a fundamental semantic gap between the low-level, quantized data produced by sensors and the high-level, intention-driven scene understanding possessed by human drivers. While existing systems can accurately detect objects and estimate their kinematic states, they struggle to infer the underlying driving intentions or contextual logic behind these motion states, especially when dealing with "long-tail scenarios" or complex interaction situations that are not well represented in the training data.

[0003] Current methods for vehicle trajectory understanding and prediction based on visual large models mainly rely on representing motion trajectories as purely geometric coordinate point sequences. This low-level representation lacks rich semantic information, forcing the visual large model to learn complex physical laws and high-level driving intentions from scratch. Research shows that this semantic gap between representation and reasoning is the core bottleneck of existing models' limited reasoning ability in complex traffic scenarios, and even merely repeating the text prompts rather than truly understanding the visual content.

[0004] In view of this, the present application proposes a motion trajectory construction method based on a visual large model to solve the above problems. SUMMARY

[0005] In order to overcome the above-mentioned defects of the prior art and achieve the above-mentioned purpose, the present application provides the following technical scheme: a motion trajectory construction method based on a visual large model, comprising:

[0006] Step F1, based on the collected original lidar point cloud and original video stream, and the preset Kalman filter state transition matrix, Kalman filter process noise covariance matrix and Kalman filter measurement noise covariance matrix, a target tracking algorithm based on Kalman filtering is applied to extract the motion trajectory of the target vehicle and align it with the video timestamp, to obtain a time-synchronized trajectory and a visual context video;

[0007] Step F2, based on the pre-acquired high-definition map data, and the preset minimum description length model primitive set and minimum description length coding precision, a hybrid segmentation method combining the minimum description length principle and visual language model semantic verification is applied to process the time-synchronized trajectory and visual context video to generate candidate kinematic segments determined according to the minimum description length principle and verified by the visual language model to ensure the continuity of driving intention;

[0008] Step F3, traversing the candidate kinematic segments, and combining the visual context video, high-definition map data and pre-defined semantic behavior ontology library, an initial semantic label is assigned to each segment using the visual language model to obtain an initial labeled trajectory;

[0009] Step F4, based on the visual context video and high-definition map data, and the preset referee visual language model confidence threshold, maximum refinement iteration number and collaborative teaching reasoning path set, an iterative loop including referee visual language model evaluation and collaborative teaching correction mechanism is adopted to process the initial labeled trajectory until the logical consistency of the label sequence converges, and an refined labeled trajectory is output;

[0010] Step F5, traversing each segment in the refined labeled trajectory, the semantic label, start time and end time of the segment are combined into a standardized text representation, and the text representations of all segments are spliced into a string to obtain a structured semantic trajectory representation.

[0011] Further, the implementation method of step F2 includes:

[0012] Step F2F1, a hybrid segmentation method based on the minimum description length principle is applied to process the time-synchronized trajectory according to the preset minimum description length model primitive set and minimum description length coding precision to generate optimal preliminary candidate segmentation points in the sense of information compression;

[0013] Step F2F2, initialize an empty final segmentation point list;

[0014] Step F2F3, for each of the preliminary candidate segment points, the steps F2F3F1 to F2F3F3 are performed cyclically: Step F2F3F1, extract the two adjacent motion segments before and after the candidate point as the segmentation point, and extract the information required for semantic verification from the visual context video and the high-definition map data to construct a visual language model semantic verification context; Step F2F3F2, construct a prompt word for guiding the step-by-step reasoning of the visual language model, and query the visual language model using the prompt word and the visual language model semantic verification context to determine whether the two adjacent motion segments before and after the candidate point jointly constitute a semantically continuous driving intention, and obtain a verification result; Step F2F3F3, if the verification result indicates that the two adjacent motion segments before and after the candidate point jointly constitute a semantically continuous driving intention, the candidate point is added to the final segment point list.

[0015] Step F2F4, using the final segment point list to split the time-synchronous trajectory to obtain a candidate kinematic segment.

[0016] Further, the implementation method of step F4 includes:

[0017] Step F4F1, copy the initial labeled trajectory as the current iteration trajectory;

[0018] Step F4F2, initialize the Boolean convergence flag to false and the integer iteration counter to 0;

[0019] Step F4F3, under the condition that the convergence flag is false and the iteration counter is less than the maximum number of refinement iterations, loop steps F4F3F1 to F4F3F4: Step F4F3F1, mark the current iteration trajectory as the last round trajectory, and set the convergence flag to true; Step F4F3F2, for each pair of adjacent semantic labels in the current iteration trajectory, guide a visual language model as a referee by constructing a prompt word, and combine the visual context video and high-definition map data corresponding to the pair of adjacent semantic labels to judge the logical consistency of the pair of adjacent semantic labels and output a confidence score, thereby identifying noisy semantic labels with logical inconsistency or a confidence score lower than a preset referee visual language model confidence threshold, and obtaining a noisy label index set; Step F4F3F3, if the noisy label index set is not empty, execute steps F4F3F3F1 and F4F3F3F2: Step F4F3F3F1, set the convergence flag to false; Step F4F3F3F2, for each noise index in the noisy label index set, loop steps F4F3F3F2F1 to F4F3F3F2F4: Step F4F3F3F2F1, for the noisy semantic label corresponding to the noise index, respectively query the visual language model using the first inference configuration and the second inference configuration included in the collaborative teaching inference path set to obtain candidate modified semantic label 1 and its confidence score 1, and candidate modified semantic label 2 and its confidence score 2; Step F4F3F3F2F2, if the candidate modified semantic label 1 and the candidate modified semantic label 2 are the same, select the same candidate modified semantic label as the final modified semantic label; Step F4F3F3F2F3, otherwise, select the candidate modified semantic label with the higher confidence score as the final modified semantic label; Step F4F3F3F2F4, update the label at the noise index in the current iteration trajectory using the final modified label; Step F4F3F4, increment the iteration counter by one.

[0020] Step F4F4, name the final current iteration trajectory as the refined annotation trajectory.

[0021] Further, the implementation method of step F2F1 includes:

[0022] Step F2F1F1, define and calculate the total description length of any subsequence of the time-synchronized trajectory, the calculation method of the total description length is: add the number of bits required to describe the model parameters of the geometric model selected from the preset minimum description length model primitive set for fitting the subsequence, and the number of bits required to encode the fitting error of all trajectory points in the subsequence to the geometric model;

[0023] Step F2F1F2, initialize the cost array and the split point index array;

[0024] Step F2F1F3, apply a dynamic programming algorithm to determine and record the optimal split points required to achieve the minimum total description length by an iterative solving process, the solving process including: for each current trajectory point in the time-synchronized trajectory, traverse all trajectory points before the current trajectory point as candidate split points, calculate the cumulative total description length when the candidate split point is taken as the last split point, select the candidate split point that makes the cumulative total description length minimum as the optimal predecessor split point of the current trajectory point, and store the selected minimum cumulative total description length and the index of the optimal predecessor split point into the cost array and the split point index array respectively;

[0025] Step F2F1F4, extract all optimal split points by backtracking the split point index array to obtain preliminary candidate segmentation points; the backtracking method is: starting from the end point of the time-synchronized trajectory, the optimal predecessor split point corresponding to the end point is found out by reverse lookup using the split point index array, and the reverse lookup is repeated with the found optimal predecessor split point as the starting point of the lookup until the starting point of the trajectory is traced back, and all optimal split points found in the entire backtracking process are recorded.

[0026] Further, the implementation method of step F2F3F1 includes:

[0027] Step F2F3F1F1, find the previous and next split points of the candidate point in the preliminary candidate segmentation point list, and define two adjacent motion segments centered on the candidate point;

[0028] Step F2F3F1F2, according to the time range of the two adjacent motion segments, extract the corresponding video clips from the visual context video;

[0029] Step F2F3F1F3, according to the spatial range of the two adjacent motion segments, extract the corresponding road network information from the high-definition map data;

[0030] Step F2F3F1F4, combine the extracted video clips and road network information to construct a visual language model semantic verification context.

[0031] Further, the implementation method of step F4F3F2 includes:

[0032] Step F4F3F2F1, initialize an empty noisy label index set;

[0033] Step F4F3F2F2, traverse all semantic labels from the first to the second last in the current iteration trajectory, for each pair of adjacent semantic labels therein, looply execute steps F4F3F2F2F1 to F4F3F2F2F4: Step F4F3F2F2F1, extract the visual context video slice and high-definition map data corresponding to the pair of adjacent semantic labels as a multi-modal context; Step F4F3F2F2F2, construct a multi-modal thinking chain prompt word, the prompt word is used to guide the visual language model as the judge to perform step-by-step reasoning, the step-by-step reasoning includes the following steps: identifying the previous semantic label, identifying the next semantic label, and determining whether the conversion from the previous semantic label to the next semantic label conforms to the driving logic based on the multi-modal context; Step F4F3F2F2F3, query the visual language model as the judge using the constructed prompt word and the multi-modal context to obtain a logical consistency judgment and a confidence score; Step F4F3F2F2F4, if the logical consistency judgment is no or the confidence score is lower than the confidence threshold of the judge visual language model, add the index of the pair of adjacent semantic labels to the noisy label index set.

[0034] Further, the implementation method of step F4F3F3F2F1 includes:

[0035] Step F4F3F3F2F1F1, extract the noisy semantic label corresponding to the noise index and its multi-modal context;

[0036] Step F4F3F3F2F1F2, use the multi-modal context to query the visual language model using the first reasoning configuration to obtain candidate modified semantic label 1 and confidence score 1;

[0037] Step F4F3F3F2F1F3, use the multi-modal context to query the visual language model using the second reasoning configuration to obtain candidate modified semantic label 2 and confidence score 2;

[0038] Wherein, the first reasoning configuration and the second reasoning configuration are not the same, the difference between them is that the multi-modal thinking chain prompt word template used to query the visual language model is different, or the queried visual language model is a model of different versions or different architectures.

[0039] The motion trajectory construction method based on a visual large model has the following advantages:

[0040] By adopting a hybrid segmentation method based on the minimum description length principle and semantic verification of visual language model, the application can generate kinematically efficient and semantically complete motion segments. Specifically, the MDL principle ensures that the segmentation is based on the optimal compression of the data structure itself, avoiding the arbitrariness of traditional heuristic methods; and the semantic verification of VLM further ensures that the segmentation points conform to the conversion of driving intention, bridging the gap between pure kinematic segmentation and high-level semantic understanding. This feature solves the problem that traditional methods are difficult to generate meaningful behavior units, providing a high-quality basis for subsequent precise semantic labeling.

[0041] By introducing an iterative cycle containing referee visual language model evaluation and collaborative teaching correction mechanism, the application significantly improves the logical consistency and reliability of the final semantic label sequence. For the logical contradictions that may be generated by VLM in single labeling, the collaborative teaching mechanism of the application generates mutually verified candidate labels through two different reasoning paths, and iteratively corrects them, effectively filtering out noise labels. This feature directly overcomes the defect of the lack of logical self-consistency in the semantic output of existing methods, ensuring the internal explainability and reliability of the trajectory representation.

[0042] Finally, the application formats the refined trajectory into a structured semantic trajectory representation, providing a high-quality, purified input for the downstream autonomous driving planning and decision-making system. The downstream module no longer needs to process the original coordinate point sequence full of noise, but can directly operate on the symbolic sequence composed of clear, logically verified action primitives. This not only fundamentally reduces the cognitive burden and reasoning complexity of the decision-making system, but also effectively decouples the perception understanding and planning decision-making modules, forming a verifiable, stable semantic log that is crucial for system safety verification and failure analysis. BRIEF DESCRIPTION OF DRAWINGS

[0043] Figure 1 The method flowchart of the motion trajectory construction method based on the visual large model of the application;

[0044] Figure 2 The system module schematic diagram of the motion trajectory construction system based on the visual large model of the application;

[0045] Figure 3 The application scenario schematic diagram of the motion trajectory construction method based on the visual large model of the application. DETAILED DESCRIPTION

[0046] With reference to the drawings of the embodiments of the present application, the technical solutions in the embodiments of the present application will be clearly and completely described. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the protection scope of the present application.

[0047] Embodiment 1

[0048] With reference to Figure 1 , the motion trajectory construction method based on a visual large model comprises steps F1 to F5.

[0049] This section describes in detail the complete implementation process of converting the original multi-modal sensor data stream into a high-level structured semantic trajectory representation through a series of processing and analysis stages. The process begins with a basic state estimation stage, where the Kalman filter algorithm is applied to convert the original noisy sensor readings into a smooth and physically reasonable time-synchronized trajectory. Subsequently, the process enters a core segmentation and labeling stage, which first performs data-driven kinematic segmentation using the Minimum Description Length principle (MDL), and then uses a visual language model (VLM) for semantic verification and preliminary labeling. Finally, to ensure the logical rigor of the final output, the process enters an iterative refinement stage that uses the Co-Teaching mechanism to self-correct and verify the consistency of the labeled sequence until the final, high-quality refined labeled trajectory is generated.

[0050] Step F1, based on the collected original laser radar point cloud and original video stream, and the preset Kalman filter state transition matrix , the Kalman filter process noise covariance matrix and the Kalman filter measurement noise covariance matrix , a target tracking algorithm based on Kalman filtering is applied to extract the motion trajectory of the target vehicle and align it with the video timestamp, obtaining a time-synchronized trajectory and a visual context video.

[0051] The original laser radar point cloud is derived from a vehicle-mounted multi-line rotating laser radar sensor, which conforms to the standard specifications of the autonomous driving industry. The sensor determines the distance by emitting laser pulses and measuring the time of flight after reflection from objects. Its internal rotating mechanism ensures periodic scanning of the surrounding environment with a scanning frequency not less than 10 Hz. The original data generated by each scan is a dense point set, where each point is quantized as a three-dimensional space coordinate relative to the sensor coordinate system and a high-precision timestamp synchronized with the vehicle's master clockstructured data. These time-stamped point cloud sequences collectively constitute the raw laser radar point cloud.

[0052] The raw video stream is captured by a front-facing, automotive industry standard, digital camera system mounted on the vehicle. The system continuously records video at a frame rate of no less than 25 frames per second to ensure smooth representation of motion. The video data is encoded and encapsulated in a standard digital container format that contains not only the sequence of image frames, but also an exact representation of the time stamp for each frame. This encapsulation ensures that the visual information can be accurately time-aligned with data from other sensors such as the raw laser radar point cloud.

[0053] Kalman filter state transition matrix is not an input obtained by measurement, but a matrix derived based on a determined kinematic model. Specifically, the Kalman filter state transition matrix is constructed based on a two-dimensional constant velocity (CV) model, which assumes that the velocity of an object remains constant within discrete time steps. For a state vector defined in the form to represent the position and velocity in and directions, , represent the position in and directions, , represent the velocity in and directions, the Kalman filter state transition matrix is mathematically constructed to linearly project the current state to the next time defined by the processing time step . Its structure in the formula directly implements the kinematic equations and , where the subscript represents the current time, the subscript represents the next time, , represent the position in and directions at the current time k, , represent the velocity in and directions at the current time k; the two kinematic equations are the theoretical core of the "prediction" step of the Kalman filter.

[0054] Kalman filter process noise covariance matrix To quantify the uncertainty of the motion model itself, it acknowledges that the constant velocity assumption is an idealized state, and real-world target vehicles will have unmodeled small random accelerations. This matrix is determined based on the "discrete Wiener process acceleration model". This model assumes that within a processing time step , the acceleration is a random walk process with a mean of zero. The magnitude of this random acceleration is controlled by the adjustable parameter process noise standard deviation . The Kalman filter process noise covariance matrix is calculated by the formula , where is the noise gain matrix responsible for mapping the noise in the acceleration dimension into the position and velocity state space, and T is the transpose of the matrix. This method provides a way to inject uncertainty into the model for the prediction step of the filter, preventing it from becoming overconfident in its own model and allowing it to better adapt to real-world maneuvering behavior.

[0055] The Kalman filter measurement noise covariance matrix represents the inherent uncertainty in the sensor measurement process. Its values are determined based on the manufacturer's technical specification sheet for the LiDAR sensor used to generate the raw LiDAR point cloud. This matrix is typically a diagonal matrix with diagonal elements representing the variance, i.e., the square of the standard deviation, of the measurement error in the and directions, respectively. These values directly quantify the measurement accuracy of the sensor and are assumed to follow independent Gaussian distributions. The Kalman filter measurement noise covariance matrix is crucial in the "update" step of the Kalman filter, as it determines the weight distribution between the trust in new measurements and the trust in the system's own predictions when fusing information.

[0056] The Kalman filter is a recursive Bayesian filtering algorithm for state estimation in dynamic systems with uncertainty. It effectively fuses prediction based on a motion model and noisy sensor measurements through a "prediction-update" loop, making it a standard and fundamental method in the field of target tracking.

[0057] This step employs the Kalman filter, whose theoretical foundation is based on linear dynamic systems and Gaussian noise assumptions. It provides an optimal linear estimator for extracting smooth and physically reasonable motion trajectories from noisy sensor data, such as LiDAR point clouds, by minimizing the mean squared error of the state estimates. The constant velocity model is chosen as the base model because it strikes a good balance between computational efficiency and accuracy in describing most driving scenarios.

[0058] The implementation method of step F1 includes: step F1F1, identifying an initial position of the target vehicle from a first frame of the original laser radar point cloud . Step F1F2, initializing a state vector of a two-dimensional constant velocity Kalman filter using the initial position and a zero initial velocity . Step F1F3, for each subsequent frame in the original laser radar point cloud, steps F1F3F1 to F1F3F4 are executed in a loop: step F1F3F1, predicting the state at the previous time using the Kalman filter state transition matrix to obtain the predicted state at the current time. Step F1F3F2, extracting the measured position of the target vehicle from the current laser radar frame. Step F1F3F3, performing the update step of the Kalman filter in combination with the predicted state, the measured position, the Kalman filter process noise covariance matrix and the Kalman filter measurement noise covariance matrix to obtain the corrected state at the current time. Step F1F3F4, appending the position and timestamp information in the corrected state to the trajectory sequence. Step F1F4, summarizing the position and timestamp information in the corrected state at all times to obtain a time-synchronized trajectory. Step F1F5, according to the timestamps in the time-synchronized trajectory, cutting and synchronizing the original video stream to obtain a visual context video.

[0059] The time-synchronized trajectory generated by this step is a time series data structure, where each element contains a timestamp and a two-dimensional position coordinate of the target vehicle after Kalman filter smoothing at the timestamp . Logically, the time-synchronized trajectory represents the optimal linear estimate of the true motion trajectory of the target vehicle, which eliminates high-frequency noise in the original sensor data and fills in possible short-term measurement gaps. The time-synchronized trajectory will serve as the core input for subsequent kinematic segmentation steps, providing a clean, continuous and physically reasonable coordinate sequence.

[0060] The visual context video generated by this step is a processed video file stream, whose time reference is completely aligned with the time-synchronized trajectory. Logically, the visual context video provides corresponding and real visual scene information for each time point in the time-synchronized trajectory, such as road conditions, traffic signals, pedestrian poses, etc. The visual context video will serve as the key multi-modal input for all subsequent steps that require semantic understanding, such as semantic verification, initial labeling and consistency refinement, providing necessary visual evidence for the reasoning of the visual language model VLM.

[0061] After obtaining the smoothed time-synchronized trajectory by Kalman filtering, the next core task is to extract meaningful, discrete motion units from this continuous sequence of coordinate points. Directly using a fixed time window or simple threshold based on speed, acceleration for segmentation, it is easy to produce fragments that are semantically incomplete or meaningless. In order to identify the key "change points" in the trajectory in an objective and data-driven manner, the subsequent step will introduce the minimum description length principle based on information theory MDL. The logical intention of this move is to reconstruct the trajectory segmentation problem as a data compression problem, so as to find the segmentation point set that can describe the entire trajectory in the most concise way, ensuring the mathematical optimality of segmentation.

[0062] Step F2, based on the pre-acquired high-definition map data, and the preset minimum description length model primitive set and minimum description length coding precision, a hybrid segmentation method combining the minimum description length principle and visual language model semantic verification is applied to process the time-synchronized trajectory and visual context video, to generate candidate kinematic segments determined according to the minimum description length principle and verified by the visual language model to ensure the continuity of driving intention.

[0063] High-definition map data is derived from professional surveying and mapping collection and is encoded and stored in vector format. This data is not generated in real time, but is input as a priori knowledge. Its quantization method involves accurately representing physical world road elements such as lane centerlines, stop lines, road boundaries, and traffic sign locations as a series of geometric primitives and their topological relationships. High-definition map data must contain centimeter-precision static road network information to provide accurate geospatial constraints for subsequent scene understanding and semantic analysis.

[0064] The minimum description length model primitive set is a list of strings determined by design selection, which defines the geometric model library that the MDL segmentation algorithm can use when fitting the trajectory. For example, [‘line’, ‘arc’] indicates that the algorithm can only use straight line and circular arc segments to describe the trajectory, where ‘line’ represents a straight line segment and ‘arc’ represents a circular arc segment. The determination of this set is based on prior knowledge of the target scene, such as typical vehicle motion patterns in urban driving, i.e., most driving behaviors can be decomposed into a combination of straight line and circular arc motion. The minimum description length model primitive set is a key design parameter that directly constrains the expressiveness and complexity of the model.

[0065] The minimum description length coding precision is a positive floating-point number, which is a design parameter used to set the numerical precision used when encoding the parameters of each primitive in the minimum description length model primitive set in bits, ​The geometric model selected from the minimum description length model base. The determination of the minimum description length encoding precision is a trade-off between model simplicity and fitting accuracy. Higher minimum description length encoding precision allows more accurate model parameters, but increases the cost; lower minimum description length encoding precision is the opposite. The specific value is set based on the analysis of the coordinate range of the trajectory data and the required fitting accuracy.

[0066] The mixed segmentation method used in this step is based on two mature theories. First, the minimum description length principle is a core concept in information theory and statistical inference. This principle formalizes model selection as data compression, considering the model that can describe the data with the shortest encoding length as the best model. Applying this principle to trajectory segmentation is equivalent to finding a mathematically simplest segment representation that captures the data's inherent structure, thereby avoiding the use of heuristics or arbitrary thresholds. Second, semantic verification of MDL-generated candidate points is feasible due to the strong zero-shot reasoning and scene understanding capabilities exhibited by modern visual language models. VLMs can integrate multi-modal information, including vision, maps, and text, and perform human-like common-sense reasoning to determine whether a kinematic breakpoint also corresponds to a driving intent transition point, ensuring that the final segments are complete in both kinematic and semantic dimensions.

[0067] The core innovation of this step is the formal combination of bottom-up information-theoretic optimality and top-down semantic reasonableness. Traditional trajectory segmentation methods either rely solely on kinematic thresholds, which can produce meaningless segments, or rely entirely on models, which can ignore the data's inherent structure. This method uses the MDL principle to first find the mathematically simplest segmentation points from the perspective of data compression, ensuring kinematic efficiency. Then, using the VLM's strong scene understanding capabilities, it performs semantic pruning on these candidate points to ensure that the segmentation boundaries align with the start and end points of high-level driving intents.

[0068] The minimum description length principle is derived from information theory, and its core idea is that the best explanation of data is the one that allows the data to be compressed the most. Treating trajectory segmentation as a data compression problem allows for the identification of inherent, non-arbitrary "change points." The introduction of VLM is based on its ability to connect visual, textual, and high-level concepts in emerging multi-modal reasoning tasks, making it an ideal tool for verifying the semantic validity of segmentation points. This combination ensures that the final segments are optimal in both kinematic and semantic dimensions.

[0069] ​The implementation method of step F2 includes: step F2F1, applying a hybrid segmentation method based on the minimum description length principle, processing the time-synchronous trajectory according to a preset minimum description length model primitive set and minimum description length coding accuracy, and generating a preliminary candidate segmentation point that is optimal in the sense of information compression. Step F2F2, initializing an empty final segmentation point list. Step F2F3, for each candidate point in the preliminary candidate segmentation point, steps F2F3F1 to F2F3F3 are cyclically executed: step F2F3F1, extracting two adjacent motion segments before and after the candidate point as the segmentation point, and extracting information required for semantic verification from the visual context video and the high-definition map data to construct a visual language model semantic verification context. Step F2F3F2, querying the visual language model using a multi-modal thinking chain prompt word to determine whether the candidate point destroys the integrity of the semantically continuous driving intention due to small fluctuations in kinematics, and obtaining a verification result. Step F2F3F3, if the verification result determines that the intention integrity is not destroyed, the candidate point is added to the final segmentation point list. Step F2F4, using the final segmentation point list to segment the time-synchronous trajectory to obtain a candidate kinematic segment.

[0070] The implementation method of step F2F1 includes: step F2F1F1, defining and calculating the total description length of any subsequence of the time-synchronous trajectory. The calculation method of the total description length is: adding the number of bits required to describe the model parameters used to fit the subsequence to the number of bits required to encode the fitting error of all trajectory points in the subsequence to the model. The total description length is represented by the formula ; in the formula, represents the total description length of the subsequence, in bits, represents the subsequence data from the trajectory point to the point , represents the encoding length of the geometric model , represents the encoding length of the fitting error, that is, the number of bits required to encode the data D under the premise that the geometric model is known, and D represents the subsequence data Step F2F1F2, initialize cost array Cost and split point index array Split. Step F2F1F3, apply dynamic programming algorithm to determine and record the optimal split points needed to achieve the minimum total description length by iterating a solving process, which includes: for each current trajectory point in the time-synchronized trajectory, traverse all the trajectory points before the current trajectory point as candidate split points, calculate the cumulative total description length when the candidate split point is taken as the last split point, select the candidate split point that makes the cumulative total description length minimum as the optimal predecessor split point for this current point, and store the selected minimum cumulative total description length and the index of the optimal predecessor split point into cost array and split point index array respectively. The iterative solving is by a recursive formula where, denotes the minimum cumulative total description length of splitting the trajectory from the start point to point denotes the minimum cumulative total description length of splitting the trajectory from the start point to point is the total description length of describing the sub-sequence from point to point , and denotes the min operation. Step F2F1F4, extract all the optimal split points by backtracking the split point index array to get the preliminary candidate segment points; the backtracking method is: starting from the end point of the time-synchronized trajectory, find out the optimal predecessor split point corresponding to the end point by using the split point index array, and repeat the reverse search with the found optimal predecessor split point as the starting point of the search until the start point of the trajectory is traced back, and record all the optimal split points found in the entire backtracking process.

[0071] This step formalizes the trajectory segmentation problem as a data compression problem, aiming to find the most concise representation of the trajectory. Dynamic programming is a standard and efficient algorithm to solve such sequence segmentation problems with optimal substructure property.

[0072] The preliminary candidate segment points generated by this step is an ordered list of time stamps, each of which represents an optimal split point in the time-synchronized trajectory calculated based on the minimum description length principle. Logically, the preliminary candidate segment points represent the critical points where the vehicle motion pattern changes significantly from the perspective of kinematics and information compression only. The preliminary candidate segment points will be used as input for the subsequent semantic verification step, which aims to filter out redundant points that do not constitute a semantic change in driving intent by leveraging the reasoning ability of the VLM.

[0073] ​​The implementation method of step F2F3F1 includes: step F2F3F1F1, finding the previous and next segmentation points of the candidate point in the preliminary candidate segmentation point list, and defining two adjacent motion segments centered on the candidate point. Step F2F3F1F2, according to the time range of the two adjacent motion segments, extracting the corresponding video clip from the visual context video. Step F2F3F1F3, according to the spatial range of the two adjacent motion segments, extracting the corresponding road network information from the high-definition map data. Step F2F3F1F4, combining the extracted video clip and road network information to construct the visual language model semantic verification context.

[0074] The visual language model semantic verification context generated by this step is a temporary data structure containing multi-modal information around a specific candidate point. In terms of data form, it aggregates a video clip and a segment of vectorized map data. Its logical role is to provide a concentrated spatio-temporal context closely related to the segmentation point to be verified for the visual language model. The purpose of the visual language model semantic verification context is to serve as direct input for the next VLM query, enabling the VLM to judge whether the segmentation point is semantically reasonable based on sufficient visual and geographic spatial evidence.

[0075] The candidate kinematic segmentation generated by this step is a list of trajectory segments, each of which is a subsequence of a time-synchronized trajectory, whose segmentation boundaries have been verified by MDL optimality calculation and VLM semantic reasonableness. In logic, the candidate kinematic segmentation represents a collection of a series of kinematically stable and semantically complete behavior units, laying a solid foundation for subsequent explicit semantic labeling. The candidate kinematic segmentation will be used as input for step F3 to assign each segment a clear label from the semantic behavior ontology library.

[0076] After generating a set of candidate kinematic segments with both kinematic and semantic completeness, the next goal is to assign clear, human-understandable semantic labels to these anonymous behavior units. The logical intent of this step is to use the powerful zero-shot classification capability of the visual language model to map the visual and map context information of each segment to a predefined driving behavior vocabulary. By constructing a prompt word containing multi-modal information and constraining VLM to select from the semantic behavior ontology library, a preliminary high-level semantic label can be generated for each segment, thus completing the preliminary conversion from continuous trajectory to discrete semantic sequence.

[0077] Step F3, traversing the candidate kinematic segments and combining the visual context video, high-definition map data, and pre-defined semantic behavior ontology library, using the visual language model to assign an initial semantic label to each segment to obtain an initial labeled trajectory.

[0078] The semantic behavior ontology is a pre-defined, closed list of strings by domain experts. The construction of this list follows the principles of driving behavior taxonomy, aiming to exhaust all possible basic driving actions or intentions in the target scenario, such as <straight driving at constant speed>, <overtaking by accelerating>, <decelerating to yield>, and <waiting at intersection>. Each label is an atomized semantic primitive. The quantized form of this ontology is a fixed enumeration set, which provides a restricted, standardized output space for the subsequent VLM annotation task, ensuring the consistency and interpretability of the labels.

[0079] This step leverages the strong capability of visual language models in zero-shot or few-shot visual question answering (VQA) and description generation. By providing rich multi-modal context such as video and map, and constraining the selection within a pre-defined semantic behavior ontology, a preliminary, high-level semantic label can be generated for each segment that is complete in both kinematics and semantics.

[0080] The implementation method of step F3 includes: step F3F1, initializing an empty initial annotation trajectory. For each motion segment in the candidate kinematic segments, steps F3F2F1 to F3F2F5 are executed in a loop: step F3F2F1, extract the corresponding time slice of the motion segment in the visual context video to obtain a segment video slice. Step F3F2F2, extract the corresponding road network information of the motion segment in the high-definition map data to obtain a segment map context. Step F3F2F3, construct a prompt word, which contains the segment video slice, the segment map context, and an instruction to select the most appropriate label from the semantic behavior ontology. Step F3F2F4, query a visual language model using the constructed prompt word and parse its output to obtain a segment semantic label. Step F3F2F5, append the motion segment and its corresponding segment semantic label as an entry to the initial annotation trajectory.

[0081] The initial annotation trajectory generated by this step is a structured list, where each entry contains a motion segment, i.e., a sequence of spatio-temporal coordinates, and its corresponding segment semantic label assigned by the VLM. Logically, the initial annotation trajectory is the first data product in the entire process that fully links continuous kinematic data with discrete semantic concepts. However, since the VLM annotation is performed independently for each segment, the trajectory may have logical inconsistencies, such as a behavior followed by an unreasonable one. Therefore, the purpose of the initial annotation trajectory is to serve as input for the next iterative refinement process to correct any noise and logical errors that may exist.

[0082] The initial annotated trajectory generated in Step F3, while assigning semantics to each motion segment, can lack global logical consistency in the output label sequence, as the VLM is independently annotating each segment. For example, the model can identify "acceleration" in one segment and "stop" in the next segment, ignoring the transition of "deceleration" that is usually required between the two behaviors in a real driving scenario. To address this issue and ensure that the final output representation is logically self-consistent, a subsequent step will introduce an iterative self-correction loop. The logical intent of this step is to explicitly treat the initial labels as hypotheses that can contain noise and to systematically detect and correct these logical inconsistencies using a robust learning framework.

[0083] In Step F4, based on the visual context video and HD map data, as well as the pre-set referee visual language model confidence threshold, maximum refinement iteration number, and set of collaborative teaching reasoning paths, an iterative loop including referee visual language model evaluation and collaborative teaching correction mechanism is used to process the initial annotated trajectory until the logical consistency of the label sequence converges, outputting the refined annotated trajectory.

[0084] The referee visual language model confidence threshold is a floating-point number between 0 and 1, which is a key design threshold. The logical basis for setting this threshold is that during the iterative consistency refinement process, a quantitative standard is needed to determine whether the "referee" VLM's evaluation of a label transition is sufficiently reliable. When the VLM's output confidence score is below this threshold, even if it is judged to be "logically consistent," the system will treat it as a low-confidence, potentially problematic "noise" sample and trigger the correction process. The specific setting of this value is usually determined by experiments on the validation set, seeking a point that balances the recall and precision, i.e., finding the best balance between identifying truly inconsistent transitions and avoiding misjudging consistent transitions as inconsistent.

[0085] The maximum refinement iteration number is a positive integer, which is one of the termination conditions for the iterative process. The logical basis for setting it is to prevent the system from failing to reach the convergence condition of logical consistency in some special cases, such as oscillating between two contradictory correction schemes and falling into an infinite loop. The determination of this value is an engineering trade-off: it needs to be large enough to ensure that the algorithm has enough opportunity to converge to a stable state in most cases; at the same time, it cannot be too large to avoid unnecessary computational overhead. Its specific value is usually set based on empirical observations of the convergence speed of typical scenarios.

[0086] The co-teaching reasoning path set is a list containing two different reasoning configurations. The logical basis for this is a core requirement of the Co-Teaching paradigm: the two models or reasoning processes involved in co-teaching must be diverse to avoid them falling into the same thought patterns and reinforcing each other's errors. To achieve this, each configuration in the list aims to create an independent, diverse reasoning path from the other. This diversity can be quantified and achieved in various ways, such as: (1) using two different multi-modal thought chain MCoT prompt word templates to guide the VLM to reason from different angles or in different steps; (2) calling two different VLM API endpoints, which may point to different versions or different architectures of the model. The co-teaching reasoning path set ensures the diversity of the two candidate correction labels generated, which is key to effectively identifying and correcting systematic bias.

[0087] The iterative correction mechanism used in this step is based on the Co-teaching paradigm in the field of robust learning. Co-teaching has been proven to be very effective in dealing with training data containing noisy labels. Its core idea is to train two or more models simultaneously and let them "teach" each other the samples they consider "clean", i.e. small loss samples, thereby effectively filtering out noise and preventing models from memorizing incorrect labels and overfitting. This scheme innovatively applies this paradigm to correcting the semantic label sequence generated by the VLM. Through the co-teaching reasoning path set, the two "teacher" VLMs are ensured to be diverse, which can effectively break the self-reinforcing confirmation bias that a single model may produce in iterations, thereby significantly improving the logical consistency and robustness of the final label sequence.

[0088] The core innovation of this step is that it explicitly models the initial output of the VLM as potentially containing noise, and for the first time applies the Co-teaching idea in robust learning to the iterative self-correction process of the VLM. Traditional methods are usually one-off feedforward labeling, while the self-consistency of the VLM can cause it to reinforce its initial biases or errors in iterations.

[0089] Co-teaching is an effective paradigm for robust training in the presence of label noise. Its core mechanism is to train two models simultaneously and let them "teach" each other the samples they consider "clean", i.e. high confidence, logically consistent labels. By introducing two diverse reasoning paths to generate correction suggestions, the co-teaching reasoning path set ensures that this step actively breaks the cycle of self-reinforcement of a single model, effectively filtering out those labels that may seem reasonable in isolation but are not logically consistent in continuous narratives, thereby significantly improving the logical consistency and reliability of the final representation.

[0090] The implementation method of step F4 includes: step F4F1, copying the initial annotation track as the current iteration track. Step F4F2, initializing the Boolean convergence flag as false and the integer iteration counter as 0. Step F4F3, under the condition that the convergence flag is false and the iteration counter is less than the maximum number of refinement iterations, the steps F4F3F1 to F4F3F4 are executed in a loop: step F4F3F1, marking the current iteration track as the last round track and setting the convergence flag to true. Step F4F3F2, using the "referee" VLM, combining the visual context video and the pre-acquired high-definition map data, and according to the preset referee visual language model confidence threshold, evaluating the logical consistency of all adjacent label pairs in the current iteration track, and identifying all "noisy" labels and their indexes that are judged to be inconsistent or have a confidence lower than the threshold, to obtain a set of noisy label indexes. Step F4F3F3, if the set of noisy label indexes is not empty, steps F4F3F3F1 and F4F3F3F2 are executed: step F4F3F3F1, setting the convergence flag to false. Step F4F3F3F2, for each noise index in the set of noisy label indexes, steps F4F3F3F2F1 to F4F3F3F2F4 are executed in a loop: step F4F3F3F2F1, for the noisy semantic label corresponding to the noise index, respectively using the first inference configuration and the second inference configuration included in the set of collaborative teaching inference paths to query the visual language model, to obtain the candidate modified semantic label 1 and its confidence score 1, and the candidate modified semantic label 2 and its confidence score 2. Step F4F3F3F2F2, if the candidate modified semantic label 1 and the candidate modified semantic label 2 are the same, the same candidate modified semantic label is selected as the final modified semantic label. Step F4F3F3F2F3, otherwise, the candidate modified semantic label with the higher confidence score is selected as the final modified semantic label. Step F4F3F3F2F4, using the final modified label to update the label at the noise index in the current iteration track. Step F4F3F4, incrementing the iteration counter by one. Step F4F4, naming the final current iteration track as the refined annotation track.

[0091] The implementation method of step F4F3F2 includes: step F4F3F2F1, initializing an empty noisy label index set. Step F4F3F2F2, traversing all semantic labels from the first to the second last in the current iteration track, for each pair of adjacent semantic labels in it, the steps F4F3F2F2F1 to F4F3F2F2F4 are executed in a loop: step F4F3F2F2F1, extracting the visual context video slice and high-definition map data corresponding to the pair of adjacent semantic labels as the multi-modal context. Step F4F3F2F2F2, constructing a multi-modal chain-of-thought (MCoT) prompt to guide the "referee" VLM to perform step-by-step reasoning, for example: "Step 1: Identify the previous behavior as . Step 2: Identify the next behavior as . Step 3: Check the video and map to determine whether the transition from to is consistent with the driving logic in the current scene. Step 4: Output the 'yes' or 'no' judgment and give a confidence score between 0 and 1." Step F4F3F2F2F3, querying the "referee" VLM using the constructed prompt and multi-modal context to obtain a logical consistency judgment and a confidence score. Step F4F3F2F2F4, if the logical consistency judgment is "no" or the confidence score is lower than the referee visual language model confidence threshold, then add the indexes and to the noisy label index set.

[0092] The multimodal chain-of-thought (MCoT) prompting technique can significantly improve the performance of the model on complex multi-modal tasks by guiding the model to generate a series of intermediate reasoning steps. This step uses MCoT to ensure that the VLM's judgment is not based on simple pattern matching, but on a structured and logical understanding of the scene.

[0093] The implementation method of step F4F3F3F2F1 includes: step F4F3F3F2F1F1, extracting the noisy semantic label corresponding to the noise index and its multi-modal context. Step F4F3F3F2F1F2, using the first reasoning configuration in the collaborative teaching reasoning path set and the multi-modal context, the first reasoning configuration may be, for example, the first MCoT prompt template, and the multi-modal context may be, for example, a video slice and map information, querying the visual language model to obtain a candidate corrected label 1 and its confidence score. Step F4F3F3F2F1F3, using the second reasoning configuration in the collaborative teaching reasoning path set and the multi-modal context, the second reasoning configuration may be, for example, the second MCoT prompt template or a different VLM model, querying the visual language model to obtain a candidate corrected label 2 and its confidence score.

[0094] In ensemble learning and robust learning, diversity of models is the key to performance improvement. This step enforces diversity by using two different inference paths, ensuring that the two candidate revised labels generated are not from the same mindset, which is crucial for effectively identifying and correcting systematic biases.

[0095] The refined labeled trajectory generated in this step is a structured list, with the same data form as the initial labeled trajectory, but its core logical value lies in the fact that the semantic label sequence it contains has been iteratively checked for logical consistency and revised. The refined labeled trajectory represents a verified, semantically coherent narrative of driving behavior. The refined labeled trajectory will serve as input for the final formatting step, to generate the final structured semantic trajectory representation for downstream tasks.

[0096] Step F5, for each segment in the refined labeled trajectory, combine the semantic label, start time, and end time of the segment into a standardized textual representation, and concatenate the textual representations of all segments into a string to obtain the structured semantic trajectory representation.

[0097] The structured semantic trajectory representation generated in this step is a single, formatted string. In terms of data form, it is a concatenation of a series of substrings following the format label (t_start = start time, t_end = end time) separated by spaces. Logically, the structured semantic trajectory representation is the final deliverable of the entire technical solution, which abstracts a complex, continuous vehicle motion process into a discrete, high-level, human- and machine-readable semantic symbol sequence. The structured semantic semantic trajectory representation aims to serve as an ideal input for downstream advanced intelligent tasks such as behavior prediction, intent reasoning, and risk assessment, greatly simplifying the complexity of these tasks by providing a pre-processed and logically verified structured information.

[0098] This embodiment details the complete process of converting raw multi-modal sensor data streams into structured semantic representations. The process begins with extracting smooth time-synchronized trajectories from raw lidar point clouds and raw video streams through Kalman filtering. The trajectory is then processed by a segmentation algorithm based on the principle of minimum description length, generating candidate kinematic segments. These segments, after preliminary labeling by a visual language model, form the initial labeled trajectory. Finally, through an iterative refinement cycle based on Co-teaching, the initial labeled trajectory is revised for logical consistency, resulting in a refined labeled trajectory, and finally formatted into a structured semantic trajectory representation.

[0099] The process successfully converts a continuous, low-dimensional kinematic problem into a discrete, high-dimensional semantic reasoning problem by systematically combining optimal state estimation, information theory, multi-modal AI reasoning, and robust learning theory. As shown in Figure 3 the final product of the process is a structured semantic trajectory representation, providing an unprecedented, highly abstract, logically verified, and machine-readable input for downstream autonomous driving planning and decision-making systems, thereby fundamentally bridging the semantic gap from raw observations to high-level intent understanding.

[0100] Embodiment 2

[0101] Referring to Figure 2 the embodiment provides a visual large model-based motion trajectory construction system, the system comprising:

[0102] An extraction module extracts the motion trajectory of the target vehicle based on the collected raw laser radar point cloud and raw video stream, and the preset Kalman filter state transition matrix, Kalman filter process noise covariance matrix, and Kalman filter measurement noise covariance matrix, applies a Kalman filter-based target tracking algorithm, and aligns the motion trajectory of the target vehicle with the video timestamp to obtain a time-synchronized trajectory and a visual context video;

[0103] A segmentation module processes the time-synchronized trajectory and the visual context video based on the pre-collected high-definition map data, and the preset minimum description length model primitive set and minimum description length coding precision, applies a hybrid segmentation method combining the minimum description length principle and semantic verification of a visual language model to generate candidate kinematic segments determined according to the minimum description length principle and verified by the visual language model to ensure the continuity of driving intent;

[0104] An initial labeling module iterates through the candidate kinematic segments, combines the visual context video, high-definition map data, and pre-defined semantic behavior ontology library, and uses a visual language model to assign an initial semantic label to each segment to obtain an initial labeled trajectory;

[0105] A labeling refinement module processes the initial labeled trajectory based on the visual context video and high-definition map data, and the preset referee visual language model confidence threshold, maximum refinement iteration number, and collaborative teaching reasoning path set, adopts an iterative loop including referee visual language model evaluation and collaborative teaching correction mechanism until the logical consistency of the label sequence converges, and outputs a refined labeled trajectory;

[0106] A semantic trajectory module iterates through each segment in the refined labeled trajectory, combines the semantic label, start time, and end time of the segment into a standardized text representation, and concatenates the text representations of all segments into a string to obtain a structured semantic trajectory representation.

[0107] The prior art is used or referred to in the application where not described.

[0108] Each of the embodiments in the specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each of the embodiments mainly describes the difference from other embodiments.

[0109] The above is only an embodiment of the application and is not intended to limit the application. Those skilled in the art can make various modifications and changes to the application. Any modification, equivalent replacement, improvement, etc. within the spirit and principle of the application shall be included in the scope of the claims of the application.

Claims

1. A method for constructing a motion trajectory based on a visual large model, characterized in that, Comprise: Step F1, based on the collected original laser radar point cloud and original video stream, and the preset Kalman filter state transition matrix, Kalman filter process noise covariance matrix and Kalman filter measurement noise covariance matrix, applying Kalman filter based target tracking algorithm, extracting the motion trajectory of the target vehicle and aligning with the video timestamp, obtaining the time sequence synchronization trajectory and visual context video; Step F2, based on the pre-acquired high-definition map data, and the preset minimum description length model primitive set and minimum description length coding precision, applying a hybrid segmentation method combining the minimum description length principle and visual language model semantic verification, processing the time sequence synchronization trajectory and visual context video, to generate candidate kinematic segments determined according to the minimum description length principle and verified by the visual language model to ensure the continuity of driving intention; Step F3, traversing the candidate kinematic segments, and combining the visual context video, high-definition map data and pre-defined semantic behavior ontology library, using the visual language model to assign an initial semantic label to each segment, to obtain an initial labeled trajectory; Step F4, based on the visual context video and high-definition map data, and the preset referee visual language model confidence threshold, maximum refinement iteration number and collaborative teaching reasoning path set, using an iterative loop including referee visual language model evaluation and collaborative teaching correction mechanism to process the initial labeled trajectory until the logical consistency of the label sequence converges, outputting the refined labeled trajectory; Step F5, traversing each segment in the refined labeled trajectory, combining the semantic label, start time and end time of the segment into a standardized text representation, concatenating the text representations of all segments into a string, to obtain a structured semantic trajectory representation.

2. The method of claim 1, wherein the method is based on a visual large model. The implementation method of step F2 comprises: Step F2F1, applying a hybrid segmentation method based on the minimum description length principle, processing the time sequence synchronization trajectory according to the preset minimum description length model primitive set and minimum description length coding precision, to generate the optimal preliminary candidate segmentation point in the sense of information compression; Step F2F2, initializing an empty final segmentation point list; Step F2F3, for each of the preliminary candidate segmentation points, the step F2F3F1 to step F2F3F3 are performed cyclically: step F2F3F1, extract the two adjacent motion segments before and after the candidate point as the segmentation point, and extract the information required for semantic verification from the visual context video and the high-definition map data to construct a visual language model semantic verification context; step F2F3F2, construct a prompt word for guiding the visual language model to perform step-by-step reasoning, and query the visual language model using the prompt word and the visual language model semantic verification context to determine whether the two adjacent motion segments before and after the candidate point jointly constitute a semantically continuous driving intention, and obtain a verification result; step F2F3F3, if the verification result indicates that the two adjacent motion segments before and after the candidate point jointly constitute a semantically continuous driving intention, the candidate point is added to the final segmentation point list; Step F2F4, using the final segmentation point list to split the time-synchronous trajectory to obtain a candidate kinematic segmentation.

3. The method of claim 1, wherein the method is based on a visual large model. The implementation method of step F4 includes: Step F4F1, copying the initial labeled trajectory as the current iteration trajectory; Step F4F2, initializing the Boolean convergence flag to false and the integer iteration counter to 0; Step F4F3, under the condition that the convergence flag is false and the iteration counter is less than the maximum number of refinement iterations, loop steps F4F3F1 to F4F3F4: Step F4F3F1, mark the current iteration trajectory as the last round trajectory, and set the convergence flag to true; Step F4F3F2, for each pair of adjacent semantic labels in the current iteration trajectory, guide a visual language model as a referee by constructing a prompt word, and combine the visual context video and high-definition map data corresponding to the pair of adjacent semantic labels to judge the logical consistency of the pair of adjacent semantic labels and output a confidence score, thereby identifying noisy semantic labels with logical inconsistency or a confidence score lower than a preset referee visual language model confidence threshold, and obtaining a noisy label index set; Step F4F3F3, if the noisy label index set is not empty, execute steps F4F3F3F1 and F4F3F3F2: Step F4F3F3F1, set the convergence flag to false; Step F4F3F3F2, for each noise index in the noisy label index set, loop steps F4F3F3F2F1 to F4F3F3F2F4: Step F4F3F3F2F1, for the noisy semantic label corresponding to the noise index, respectively query the visual language model using the first inference configuration and the second inference configuration included in the collaborative teaching inference path set to obtain candidate modified semantic label 1 and its confidence score 1, and candidate modified semantic label 2 and its confidence score 2; Step F4F3F3F2F2, if the candidate modified semantic label 1 and the candidate modified semantic label 2 are the same, select the same candidate modified semantic label as the final modified semantic label; Step F4F3F3F2F3, otherwise, select the candidate modified semantic label with the higher confidence score as the final modified semantic label; Step F4F3F3F2F4, update the label at the noise index in the current iteration trajectory using the final modified label; Step F4F3F4, increment the iteration counter by one; Step F4F4, name the final current iteration trajectory as the refined annotation trajectory.

4. The method of claim 2, wherein the method further comprises: The implementation method of step F2F1 includes: Step F2F1F1, define and calculate the total description length of any subsequence of the time-synchronous trajectory, the calculation method of the total description length is: add the number of bits required to describe the model parameters of the geometric model selected from the preset minimum description length model primitive set for fitting the subsequence, and the number of bits required to encode the fitting error of all trajectory points in the subsequence to the geometric model; Step F2F1F2, initialize the cost array and the split point index array; Step F2F1F3, applying a dynamic programming algorithm, determining and recording the optimal split points required to achieve the minimum total description length by an iterative solving process, the solving process including: for each current trajectory point in the time-synchronized trajectory, traversing all trajectory points before the current trajectory point as candidate split points, calculating the cumulative total description length when the candidate split point is taken as the last split point, selecting the candidate split point that minimizes the cumulative total description length as the optimal predecessor split point of the current trajectory point, and storing the selected minimum cumulative total description length and the index of the optimal predecessor split point in a cost array and a split point index array, respectively; Step F2F1F4, extracting all optimal split points by backtracking the split point index array to obtain preliminary candidate segmentation points; the backtracking method is: starting from the end point of the time-synchronized trajectory, using the split point index array to find the optimal predecessor split point corresponding to the end point in reverse, and repeating the reverse search with the found optimal predecessor split point as the starting point, until the starting point of the trajectory is traced back, and all optimal split points found in the entire backtracking process are recorded.

5. The method of claim 2, wherein the method further comprises: The implementation method of step F2F3F1 includes: Step F2F3F1F1, finding the previous and next split points of the candidate point in the preliminary candidate segmentation point list, and defining two adjacent motion segments centered on the candidate point; Step F2F3F1F2, according to the time range of the two adjacent motion segments, extracting the corresponding video slices from the visual context video; Step F2F3F1F3, according to the spatial range of the two adjacent motion segments, extracting the corresponding road network information from the high-definition map data; Step F2F3F1F4, combining the extracted video slices and road network information to construct a visual language model semantic verification context.

6. The method of claim 3, wherein the method further comprises: The implementation method of step F4F3F2 includes: Step F4F3F2F1, initializing an empty noisy label index set; Step F4F3F2F2, traverse all semantic labels from the first to the second last in the current iteration trajectory, for each pair of adjacent semantic labels therein, looply execute steps F4F3F2F2F1 to F4F3F2F2F4: Step F4F3F2F2F1, extract the visual context video clip and high-definition map data corresponding to the pair of adjacent semantic labels as a multi-modal context; Step F4F3F2F2F2, construct a multi-modal thinking chain prompt word, the prompt word is used to guide the visual language model as the judge to perform step-by-step reasoning, the step-by-step reasoning includes the following steps: identifying the previous semantic label, identifying the next semantic label, and determining whether the conversion from the previous semantic label to the next semantic label conforms to the driving logic based on the multi-modal context; Step F4F3F2F2F3, query the visual language model as the judge using the constructed prompt word and the multi-modal context to obtain a logical consistency judgment and a confidence score; Step F4F3F2F2F4, if the logical consistency judgment is no or the confidence score is lower than the confidence threshold of the judge visual language model, add the index of the pair of adjacent semantic labels to the noisy label index set.

7. The method of claim 3, wherein the method further comprises: The implementation method of step F4F3F3F2F1 includes: Step F4F3F3F2F1F1, extract the noisy semantic label corresponding to the noise index and its multi-modal context; Step F4F3F3F2F1F2, use the first reasoning configuration to query the visual language model using the multi-modal context to obtain a candidate modified semantic label 1 and a confidence score 1; Step F4F3F3F2F1F3, use the second reasoning configuration to query the visual language model using the multi-modal context to obtain a candidate modified semantic label 2 and a confidence score 2; Wherein, the first reasoning configuration and the second reasoning configuration are not the same, the difference between them is that the multi-modal thinking chain prompt word templates used to query the visual language model are different, or the queried visual language model is a model of different versions or different architectures.

Images