A method, device, mobile vehicle, and storage medium for predicting driving scenarios.

Abstract

This invention relates to the field of computer technology and discloses a method, apparatus, mobile vehicle, and storage medium for predicting driving scenes. The method includes: obtaining a subtask to be executed within a first time period after the current moment from the driving task of the mobile vehicle as a target subtask; wherein the driving task includes at least two subtasks with a temporal relationship; the first time period includes at least two time blocks with a temporal relationship; obtaining the real-time motion parameters of the mobile vehicle at the current moment and the historical image sequence collected within a second time period before the current moment; and predicting the image sequence and motion parameters of the mobile vehicle for each time block in the first time period based on the historical image sequence, the real-time motion parameters, and the target subtask, to obtain a driving scene prediction result. This invention solves the problem of discontinuous dynamic prediction of future image sequences and driving states in existing driving scene prediction methods.

Description

Technical Field

[0001] This invention relates to the field of computer technology, and specifically to a method, apparatus, mobile vehicle, and storage medium for predicting driving scenarios. Background Technology

[0002] In autonomous driving systems, mobile vehicles need to predict future environmental changes to support safe and efficient driving decisions. Currently, the Vision-Language-Action (VLA) model, as the mainstream architecture, achieves a comprehensive understanding of driving scenarios through unified perception, reasoning, and control. Video, as a rich form of environmental representation, not only contains low-level visual details (such as color, texture, and object shape) but also high-level semantic information (such as the movement patterns of traffic participants) and implicitly reflects the vehicle's trajectory. Therefore, it is considered an important carrier of knowledge about the physical world and plays a crucial role in future prediction and decision-making.

[0003] Existing driving scene prediction methods typically predict future image sequences frame-by-frame or sparsely framed. When predicting multiple future time points, it is difficult to continuously and completely depict the dynamic changes of future image sequences and driving states, resulting in discontinuous and incomplete capture of changes in the driving state of mobile vehicles. Summary of the Invention

[0004] In view of this, embodiments of the present invention provide a driving scene prediction method, device, mobile vehicle, and storage medium to solve the problem of discontinuous dynamic prediction of future image sequences and driving states in existing driving scene prediction methods.

[0005] In a first aspect, embodiments of the present invention provide a driving scenario prediction method, the method comprising: Obtain the subtasks to be executed within a first duration after the current moment from the driving task of the mobile vehicle as target subtasks; wherein, the driving task includes at least two subtasks with a temporal relationship; the first duration includes time blocks with at least two temporal relationships; The real-time motion parameters of the mobile vehicle at the current moment, and the historical image sequence collected during the second time period prior to the current moment are obtained; Based on the historical image sequence, the real-time motion parameters, and the target sub-task, the image sequence and motion parameters of the mobile vehicle in each time block within the first duration are predicted to obtain the driving scene prediction result.

[0006] Furthermore, the method for predicting the image sequence and motion parameters of the mobile vehicle in the first time block of the time series includes: Based on the visual features extracted from the historical image sequence, the driving features extracted from the real-time motion parameters, and the target sub-task, a first reference feature corresponding to the first time block is generated; Based on the first reference feature, a first predicted image sequence associated with the first time block and a first predicted motion parameter are predicted.

[0007] Furthermore, the method for predicting the image sequence and motion parameters of the mobile vehicle in a target time block other than the first time block includes: Determine the preceding reference feature corresponding to the preceding time block; wherein, the preceding time block is the time block before the target time block; if the preceding time block is a first time block, then the preceding reference feature is the first reference feature; if the preceding time block is any time block after the first time block, then the preceding reference feature is the reference feature corresponding to that arbitrary time block. Based on the aforementioned prior reference features, predict the target reference features corresponding to the target time block; Based on the target baseline features, predict the target predicted image sequence and target predicted motion parameters associated with the target time block.

[0008] Furthermore, the step of generating the first baseline feature corresponding to the first time block based on the visual features extracted from the historical image sequence, the driving features extracted from the real-time motion parameters, and the target sub-task includes: Extract visual features from the historical image sequence; Spatiotemporal compression is applied to the visual features to obtain visual implicit features; Based on the real-time motion parameters, driving features are generated, and guidance information associated with the first time block is obtained. The guidance information is used to guide the target prediction model to predict the image sequence and motion parameters of the first time block based on the visual features, the visual implicit features, and the driving features. The visual features, the visual implicit features, the driving features, the target sub-task, and the guidance information are used as the first reference features.

[0009] Furthermore, predicting the target reference features corresponding to the target time block based on the preceding reference features includes: Using the target sub-task in the preceding reference features as the basis for predicting the mobile vehicle in the target time block, the target visual features of the mobile vehicle in the target time block are predicted using the visual features in the preceding reference features, and the target driving features of the vehicle in the target time block are predicted using the driving features in the preceding reference features. Spatiotemporal compression is performed on the target visual features to obtain target visual implicit features, and target guidance information corresponding to the target time block is obtained. The target guidance information is used to guide the target prediction model to predict the image sequence and motion parameters of the target time block based on the target visual features, the target visual implicit features, and the target driving features. The target visual features, the target visual implicit features, the target driving features, the target sub-tasks, and the target guidance information are used as the target baseline features.

[0010] Furthermore, the step of predicting the first predicted image sequence and the first predicted motion parameters associated with the first time block based on the first reference feature includes: The visual implicit features in the first reference features are input into the visual renderer, and the first predicted image sequence of the first time block is constructed using the historical image sequence as a condition. At the same time, the driving features in the first reference features are input into the state decoder to obtain the first predicted motion parameters of the first time block.

[0011] Furthermore, the step of predicting the image sequence and motion parameters of the mobile vehicle for each time block within the first duration based on the historical image sequence, the real-time motion parameters, and the target sub-task, to obtain the driving scene prediction result, includes: The historical image sequence, the real-time motion parameters, and the target sub-task are input into a pre-trained target prediction model. The target prediction model analyzes the historical image sequence and the real-time motion parameters according to the target sub-task, and outputs the image sequence and motion parameters associated with each time block of the mobile vehicle in the first time period after the current moment to predict the driving scene prediction result.

[0012] Furthermore, before inputting the historical image sequence, the real-time motion parameters, and the subtask into the pre-trained target prediction model, the method further includes: Obtain a training data sequence, wherein the training data sequence includes at least one set of samples, the set of samples including image sequence samples and corresponding motion parameter samples and subtask samples; Based on the training data sequence, in the short time domain prediction task, the duration of the first sample is divided into at least one first sample time block according to the first time interval, and the initial decision model is trained in the first stage, so that the initial decision model learns the correlation between the image sequence samples and the corresponding motion parameter samples of each first sample time block according to the sub-task samples, until the first convergence condition is met, and a candidate decision model is obtained. Based on the training data sequence, in the long-term prediction task, the duration of the first sample is divided into at least one second sample time block according to the second time interval, and the candidate decision model is trained in the second stage. The candidate decision model learns the correlation between the image sequence samples and the corresponding motion parameter samples of each second sample time block according to the sub-task samples, until the second convergence condition is met, and the target prediction model is obtained.

[0013] Furthermore, before acquiring the training data sequence, the method further includes: Obtain the original training data sequence, wherein the original training data sequence includes a set of original samples from multiple consecutive time segments, and each set of original samples includes original image sequence samples and corresponding original motion parameter samples and original subtask samples; In the original training data sequence, the original sample set is sampled according to a third time interval to obtain a first training sequence, wherein the first training sequence includes: a first image sequence sample and corresponding first motion parameter samples and first subtask samples; Identify key time segments in the time segments, and perform weighted sampling on the original sample set associated with the key time segments in the original training data sequence to obtain a second training sequence, wherein the second training sequence includes: a second image sequence sample and corresponding second motion parameter samples and second subtask samples; The training data sequence is generated using the first training sequence and the second training sequence.

[0014] Secondly, embodiments of the present invention provide a driving scenario prediction device, the device comprising: The first acquisition module is used to acquire, from the driving task of the mobile vehicle, a subtask to be executed within a first duration after the current moment as a target subtask; wherein, the driving task includes at least two subtasks with a temporal relationship; the first duration includes at least two time blocks with a temporal relationship. The second acquisition module is used to acquire the real-time motion parameters of the mobile vehicle at the current moment, and the historical image sequence collected within a second time period before the current moment; The prediction module is used to predict the image sequence and motion parameters of the mobile vehicle in each time block of the first duration based on the historical image sequence, the real-time motion parameters, and the target sub-task, so as to obtain the driving scene prediction result.

[0015] Thirdly, embodiments of the present invention provide a mobile vehicle, the mobile vehicle comprising: a controller, the controller comprising: a memory and a processor, the memory and the processor being communicatively connected to each other, the memory storing computer instructions, and the processor executing the computer instructions to perform the method described in the first aspect or any corresponding embodiment thereof.

[0016] Fourthly, embodiments of the present invention provide a computer device, including: a memory and a processor, the memory and the processor being communicatively connected to each other, the memory storing computer instructions, and the processor executing the computer instructions to perform the method described in the first aspect or any corresponding embodiment thereof.

[0017] Fifthly, embodiments of the present invention provide a computer-readable storage medium storing computer instructions for causing a computer to perform the method described in the first aspect or any corresponding embodiment thereof.

[0018] The method provided in this application has the following beneficial effects: The method provided in this application, by obtaining the target sub-tasks to be executed within a first time period after the current moment from the driving task, and the first time period containing at least two time blocks with temporal relationships, can decompose the macroscopic driving target into short-term executable prediction-guided units, thereby improving the alignment accuracy between the prediction task and high-level planning and reducing interference from irrelevant scenes; by obtaining the real-time motion parameters of the mobile vehicle at the current moment and the historical image sequence within a second time period before the current moment, it can provide continuous and dynamic contextual information for subsequent predictions, so as to capture the temporal dependency of the mobile vehicle's motion trend and environmental changes, and enhance the understanding of historical states. It fully utilizes historical image sequences, real-time motion parameters, and target sub-tasks to predict the image sequences and motion parameters of a mobile vehicle in each time block within the first time duration. This decomposes the long-term prediction task into multiple structured time blocks, with each time block independently predicting its corresponding image sequence and motion parameters. This reduces computational redundancy while maintaining temporal resolution, achieving a balance between fine-grained dynamic characterization and computational efficiency. It can also output continuous future scene information in parallel or recursively, generating consistent driving scene prediction results in units of time blocks, thereby improving the ability to continuously characterize the future trajectory and visual changes of mobile vehicles over long time. Attached Figure Description

[0019] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0020] Figure 1 This is a flowchart illustrating the driving scenario prediction method according to an embodiment of the present invention; Figure 2 This is a schematic diagram of the visual feature processing and image reconstruction process according to an embodiment of the present invention; Figure 3 This is a visualization diagram of the prediction results of the future driving scenario of a mobile vehicle according to an embodiment of the present invention; Figure 4 This is a flowchart illustrating another driving scenario prediction method according to an embodiment of the present invention; Figure 5 This is a flowchart illustrating the training method of a prediction model according to an embodiment of the present invention; Figure 6 This is a schematic diagram of the architecture of a driving scenario prediction model according to an embodiment of the present invention; Figure 7 This is a structural block diagram of a driving scene prediction device according to an embodiment of the present invention; Figure 8 This is a schematic diagram of the hardware structure of a computer device according to an embodiment of the present invention. Detailed Implementation

[0021] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0022] According to embodiments of the present invention, a driving scenario prediction method, apparatus, mobile vehicle, and storage medium are provided. It should be noted that the steps shown in the flowcharts in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowcharts, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0023] This embodiment provides a method for predicting driving scenarios. Figure 1 This is a flowchart of a driving scenario prediction method according to an embodiment of the present invention, such as... Figure 1 As shown, the process includes the following steps: Step S101: Obtain the subtask to be executed within a first duration after the current moment from the driving task of the mobile vehicle as the target subtask; wherein, the driving task contains at least two subtasks with a temporal relationship; the first duration includes time blocks with at least two temporal relationships.

[0024] In this embodiment, the mobile vehicle can refer to an autonomous vehicle, aircraft, intelligent robot, etc., equipped with an x-Foresight predictive world model. It can perceive its surrounding environment and perform driving tasks through sensors (such as surround-view cameras).

[0025] A driving task is a complete set of travel actions that a mobile vehicle must perform, such as "navigating from point A to point B", "making a safe left turn at an intersection", or "passing through a tollbooth". This task is predefined as a sequence of at least two subtasks that have a temporal logical relationship.

[0026] Subtasks are the breakdown of a complete driving task into stages, each with a start and finish point. Tasks like changing lanes, stopping at intersections, following other vehicles, and crossing intersections fall into this category. For example, the subtasks for "navigating from point A to point B" include: planning the route, driving normally along the road, yielding to other vehicles, and arriving at the designated point B. The subtasks for "safely turning left at an intersection" include: changing left-turn lanes in advance, waiting at the stop line at the intersection, observing road conditions, and turning left out of the intersection.

[0027] The current moment is not limited to the very beginning of the entire driving task, but can be retried before each subtask needs to be executed (i.e., the interval between the end of the previous subtask and the start of the next subtask). Specifically: when the prediction is first initiated, the current moment is before the start of the first subtask of the entire driving task. Subsequently, whenever a subtask is completed and the next subtask is about to begin, a new "current moment" can be set. At each such current moment, the subtasks to be executed within the first time interval after the current moment are reacquired as target subtasks, and a round of forward prediction is re-executed to continuously update future predictions based on the latest status. Therefore, the current moment is the decision gap between the end of the previous subtask and the start of the first target subtask of the current driving task. For example, after the subtask of driving straight is completed, the moment before the target subtask of changing the left-turn lane in advance is the current moment corresponding to this round of prediction.

[0028] The first time period is a fixed predicted time interval, such as the next 6 seconds. The first time period is divided according to the LARCH long-horizon autoregressive chunking strategy of the x-Foresight framework. Specifically, the Long-horizon Autoregressive Chunking (LARCH) strategy divides the first time period into multiple consecutive time blocks, and then calculates the predictions sequentially. Except for the first time block, the predictions for each subsequent time block reference the results obtained from the previous time block.

[0029] Each time block contains multiple consecutive frames of images and motion data to capture short-term, instantaneous dynamic changes (such as a vehicle's sudden minor steering adjustment or a pedestrian's starting action). Long-term causal relationships between different time blocks are learned through "cross-block" predictions in the LARCH strategy. For example, the first duration (6 seconds) can be divided into 3 time blocks, each corresponding to a 2-second sequence of image frames.

[0030] A target subtask refers to the subtask selected from the sequence of time-related subtasks included in the "driving task" that needs to be performed by the moving vehicle within the "first duration" following the "current moment". For example, if the driving task is "turn right at the next traffic light intersection", it includes subtasks such as "entering the right turn lane", "slowing down and observing", and "performing the right turn". If the first duration after the current moment (the next 6 seconds) happens to cover the "entering the right turn lane" stage, then "entering the right turn lane" is the target subtask.

[0031] In this specific implementation, the mobile vehicle's decision-making system (i.e., the visual language model part of the x-Foresight framework) first receives a "driving task," such as "merge into the main road from the current lane." The system's internal task parser then decomposes this driving task into an ordered list of subtasks: [Subtask A: Find a merging gap; Subtask B: Adjust vehicle speed to match the main road traffic flow; Subtask C: Perform a lane change and merge].

[0032] Next, the system obtains the first duration given by the planning module or user instructions, for example, T_pred = 6 seconds. At the same time, according to the LARCH strategy configuration, the system divides T_pred into multiple time blocks of equal or unequal length, for example, into three consecutive time blocks of T_chunk = 2 seconds [Chunk_1 (0-2s), Chunk_2 (2-4s), Chunk_3 (4-6s)].

[0033] Then, obtain the timestamp of the current time t=0 (the instant before the first subtask is about to begin execution), and iterate through the above list of subtasks to estimate the expected execution time of each subtask. For example, based on historical statistics or rules, "finding the merging gap" takes an average of 1.5 seconds, "adjusting speed" takes 2 seconds, and "performing a lane change to merge" takes 2 seconds.

[0034] These subtasks are then mapped sequentially onto the timeline of the first duration: Subtask A (0-1.5s) covers part and all of the first time block (0-2s); Subtask B (1.5-3.5s) covers the remainder of the first time block, all of the second time block (2-4s), and part of the third time block; Subtask C (3.5-5.5s) covers the remainder of the third time block.

[0035] Next, identify the subtasks that begin execution within the first time interval [0, 6s]. Since the start times of subtasks A, B, and C (0s, 1.5s, 3.5s) all fall within [0, 6s], they are theoretically all "subtasks to be executed." However, to ensure that the prediction model has a clear and singular objective, the system will further select the most important subtask or the one with the longest coverage time as the "target subtask."

[0036] As an example, the highest-priority subtask that needs to be executed within the first time block is selected as the target subtask. Therefore, subtask A is selected as the target subtask. This target subtask is encoded as a text token (e.g., "find merging gap") and serves as part of a subsequent system prompt or user prompt given to the prediction model, guiding the model to generate image sequences and motion parameters consistent with that subtask. For example, when the model receives the target subtask "find merging gap," its predicted 6-second image sequence will be more likely to show the vehicle's dynamics of slightly accelerating or decelerating to find a gap, rather than performing a sharp turn.

[0037] Step S102: Obtain the real-time motion parameters of the mobile vehicle at the current moment, and the historical image sequence collected during the second time period before the current moment.

[0038] In this embodiment, real-time motion parameters can be derived or measured from current control actions (such as steering wheel angle, throttle opening, and brake pressure) or by onboard sensors (such as inertial measurement units, wheel speedometers, and GPS / inertial navigation systems). Specifically, real-time motion parameters include, but are not limited to: longitudinal velocity, lateral velocity, acceleration, yaw rate, heading angle, and absolute or relative position coordinates.

[0039] The second duration refers to the length of a time window preceding the "current moment," used to define the time range of the acquired "historical image sequence." This duration is a predefined hyperparameter, and its value selection needs to balance the length of the perceptual context with computational efficiency. Based on the requirements for long-term causal inference in the world model, the second duration is typically set to cover the complete evolution of key dynamic behaviors in typical driving scenarios (such as vehicles cutting in from the side or pedestrians crossing), for example, 2 to 4 seconds. The second duration is distinguished from the first duration (future prediction window); the second duration provides historical context, while the first duration defines the range of future predictions.

[0040] A historical image sequence refers to a collection of image frames continuously captured and arranged chronologically within a "second duration" time window preceding the "current moment" by multi-view surround-view cameras deployed on a mobile vehicle (e.g., seven views: left rear, left front, forward narrow view, forward wide view, rear narrow view, right front, and right rear). This sequence is one of the core inputs to the visual language model in the x-Foresight model, carrying information such as the spatial structure of the environment, the movement patterns of traffic participants, and the historical trajectory of the mobile vehicle. Before being input into the model, these images are typically converted into visual tokens by a Vision Transformer (ViT) to extract low-level visual details (color, texture, shape) and high-level semantic information (vehicle and pedestrian movement patterns), thereby supporting causal predictions of the future environment.

[0041] In the specific implementation, firstly, the chassis CAN bus or real-time communication interface of the mobile vehicle provides raw sensor data, including but not limited to: triaxial acceleration and angular velocity output by the inertial measurement unit; wheel speeds output by wheel speed sensors; steering angle output by the steering wheel angle sensor; and absolute speed, heading angle, and position output by differential GPS or high-precision integrated navigation system. The real-time preprocessing module performs time synchronization, filtering (such as Kalman filtering), and coordinate transformation on the above raw data at a fixed frequency (e.g., 50Hz or 100Hz) to obtain a set of state vectors at the current time t=0. This vector at least includes: longitudinal velocity v_x (m / s), lateral velocity v_y (m / s), yaw rate ω (rad / s), and absolute heading angle ψ (rad). If the model requires higher-order information, accelerations a_x and a_y can also be added. Finally, the real-time motion parameters are organized into a fixed-length feature vector, such as [v_x, v_y, ω, ψ], as input for driving features. If a sensor signal is lost (such as GPS lockout), a short-term calculation is performed using a kinematic model (e.g., a recursion based on wheel speed and the state at the previous moment), and the confidence level is marked.

[0042] Secondly, multiple surround-view cameras (e.g., 7 cameras covering 360°) are deployed on the mobile vehicle. These cameras have all been pre-calibrated for internal and external parameters and time synchronization. Each camera continuously captures images at a fixed frame rate (e.g., 15fps or 30fps).

[0043] Read the pre-configured hyperparameter T_history (i.e., the second duration), for example, set to 3 seconds. Combined with the camera's frame rate (Frames Per Second, FPS) (e.g., 15fps), calculate the total number of frames to be acquired, N = T_history * FPS = 45 frames. Then, starting from the current time t=0, backtrack along the timeline and retrieve the most recent N frames from the circular buffer of each camera. To ensure the temporal continuity and alignment of the sequence, images from all cameras are synchronously extracted according to a unified timestamp; that is, for each timestamp t_i (t_i = -T_history, -T_history+Δt, …, 0), images from all 7 perspectives are acquired simultaneously. Perform necessary normalization (e.g., scaling to 256×256 or a resolution matching the visual encoder input size), distortion correction, and color space conversion (e.g., BGR to RGB) on each raw image. Consolidate the multi-view images at the same time along the channel dimension or time dimension, or keep them in an independent list format for subsequent feature extraction.

[0044] The resulting historical image sequence is a five-dimensional tensor or list: [Time steps, number of views, channels, height, width], where time steps = 45 and number of views = 7. The system maintains a fixed-length circular queue (FIFO) to store image data for T_history + δ seconds in real time (δ is a small margin, such as 0.5 seconds, to prevent insufficient data during backtracking). Each execution reads the latest T_history seconds of data directly from this queue to ensure real-time performance.

[0045] Through the above implementation, the prediction model provides accurate current motion state and rich historical visual context, enabling x-Foresight to infer future dynamic scenes based on past causal evolution.

[0046] Step S103: Based on historical image sequences, real-time motion parameters, and target sub-tasks, predict the image sequences and motion parameters of the mobile vehicle for each time block in the first duration to obtain the driving scene prediction result.

[0047] In this application embodiment, a schematic diagram of the visual feature processing and image reconstruction process is shown, as follows: Figure 2As shown, the input is a sequence of historical images from img1 to img4. First, the image sequence is processed by a visual encoder to extract visual features. Then, these visual features are fed into a Feature Alignment Encoder for spatiotemporal compression: first, a 3D convolutional layer efficiently downsamples the temporal dimension, then a Transformer encoder block consisting of a single-head self-attention layer and a Feed Forward Network (FFN) is used to obtain compact and low-redundancy visual implicit features. Simultaneously, the historical image sequence is also directly passed as conditional information to subsequent modules. The visual implicit features and conditional information are input together into a pixel decoder, which gradually recovers high-resolution pixel details through continuous temporal upsampling and multi-scale spatial upsampling, ultimately reconstructing and outputting an image sequence (or future predicted frames) containing img1 to img4. This entire process achieves spatiotemporal compression and pixel-level reconstruction from historical observation features to high-fidelity future images, effectively alleviating the low-entropy redundancy problem of video tokens and perfectly adapting to the lightweight real-time look-ahead prediction application requirements of the x-Foresight framework.

[0048] It should be noted that the driving scenario prediction results demonstrate the high-fidelity prediction capability of this method for the future environment, such as... Figure 3 As shown, the visualization results of the Vision Renderer, conditioned on visual implicit features, are as follows: the horizontal axis represents the seven panoramic camera views (left rear, left front, forward narrow view, forward wide view, rear narrow view, right front, and right rear), and the vertical axis represents time from t=0 s (actual observation frame) to t=6 s (future prediction frame). The blue superimposed trajectories in the forward narrow and forward wide views represent future trajectories predicted by the visual language model based on action tokens; this trajectory information was not directly observed by the renderer. The predicted images output by the renderer achieve high fidelity in texture, color, and motion consistency. This visualization result intuitively verifies the practical application of the x-Foresight framework in multi-view global scene look-ahead prediction.

[0049] In one embodiment of this application, the method for predicting the image sequence and motion parameters of a mobile vehicle in a first time block of time sequence includes: Step A1: Based on the visual features extracted from historical image sequences, the driving features extracted from real-time motion parameters, and the target sub-task, generate the first baseline features corresponding to the first time block.

[0050] Specifically, the historical image sequence is first processed using a Vision Transformer (ViT) to extract visual features containing environmental semantic information. Then, the visual features are spatiotemporally compressed using a feature alignment encoder to obtain visual implicit features. Next, the real-time motion parameters of the vehicle are analyzed to generate driving features that reflect its own operating status and control intentions. At the same time, guidance information for the corresponding first time block is obtained in combination with prediction requirements. Finally, the visual features, visual implicit features, driving features, target sub-tasks and guidance information are integrated as the first baseline features.

[0051] Step A2: Based on the first baseline features, predict the first predicted image sequence associated with the first time block and the first predicted motion parameters.

[0052] Specifically, based on the first reference features, predicting the first predicted image sequence and the first predicted motion parameters associated with the first time block includes: inputting the visual implicit features in the first reference features into a visual renderer, constructing the first predicted image sequence of the first time block using the historical image sequence as a condition, and simultaneously inputting the driving features in the first reference features into a state decoder to obtain the first predicted motion parameters of the first time block.

[0053] The Vision Renderer is a high-fidelity image generation module based on a diffusion model. Its core function is to reconstruct detailed and realistic multi-view future image frames by using compact, inference-optimized "visual implicit features" as conditions, combined with historical images within a short time window. The Vision Renderer is decoupled from the visual-language-action model, handling only pixel-space reconstruction without participating in high-level causal inference. This design avoids excessive preservation of visual details in implicit features, forcing the language model to focus on learning structured world knowledge, while simultaneously providing high-quality closed-loop visual feedback through the renderer. The State Decoder is a lightweight neural network module (e.g., a multilayer perceptron or linear layer) used to map driving features (i.e., dynamic vectors encoded from real-time motion parameters) to specific future motion parameters. Its output is the vehicle's trajectory points or control variables over a future time period (e.g., discrete path points or steering wheel angle sequences for the next 6 seconds). The State Decoder works in parallel with the Vision Renderer, handling action prediction and image generation separately, reflecting the design philosophy of separating imagination and control in x-Foresight.

[0054] The first predicted image sequence refers to the set of future multi-view surround view images generated for the first time block (i.e., the first time block in the first duration). This sequence contains predicted images from the seven camera views corresponding to all time steps within this time block and is the direct output of the visual renderer. These predicted images are used both in subsequent autoregressive loops (as input conditions for the next time block) and in calculating supervised loss with real future images (during the training phase). The first predicted motion parameters refer to the predicted future motion state of the moving vehicle for the first time block, usually represented as a trajectory point sequence or control variable. The model predicts the trajectory of the moving vehicle for the next 6 seconds, and the first time block refers to the first segment within 6 seconds, such as 0 to 2 seconds. The state decoder, combined with driving features, calculates the first round of predicted motion parameters, including longitudinal speed, lateral offset, yaw angle changes, etc., to guide the actual control of the vehicle.

[0055] This step is implemented in two parts: future image sequence prediction (via a visual renderer) and future motion parameter prediction (via a state decoder). Both share different components of the same first baseline feature, but are computed independently. On one hand, the visual implicit feature Zlatent is extracted from the first baseline feature. This feature has been compressed into a compact spatiotemporal representation (e.g., a vector of shape [Nt′, V×L, D] or after global pooling). The historical image sequence Ihist is obtained, which consists of real multi-view images (raw pixel data) acquired within the second time period prior to the current moment. This historical image sequence serves as the conditional image for the renderer, providing a reference for high-frequency visual details. The visual renderer is based on a diffusion model architecture, and its input conditions include: Condition 1: Visual implicit feature Zlatent (providing structured semantics of the future scene, such as object movement trends and expected vehicle trajectories); Condition 2: Historical image sequence Ihist (providing realistic textures, lighting, background details, etc., as a pixel-level reference). The renderer's goal is to generate a complete panoramic image for each time step t=t1,t2,…,tK (a total of K frames) within the first time block, given the above conditions.

[0056] The specific diffusion process is as follows: Starting from pure noise, denoising is iteratively performed through a denoising network (such as U-Net). In each denoising step, visual implicit features and historical images are injected into the network through a cross-attention mechanism. The final output is a clear image frame, represented as [K,V,C,H,W][K,V,C,H,W], where VV is the number of viewpoints (e.g., 7), CC is the color channel (3), and H,WH,W are the image height and width. The renderer is not conditioned on action tokens, but only on camera tokens (visual implicit features) and historical images, thus forcing the renderer to faithfully decode the future content predicted by the Large Language Model (LLM) and avoid shortcut decoupling. The K-frame image sequence generated by the renderer is denoted as Ipred,1, which is the first predicted image sequence. This sequence can be used for closed-loop autoregression (feedback to the visual encoder as input to the next time block) or for visualization.

[0057] On the other hand, the driving feature fdyn is extracted from the first baseline feature. This feature is a fixed-dimensional vector (e.g., D=768) that encodes the vehicle's current real-time motion state (speed, acceleration, yaw rate, etc.). The state decoder is a regression network, typically consisting of 2-3 fully connected layers with ReLU activation. The output dimension depends on the form of the predicted motion parameters. The model predicts the vehicle's trajectory for the next 6 seconds, and the first time block corresponds to the first sub-interval (e.g., 0-2 seconds). Therefore, the output can be a 2×M2×M matrix (M trajectory points, each containing longitudinal displacement x and lateral displacement y), or more refined frame-by-frame control quantities (e.g., desired speed, steering wheel angle sequence). During training, this output is compared with the true motion parameters to calculate the Mean Squared Error (MSE) loss; during inference, the output is directly used as a reference for control commands or trajectory planning. The first predicted motion parameter is output: the output of the state decoder is denoted as Ppred,1, which is the first predicted motion parameter. This parameter can be used by the downstream control module and will also be part of the preceding driving features for subsequent time block predictions.

[0058] The visual renderer generates an image sequence, and the state decoder generates motion parameters; together, they form the complete prediction result for the first time block. This result serves two purposes: if there are subsequent time blocks (second and third blocks), the first predicted image sequence is fed back to the visual encoder, serving as part of the preceding reference features along with the first predicted motion parameters for the next time block. If it is the last time block, the overall prediction result is directly output. Through this implementation, x-Foresight can simultaneously obtain high-fidelity future images and accurate trajectory predictions within the first time block.

[0059] In one embodiment of this application, a first reference feature corresponding to a first time block is generated based on visual features extracted from historical image sequences, driving features extracted from real-time motion parameters, and a target sub-task, including: Step A101: Extract visual features from the historical image sequence.

[0060] Specifically, visual features refer to the structured, high-dimensional semantic representations extracted from raw images using a Vision Transformer (ViT). In the x-Foresight framework, visual features are a multi-layered abstraction of the driving environment. They not only contain low-level visual information of the image (such as color, texture, edges, and shape) but also encode high-level semantic information such as the spatial location and motion patterns of traffic participants (vehicles, pedestrians, etc.) and the geometric structure of the scene. Visual features exist in the form of a sequence of "visual tokens," each corresponding to a local region (patch) in the image, aggregating the global context through a self-attention mechanism. Compared to raw pixels, visual features significantly reduce data dimensionality while preserving key semantics, providing a compact and expressive foundational representation for subsequent spatiotemporal compression, causal inference, and future prediction.

[0061] This step is specifically implemented based on the VLA visual backbone network in x-Foresight, extracting visual features from historical image sequences according to the following process: First, the acquired historical image sequence is denoted as I={It,v}, where t= Thist, Thist+Δt,…,0 represents the time step (Nt frames in total), v=1,…,V represents the viewpoint number of the surround-view camera (e.g., V=7). Each frame image It,v undergoes preprocessing (scaled to a uniform resolution such as H×W, normalized to the range [0,1]). For each frame image It,v, the visual encoder first divides it into fixed-size two-dimensional image patches. For example, if each patch is 16×16, then each image can be divided into L=(H / 16)×(W / 16) patches. Subsequently, each patch is flattened and mapped to a dimension D embedding vector through a trainable linear projection layer to obtain preliminary patch embeddings. At the same time, in order to preserve spatial position information, learnable absolute position embeddings or relative position biases are added.

[0062] Next, the sequence of patch embeddings is input into a multi-layer Transformer encoder (e.g., 12 or 24 layers, each containing multi-head self-attention and a feed-forward network). Through the self-attention mechanism, each patch token can interact with all other patch tokens, thereby capturing the global scene context. The encoder output is the visual feature Ft,v∈RL×D of that frame image, which is a sequence containing L visual tokens, each with dimension D. This process is repeated for all time steps and all viewpoints, ultimately yielding the visual feature tensor F∈RNt×V×L×D for the entire historical image sequence. This tensor preserves the temporal order, viewpoint differences, and spatial structure of the original data and can be directly used for spatiotemporal compression or as input to subsequent prediction models.

[0063] Step A102: Spatiotemporal compression of visual features to obtain visual implicit features.

[0064] Specifically, visual implicit features refer to the compact, low-dimensional, and discriminative latent representation obtained by spatiotemporally compressing the extracted visual features (i.e., the structured visual token sequence output by the visual encoder). In the x-Foresight framework, visual implicit features are located between the visual backbone network and the diffusion renderer, and are the core hidden state of the world model: they compress the spatiotemporal redundancy in the original visual features (such as highly similar background textures and static objects between adjacent frames) while retaining key dynamic information driving environmental evolution (such as vehicle motion, pedestrian trajectories, and vehicle posture changes) and the geometric topology of the scene. This feature is an abstract representation oriented towards reasoning, avoiding excessive retention of pixel-level high-frequency details, thereby forcing the model to learn higher-level physical causal relationships and temporal dynamic laws, rather than simply extrapolating pixel changes between adjacent frames. The visual implicit features are finally input into the Vision Renderer (a diffusion-based rendering module) to reconstruct high-fidelity future multi-view images conditioned on historical images.

[0065] This step is specifically implemented based on the Feature Alignment Encoder architecture, which employs a Transformer-based autoencoder framework specifically designed for efficient spatiotemporal compression of multi-frame continuous visual features. Its implementation process is as follows: Before feeding the visual features into the Transformer encoder, a 3D convolutional layer is first used to efficiently downsample the temporal dimension. Specifically, the visual features are reshaped into a tensor of shape [Nt, V×L, D], and then treated as a pseudo-video sequence, where the temporal dimension is Nt, the number of channels is V×L, and the feature dimension D can be considered as the number of channels. The visual features (multiple views) at each time step are treated as a holistic feature map (e.g., stitching multiple views into a high-dimensional feature map, or using 3D convolution to operate in the temporal-spatial-channel three dimensions). In a typical implementation, a 3D convolutional kernel of size kt×1×1 (i.e., sliding along the temporal dimension, without downsampling in the spatial and view dimensions) with a stride of st can be set, compressing the temporal length from Nt to Nt′=(Nt The operation involves setting kt) / st+1, while keeping the number of output channels unchanged. This effectively reduces the number of time steps involved in subsequent attention calculations, thus reducing computational complexity.

[0066] The feature sequence after 3D convolution (length Nt′, feature dimension D at each time step) is input into the Single Attention module. This module is a variant of the standard Transformer encoder block and includes: A single-head self-attention layer: Compared to multi-head attention, single-head self-attention significantly reduces the number of parameters while maintaining the global receptive field, making it suitable for compression tasks. This layer computes pairwise association weights between features at all time steps, enabling the model to capture long-distance temporal dependencies (e.g., the causal relationship between the brake lights of the car in front illuminating and the deceleration of the car several frames later). The attention output passes through residual connections (one of the double residual connections). A feedforward network: Consists of two linear layers and an activation function (such as GELU), which independently performs non-linear transformations on the features at each time step, enhancing expressive power. It also incorporates residual connections (the second of the double residual connections) and layer normalization (LayerNorm).

[0067] The Transformer encoder blocks described above can be stacked M times (e.g., M=2 or 3), with information exchange and compression occurring in both the temporal and feature dimensions each time. The final output feature sequence Z∈RNt′×(V×L)×D, or a more compact representation obtained after global average pooling / temporal concatenation, i.e., visual implicit features, is obtained. The main compression occurs in the temporal dimension, while the spatial dimension remains intact, ensuring that the visual implicit features still contain the semantic information of each patch.

[0068] After the spatiotemporal compression described above, the resulting tensor Z is the visual implicit feature. In practical engineering implementation, one can choose to: retain the entire compressed time series (Nt′ time steps), with the feature shape of each time step being [V×L,D], for use in subsequent autoregressive predictions within chunks; or concatenate the features of all time steps into a long vector along the time dimension, and then map it to a fixed-dimensional latent variable through a linear projection layer, serving as the global spatiotemporal representation of the entire historical sequence.

[0069] According to x-Foresight's design, visual implicit features will be used for two downstream tasks: as input to the Vision Renderer (diffusion renderer), combined with historical images to generate high-fidelity future multi-view images; and together with driving features, target sub-tasks, and guidance information, they will form the first baseline features for block-by-block prediction in subsequent time blocks.

[0070] Step A103: Generate driving features based on real-time motion parameters and obtain guidance information associated with the first time block. The guidance information is used to guide the target prediction model to predict the image sequence and motion parameters of the first time block based on visual features, visual implicit features, and driving features.

[0071] Specifically, driving features refer to those extracted from the real-time motion parameters of a mobile vehicle, used to describe the vehicle's operating state and control intentions. Real-time motion parameters (such as speed, acceleration, yaw rate, steering wheel angle, etc.) are encoded to form driving features, reflecting the vehicle's "instantaneous action tendency" at the current moment and serving as a crucial basis for subsequent predictive models to regress future motion parameters (such as trajectory). Driving features complement visual features and visual implicit features: visual features describe the environment, while driving features describe the operating state and control intentions.

[0072] Guide information refers to a learnable or fixed conditional vector used to guide the fusion of multimodal information and the direction of future predictions in a target prediction model. Similar to query tokens or task embeddings in a multimodal prompt, guide information helps the model perform causal reasoning across different time blocks and determine which visual and motion information should be prioritized in the current prediction step. Guide information can be dynamically generated based on the index of the current time block, prediction instructions that trigger future variables, the semantics of the target subtask, and historical context, or selected from a fixed codebook.

[0073] On one hand, the acquired real-time motion parameters are typically represented as vectors p∈RK, where K is the parameter dimension. Real-time motion parameters include: longitudinal velocity vx, lateral velocity vy, yaw rate ω, acceleration ax, ay, steering wheel angle δ, etc. Then, a trainable fully connected layer (Linear layer) is used to map p to the same dimension D as the visual features, resulting in the driving feature fdyn=Linear(p)∈RD. Alternatively, if the goal is to capture the temporal trend of motion parameters, a short sequence of motion parameters from the current moment and the previous few moments can be formed and fed into a small multilayer perceptron (MLP) or gated recurrent unit (GRU) network to output the driving feature at the current moment.

[0074] On the other hand, the method of obtaining the guiding information g1 (corresponding to the first time block) differs depending on the training / inference phase: In the inference phase: If the first time block is the starting time block of the current prediction, the guidance information can be set to a fixed starting token (such as a learnable embedding vector [START_CHUNK]), or based on the query token used to trigger the prediction of future variables, the text token used to specify the prediction time range, the time block index (obtained through a text encoder), the system prompt encoding general instructions, and high-level information about vehicle status and long-term navigation targets, the text description can be input into a lightweight text encoder (such as the last layer of a bidirectional encoder or a simple word embedding average) and output a semantic vector as guidance information. To simplify the implementation, a learnable parameter matrix can also be used directly, which can be obtained by querying the time block index: g1=EmbeddingTable[1], where 1 is the index of the first time block.

[0075] During the training phase: the guiding information can be part of the true future features under teacher-forcing or generated from the output latent state of the model's previous time block. However, in the first time block, since there is no preceding block, the guiding information is usually directly set as the learnable basic embedding.

[0076] Ultimately, the dimension of the guiding information g1 is consistent with that of the visual and driving features (e.g., D=768) to facilitate subsequent splicing or cross-attention in the model. This guiding information will be input into the target prediction model along with other features, helping the model understand which time block is being predicted and what dynamic patterns should be focused on.

[0077] Step A104: Visual features, visual implicit features, driving features, target sub-tasks, and guidance information are used as the first baseline features.

[0078] Specifically, the first baseline feature refers to a multimodal feature set constructed for the first time block (the first time block), used to uniformly represent all the contextual information required for prediction in that time block. It is not a single vector, but a "feature" containing five types of data: visual features, visual implicit features, driving features, target subtasks, and guidance information. In x-Foresight's long-term temporal block autoregressive strategy, the first baseline feature serves as the initial condition for autoregressive prediction, input into the model to generate the future image sequence and motion parameters for the first time block. The baseline features for subsequent time blocks are derived from the previous baseline feature in an autoregressive manner.

[0079] The implementation method is to organize and package all the features generated in the previous steps to form a structured first baseline feature. The specific operation is as follows: First, obtain the visual feature Fvis (four-dimensional tensor or list form), the visual implicit feature Zlatent (compressed spatiotemporal representation), the driving feature fdyn (vector) and the guidance information g1, as well as the target subtask Taskgoal (text or embedded vector).

[0080] The first baseline feature does not concatenate all features into a single long vector, but rather preserves their individual tensor / vector structures for use in different downstream modules. For example, the visual feature Fvis retains its [time step, viewpoint, patch number, dimension] structure for use in the cross-attention of the target prediction model. The visual implicit feature Zlatent retains its compressed sequence structure (time step reduction) and is specifically input to the Vision Renderer. The driving feature fdyn and guidance information g1 are presented as one-dimensional vectors, concatenated with other features, and input into the input layer of the visual language model. Target subtasks are typically converted into text token sequences or single task embedding vectors. To distinguish different types of information within the model, type embeddings can be attached to each feature. For example, type ID 0 is assigned to visual features, type ID 1 to visual implicit features, type ID 2 to driving features, type ID 3 to guidance information, and type ID 4 to target subtasks. These type embeddings are added to or concatenated with the original features before being fed into subsequent networks.

[0081] All the above features are encapsulated into a dictionary or tuple {Fvis, Zlatent, fdyn, g1, Taskgoal}. In this baseline feature set, Zlatent is fed into the Vision Renderer to generate future images, fdyn is fed into the state decoder to generate motion parameters, while Fvis, g1, and Taskgoal serve as context for the target prediction module to assist in prediction (e.g., generating baseline features for the next time block through autoregression).

[0082] In one embodiment of this application, the method for predicting the image sequence and motion parameters of a mobile vehicle in a target time block other than a first time block includes: Step B1: Determine the preceding reference feature corresponding to the preceding time block; wherein, the preceding time block is the time block before the target time block; if the preceding time block is the first time block, then the preceding reference feature is the first reference feature; if the preceding time block is any time block after the first time block, then the preceding reference feature is the reference feature corresponding to that arbitrary time block.

[0083] Specifically, a preceding time block refers to the time block that precedes the current target time block in temporal order. According to x-Foresight's LARCH (Long-Term Block Autoregressive) strategy, the first time period (e.g., the next 6 seconds) is divided into multiple consecutive time blocks with temporal relationships, such as Chunk1, Chunk2, and Chunk3. For any time block other than the first one (i.e., Chunk2 or Chunk3), the immediately preceding time block is the "preceding time block." For example, when the target time block is Chunk2, the preceding time block is Chunk1; when the target time block is Chunk3, the preceding time block is Chunk2. The existence of preceding time blocks allows the model to utilize information from the previously predicted block to recursively predict subsequent blocks in an autoregressive manner, thereby learning long-term temporal causal relationships across blocks.

[0084] Preceding baseline features refer to the baseline features associated with the "preceding time block". A baseline feature is a multimodal feature set containing visual features, visual implicit features, driving features, target subtasks, and guidance information, used to uniformly represent all the context required for prediction of a given time block. Specifically, if the preceding time block is the first time block (Chunk1), then the preceding baseline feature is the first baseline feature; if the preceding time block is any time block after the first time block (e.g., Chunk2), then the preceding baseline feature is the baseline feature corresponding to that time block (generated from the previous round of autoregressive prediction). Preceding baseline features provide the latest state in the "historical causal chain" for predicting the target time block and are the core carrier for the LARCH strategy to achieve autoregressive recursion.

[0085] Since the entire prediction process is executed sequentially according to the time block order, the prediction results (including their baseline features) of the previous time block should already exist in memory or cache when this step is executed. The specific implementation is as follows: Let the target time block to be predicted be the i-th block (i≥2), and its previous time block index be i. 1. The system first obtains the sequence number i of the target time block from the control logic. During the model inference process, a dictionary or list cache is maintained to store the generated baseline features by time block index. For example, cache[1] stores the first baseline feature, cache[2] stores the second baseline feature, and so on.

[0086] For i=2, the preceding time block index is 1, so cache[1] is directly read as the preceding baseline feature. For i=3, the preceding time block index is 2, so cache[2] is read. The specific content of the preceding baseline feature: Regardless of which time block the preceding baseline feature comes from, its data structure is completely consistent with the first baseline feature, and it contains the following five parts: visual feature Fvis, the original visual token sequence extracted from the (predicted or real) image sequence of the time block; visual implicit feature Zlatent, the compact causal representation obtained after spatiotemporal compression of the visual feature; driving feature fdyn: the dynamic state encoded from the motion parameters (predicted or real) of the time block; target subtask Taskgoal: the global subtask that runs through the entire first time period; guidance information gi, the learnable or computed conditional vector corresponding to the i-th block.

[0087] In the prediction phase (inference), the visual and driving features in the preceding time block are typically derived from the model's previous prediction output (i.e., encoded predicted image sequences and predicted motion parameters), rather than real data. This enables closed-loop autoregression, allowing the model to continue inferring the future based on its own assumptions. If the target time block index is 2 and the preceding time block is 1, then the preceding baseline feature is the first baseline feature. It's important to note that the visual features in the first baseline feature are extracted from real historical image sequences (because no predictions were made before the first time block), while the driving features are generated from real-time motion parameters at the current moment. This design provides a solid initial condition for subsequent autoregression. The acquired preceding baseline features are then used to generate the target baseline features corresponding to the target time block.

[0088] Through the above implementation, an autoregressive chain between consecutive time blocks is successfully established, enabling x-Foresight to extrapolate future environmental states block by block and learn the true physical causal relationships in cross-block predictions, rather than simply extrapolating from adjacent frames.

[0089] Step B2: Based on the prior reference features, predict the target reference features corresponding to the target time block.

[0090] Specifically, based on the preceding baseline features, the target baseline features corresponding to the target time block are predicted, including: Step B201: Using the target sub-task in the preceding reference features as the basis for predicting the mobile vehicle in the target time block, the visual features in the preceding reference features are used to predict the target visual features of the mobile vehicle in the target time block, and the driving features in the preceding reference features are used to predict the target driving features of the vehicle in the target time block.

[0091] Specifically, target visual features refer to the abstract visual representation of the environmental state within a target time block (e.g., the second or third time block) predicted by a visual language model based on visual features from preceding baseline features and contextual information such as target sub-tasks. Unlike visual features extracted from real historical images, target visual features are compact semantic encodings of future visual information generated by the model. They contain a high-level understanding of dynamic elements such as the movement of traffic participants, scene geometric changes, and the expected trajectory of vehicles within the target time block, but do not contain pixel-level details. Target visual features are spatiotemporally compressed and ultimately used to drive the Vision Renderer to generate high-fidelity future images. Target driving features refer to the future dynamic state representation of a vehicle within a target time block predicted by a visual language model based on driving features from preceding baseline features and information such as target sub-tasks. Unlike "driving features" encoded from real-time motion parameters, target driving features are compact abstract encodings of future motion parameters, rather than the current real state. It implies the vehicle's expected control intentions, such as speed, acceleration, and steering, within a future time block, which will be input into the state decoder to generate a specific sequence of motion parameters (e.g., trajectory points).

[0092] This step is the core of autoregressive prediction in the LARCH strategy. Its goal is to predict the abstract visual and driving features of the target time block based on the preceding baseline features (i.e., the state of the previous time block) through causal inference from a visual language model. The specific implementation process is as follows: First, the following components are extracted from the preceding baseline features: preceding visual features Fvis(i) 1): The visual token sequence of the previous time block (if i If 1=1, then it represents a true historical characteristic; if i If 1>1, then it is the visual feature predicted in the previous step). Previous driving feature fdyn(i) 1): Dynamic encoding of the previous time block (if i If 1=1, it represents the real-time motion parameter encoding at the current moment; otherwise, it represents the driving characteristics predicted in the previous step. Target subtask (Taskgoal): A globally invariant high-level navigation target (e.g., "change lanes to the left"). Pre-guidance information (gi) 1: The condition vector corresponding to the previous time block.

[0093] x-Foresight's core visual language model employs a decoder-only architecture and receives the aforementioned input via a multimodal prompt. During inference, predictions for the target time block are generated as follows: The modal information from the preceding baseline features is concatenated sequentially into a token sequence. A typical structure is: [system_prompt], [task_goal], [guidance_{i-1}], [vis_tokens_{i-1}], [dyn_token_{i-1}], [query], where: system_prompt encodes a fixed instruction (e.g., "predict the visual and motion state of the next time block"); task_goal is a text embedding representing the target subtask; guidance_{i-1} is a learnable or computationally achievable guidance vector; vis_tokens_{i-1} are preceding visual features (tokens of sequence length L); dyn_token_{i-1} is preceding motion features (a single vector, which can be considered a special token); and query is a learnable placeholder token used to trigger the visual language model to output the predicted embedding for the corresponding target time block.

[0094] The visual language model uses the above input sequence as context and models cross-modal dependencies through a self-attention mechanism. After multi-layer Transformer decoding, a new embedding vector `hout∈RD` is obtained at the output position corresponding to the query. This embedding vector integrates the preceding state, the task objective, and the physical causal laws learned by the model. The output embedding `hout` is fed into two parallel lightweight prediction heads (linear layers): Visual prediction head: maps `hout` to a sequence output, reshaping it into the target visual feature `Fvis(i)`, whose dimension is the same as the preceding visual feature (e.g., L×DL×D). This feature represents an abstract description of the environmental state within the target time block predicted by the model. Driving prediction head: maps `hout` to a fixed-length vector, obtaining the target driving feature `fdyn(i)∈RD`, representing a compact encoding of the vehicle's future dynamics.

[0095] It's important to note that because the visual language model learns short-term (intra-chunk) predictions during training and then expands to long-term (cross-chunk) predictions, it can effectively learn the true causal relationships from preceding blocks to target blocks, rather than simple inter-frame extrapolation. For example, if a vehicle intrudes in the preceding block, the model can infer the visual and motion changes that the vehicle in the target block should slow down or avoid it. The target subtask is always present as a global condition, ensuring the model maintains consistency in behavioral intent when predicting different time blocks. For example, if the target subtask is "change lanes to the right," the predicted target driving features will tend towards lateral movement to the right, increasing the feature weight of the right lane region in the target visual features. The output target visual and driving features here are abstract representations, excluding high-frequency details. High-fidelity images will be generated by the subsequent Vision Renderer based on the compressed target visual implicit features, while driving features will be decoded into specific trajectories by the state decoder. This hierarchical design avoids the visual language model directly learning complex pixel distributions, allowing it to focus on causal reasoning.

[0096] Step B202 involves spatiotemporal compression of the target visual features to obtain the target visual implicit features, and acquisition of the target guidance information corresponding to the target time block. The target guidance information is used to guide the target prediction model to predict the image sequence and motion parameters of the target time block based on the target visual features, target visual implicit features, and target driving features.

[0097] Specifically, target visual implicit features refer to compact, low-dimensional, and causal latent representations obtained by spatiotemporally compressing the predicted target visual features. Similar to visual implicit features, target visual implicit features are further compressed by the FeatureAlignment Encoder to remove spatiotemporal redundancy (such as static backgrounds and repetitive textures between adjacent frames) from the original visual features, retaining only key dynamic information driving the evolution of future scenes (such as object motion trends, expected vehicle trajectories, and interactions between traffic participants). This feature is designed as an abstract representation oriented towards reasoning, without pixel-level details, and will serve as the core input condition for the Vision Renderer to generate high-fidelity future images of target time blocks. Because this feature originates from the future visual features predicted by the model, it is called target visual implicit features, distinguishing it from implicit features obtained by compressing real historical images.

[0098] Target guidance information refers to the conditional vector corresponding to the "target time block," used to guide the model to focus on specific information needed for the current prediction block during multimodal inference in visual language models. Guidance information can be a learnable embedding vector (looked up from the code table by time block index) or dynamically generated based on the target subtask, prior guidance information, prediction time offset, etc. Its role is to enable the model to distinguish prediction tasks for different time blocks and to help the visual language model maintain temporal consistency during autoregression, avoiding prediction confusion between different blocks.

[0099] The specific implementation involves obtaining the target's visual implicit features through spatiotemporal compression, and acquiring target guidance information. Both together provide necessary components for the subsequent construction of target baseline features.

[0100] On one hand, the target visual features are treated as a spatiotemporal sequence. First, a 3D convolutional layer is used to downsample the temporal dimension. The convolutional kernel size is set to kt×1×1 (sliding only in the temporal dimension), and the stride is st, compressing the number of time steps from Nt(i) to Nt′(i) = (Nt(i)). kt) / st+1. This operation reduces the computational cost of subsequent attention without changing the spatial and viewpoint resolution, while forcing the model to focus on coarse-grained temporal changes. The downsampled feature sequence is input into the Transformer encoder block, which consists of a Single Attention layer and a Feedforward Network (FFN). This module performs joint compression of the temporal and feature dimensions: the Self Attention layer calculates the global dependencies between features at each time step, capturing dynamic patterns across frames (e.g., the process of a vehicle appearing and approaching); the FFN performs independent nonlinear transformations for each time step to enhance expressiveness; residual connections and layer normalization stabilize gradient propagation. This module can be stacked M times (e.g., M=2), and finally outputs a compressed feature sequence with the shape [Nt′(i), V×L, D]. The compression result is denoted as Zlatent(i), which is the target visual implicit feature. This feature retains the key dynamic information driving scene evolution but removes high-frequency visual details and spatiotemporal redundancy. It will be stored in the target baseline feature and then input into the Vision Renderer to generate pixel-level future images.

[0101] On the other hand, the acquisition of target guidance information gi (corresponding to the i-th time block) adopts one of the following strategies during the inference phase: Method 1: The system maintains a learnable embedding matrix E∈RNchunks×D, where Nchunks is the total number of time blocks divided within the first time duration. For the target time block index i (i≥2), gi=E[i] can be obtained directly by querying.

[0102] Method 2: Use a small multilayer perceptron (MLP) with pre-guided information gi 1. Using the target driving feature fdyn(i) as input, predict the current guidance information: gi = MLP([gi 1;fdyn(i)]).

[0103] Method 3: Combine the text embedding of the target subtask with the current time block index, for example: gi=Linear(TextEmb(Taskgoal)+PosEmb(i)), where PosEmb is a sinusoidal positional code or a learnable positional embedding.

[0104] The final output consists of the target visual latent feature Zlatent(i) and the target guidance information gi. These two quantities, together with the target visual feature Fvis(i), the target driving feature fdyn(i), and the globally invariant target subtask Taskgoal, are combined to form the complete target baseline features, which are used to predict the image sequence and motion parameters of the target time block.

[0105] Step B203: Use the target visual features, target visual implicit features, target driving features, target sub-tasks, and target guidance information as the target baseline features.

[0106] Specifically, the target baseline feature refers to a comprehensive set of multimodal features constructed for the current "target time block" to be predicted (such as the second or third time block), used to uniformly represent all the contextual information required for the prediction of that time block. It has the exact same structure as the first baseline feature, but the data sources differ: the first baseline feature is generated based on real-time historical observations and motion parameters, while the target baseline feature is a combination of future features predicted by a visual language model (including target visual features, target implicit visual features, target driving features), and target guidance information. The target baseline feature is the core carrier for the autoregressive recursion of the LARCH (Long-Term Block Autoregressive) strategy in x-Foresight: once constructed, it will be directly used to predict future image sequences and motion parameters of the target time block. Simultaneously, this target baseline feature will also be cached as a preceding baseline feature for subsequent, more distant time blocks, thus achieving closed-loop prediction across blocks.

[0107] The specific implementation is as follows: Obtain the target visual features Fvis(i), target visual implicit features Zlatent(i), target driving features fdyn(i), target sub-task Taskgoal, and target guidance information gi. The target baseline features do not force all components to be concatenated into a single ultra-long vector; instead, they retain their respective tensor / vector structures for later use by different modules as needed. For example, visual features and visual implicit features retain their spatiotemporal dimensions for use by the VisionRenderer and the visual language model's cross-attention mechanism. Driving features, guidance information, and target sub-tasks are maintained as one-dimensional vectors or embeddings, facilitating direct input to the state decoder or as part of the prompt. To facilitate the differentiation of modal information in the visual language model, a learnable type embedding can be attached to each feature. Before actual input to the visual language model or renderer, the type embedding is added to or concatenated with the corresponding feature. However, this step only handles packaging; the type embedding can be dynamically added later. The above five components are encapsulated into a dictionary or tuple structure, for example: {Fvis(i), Zlatent(i), fdyn(i), Taskgoal, gi}. In the code implementation, namedtuples, dataclasses, or dictionaries can be used for storage to ensure that each field can be directly indexed and accessed in subsequent steps. After generating the target baseline features, they are stored in a cache dictionary cache[i] organized by time block index. In this way, when the model needs to predict the next time block (i+1), it can directly retrieve the previous baseline features through cache[i] to form a complete autoregressive chain.

[0108] It should be noted that although the target baseline feature is derived from predicted features rather than real data, its data structure and field meanings are completely consistent with the first baseline feature. This unified design allows the processing of the first block and subsequent blocks to share the same prediction logic (e.g., the same renderer calls and state decoder calls), only needing to determine whether teacher-mandated training techniques are required based on the source of the baseline feature (real or predicted). During the pure inference phase, both are processed in exactly the same way.

[0109] Through the above implementation, a complete baseline feature corresponding to the target time block was successfully constructed, which prepared for generating future image sequences and motion parameters of the time block, and also provided a cache foundation for autoregressive prediction of longer time blocks.

[0110] Step B3: Based on the target baseline features, predict the target prediction image sequence and target prediction motion parameters associated with the target time block.

[0111] Specifically, the target prediction image sequence refers to a high-fidelity, multi-view future image set generated by the VisionRenderer for the current "target time block" to be predicted (e.g., the second or third time block). This set is based on the target's implicit visual features from the target baseline features and preconditions (usually historical or predicted images from the previous time block). This sequence contains dense image frames captured by multiple surround-view cameras within the target time block; it can be understood as image frames captured at a high sampling frequency with small time spans between frames within the time block. Compared to the first prediction image sequence, the target prediction image sequence is a prediction of the more distant future. Its generation relies on abstract features derived from the model's autoregression, rather than directly using real historical images as conditions (although the renderer's conditions will include real or predicted images from the previous time block).

[0112] These images will be used for closed-loop feedback (input to the visual encoder to predict subsequent time blocks) or as part of the driving scene prediction result in the final output. Target predicted motion parameters refer to the specific future motion parameters (e.g., discrete trajectory points of the vehicle within the next 2 seconds, expected speed sequence, or steering wheel angle sequence) decoded by the state decoder based on the target driving features in the target baseline features for a target time block. This parameter is the control basis for the final output of the x-Foresight model and is completely consistent with the first predicted motion parameters in form and data dimension, differing only in the corresponding time interval (e.g., 0-2 seconds for the first block, 2-4 seconds for the second block). Target predicted motion parameters can be output along with the target predicted image sequence for use by downstream planning or control modules and participate in autoregression as pre-encoded driving features for the prediction of the next time block.

[0113] This step is logically symmetrical to the prediction in the first time block, the only difference being that the input baseline features come from the prediction (target baseline features) rather than from the actual observations. The specific process is as follows: First, extract the following from the target baseline features: target visual implicit feature Zlatent(i) and preceding image conditions: For the target time block (i≥2), the visual renderer needs a historical image as a texture reference. The renderer is conditional on the historical image of a short time window and the camera token predicted by LLM. Here, the historical image refers to the predicted image sequence of the previous time block (if i ≥ 2). If 1 is the first time block, then it is the real historical image sequence; otherwise, it is the predicted image sequence output from the previous round. The system retrieves the last few frames (e.g., the last 1-2 frames) of the previous time block from the cache as conditional images to ensure visual continuity.

[0114] Secondly, the target visual implicit feature Zlatent(i) and the preceding conditional image are jointly input into the diffusion model-based VisionRenderer. The renderer uses the preceding image as initial noise or conditional injection, and gradually generates a high-fidelity panoramic image for each frame within the target time block through a denoising network. This renderer is also not conditional on action tokens, but only relies on camera tokens (target visual implicit features) and preceding images to ensure that it faithfully decodes the future content predicted by LLM. The output tensor shape is [K,V,C,H,W], where K is the number of frames in the target time block, V=7, C=3, and H and W are the image resolutions. The image sequence generated by the renderer is denoted as Ipred,i, which is the target prediction image sequence. This sequence is used as follows: if there are subsequent time blocks, it is fed back to the visual encoder as the preceding historical image for the next time block. If the current time block is the last time block, it is collected into the final driving scene prediction result.

[0115] Additionally, the target driving feature fdyn(i) is extracted from the target baseline features. A state decoder (lightweight MLP) is used to regress the target driving feature, and the output dimension is consistent with the first predicted motion parameter, such as trajectory points (e.g., M points, each containing x and y displacements) or frame-by-frame control quantities within the target time block. The output of the state decoder is denoted as Ppred,i, which is the target predicted motion parameter. This parameter can be used by the downstream control module and is also stored in the cache for driving feature prediction in subsequent time blocks.

[0116] After generation, Ipred,i and Ppred,i are temporarily stored. If a next time block exists (i+1≤Nchunks), then: Ipred,i is fed into the visual encoder to extract the visual features of the target time block. Ppred,i is converted into the preceding driving features of the next time block through a lightweight encoding network (or directly as such). If i is the last time block, the accumulated predicted image sequences and motion parameters of all blocks are summarized as the final driving scene prediction result output.

[0117] Through the above implementation, x-Foresight can predict multi-view video and vehicle trajectories up to 6 seconds (or longer) in the future in an autoregressive manner, and learn long-term temporal causality in cross-block prediction using the LARCH strategy, while maintaining real-time response capability.

[0118] This embodiment provides a method for predicting driving scenarios. Figure 4 This is a flowchart of a driving scenario prediction method according to an embodiment of the present invention, such as... Figure 4 As shown, the process includes the following steps: Step S201: Obtain the subtasks to be executed within a first duration after the current moment from the driving task of the mobile vehicle as the target subtasks; wherein, the driving task includes at least two subtasks with a temporal relationship; the first duration includes at least two time blocks with a temporal relationship. For details of the specific implementation process, please refer to step S101 in the above embodiment, which will not be repeated here.

[0119] Step S202: Obtain the real-time motion parameters of the mobile vehicle at the current moment, and the historical image sequence collected within a second time period prior to the current moment. For details of the implementation process, please refer to step S102 in the above embodiment, which will not be repeated here.

[0120] Step S203: Input the historical image sequence, real-time motion parameters, and target sub-task into the pre-trained target prediction model, so that the target prediction model analyzes the historical image sequence and real-time motion parameters according to the target sub-task, and outputs the image sequence and motion parameters associated with each time block in the first time period after the current moment to predict the driving scene prediction result.

[0121] Specifically, the pre-trained target prediction model refers to an end-to-end neural network model pre-trained with a large amount of driving data to predict driving scenarios, i.e., the x-Foresight overall architecture. This model consists of three main parts: Visual language model: responsible for multimodal information understanding and causal reasoning, it encodes inputs such as historical image sequences, real-time motion parameters, and target sub-tasks into a unified semantic representation, and predicts the abstract visual features and driving features of future time blocks through an autoregressive approach.

[0122] VisionRenderer: Based on the diffusion model, it generates high-fidelity multi-view future image sequences by using the visual implicit features output by the visual language model and historical images as conditions.

[0123] StateDecoder: A lightweight regression network that decodes the driving features output by the visual language model into specific future motion parameters (such as trajectory points, speed, etc.).

[0124] This model employs LARCH (Long Time Domain Block Autoregressive Strategy) for training and inference, dividing the first duration of the future (e.g., 6 seconds) into multiple time chunks and predicting the corresponding multi-view image sequences and vehicle motion parameters for each chunk. The model first masters short-time domain prediction through a learning strategy, then extends to long-time domain prediction. Simultaneously, it utilizes a hybrid sampling strategy to focus on safety-critical scenarios, thus maintaining real-time performance while possessing forward-looking decision-making capabilities.

[0125] The specific implementation is as follows: Obtain the target subtask and encode the target subtask (e.g., "change lanes to the left") into a text embedding vector or a discrete token sequence. Obtain the historical image sequence (multi-view panoramic images within the second time period before the current moment, such as 3 seconds × 15fps = 45 frames) and real-time motion parameters (such as current speed, acceleration, yaw rate, etc., encoded into a fixed-length feature vector).

[0126] The three types of inputs are organized according to the x-Foresight multimodal prompt design. Typically, this forms the following token sequence: [system_prompt][task_goal][history_vis_tokens][current_dyn_token][query], where: system_prompt is a fixed instruction embedding (e.g., "predict the image and trajectory of each time block within the next 6 seconds"); task_goal is the embedding of the target subtask; history_vis_tokens is a sequence of visual feature tokens extracted from historical image sequences by a visual encoder; current_dyn_token is a driving feature token obtained by linear mapping of real-time motion parameters; and query is a learnable placeholder token used to trigger the model's output for the first time block.

[0127] Forward inference of the target prediction model: First, the above input sequence is fed into a pre-trained visual language model (a Transformer-based decoder architecture). The visual language model understands the causal relationship between the historical environment, the current state, and the task target through a self-attention mechanism. According to the LARCH strategy, the model does not output all future frames at once, but adopts block autoregression: it first predicts the abstract visual and driving features of the first time block (e.g., 0-2 seconds), and then uses the output of the previous block as a condition to progressively predict subsequent time blocks (2-4 seconds, 4-6 seconds). Dense image frames are retained within each time block to capture instantaneous dynamics, and long-term temporal causal relationships are learned across blocks. Second, for each time block, the output embedding of the visual language model is fed into two task heads: the visual head outputs the target visual features (compact spatiotemporal representation) of the time block, and the motion head outputs the target driving features (dynamic intent encoding) of the time block. Subsequently, the target visual features of each time block are compressed into visual implicit features by the Feature Alignment Encoder and input into the visual renderer (diffusion model). The renderer generates a high-fidelity sequence of future images from multiple perspectives, conditioned on the implicit feature and the historical image of the previous time block (the first time block uses a real historical image). The target driving features of each time block are directly input into the state decoder, which outputs specific motion parameters (e.g., a sequence of trajectory points).

[0128] For the first time block: historical images and real-time motion parameters are directly used as conditions to obtain the predicted image sequence and motion parameters for the first block. For the second and third time blocks: the predicted image sequence (after ViT encoding) and predicted motion parameters (after encoding) from the previous block are used as new historical conditions and input into the visual language model again along with the target subtask, repeating the above process. The prediction results of all time blocks are sequentially concatenated to form a complete future image sequence and continuous motion trajectory covering the entire first time duration.

[0129] Finally, the model outputs a structured prediction result, which includes: a multi-view surround view image sequence corresponding to each time block (e.g., 7 views, K frames per block, N time blocks in total); and vehicle motion parameters corresponding to each time block (e.g., trajectory points at equal time intervals within the next 6 seconds). This result is the driving scene prediction result, which can be directly used by downstream planning, control, or safety warning modules.

[0130] Through the above implementation, the pre-trained target prediction model can combine historical observations, real-time status, and task objectives to efficiently generate high-quality future scene predictions in a block autoregressive manner, while maintaining real-time response capabilities.

[0131] In one embodiment of this application, before inputting historical image sequences, real-time motion parameters, and subtasks into a pre-trained target prediction model, such as... Figure 5 As shown, the method also includes: Step S301: Obtain training data sequence, wherein the training data sequence includes at least one set of samples, the set of samples includes image sequence samples and corresponding motion parameter samples and subtask samples.

[0132] Specifically, the training data sequence refers to a large-scale multimodal dataset used to train the target prediction model (x-Foresight), consisting of multiple sample sets arranged in chronological order. Each sample set corresponds to a continuous time segment (e.g., a driving process lasting several seconds to tens of seconds), containing complete environmental perception information and vehicle status information within that time period. The training data sequence is constructed using a hybrid sampling strategy, ensuring both broad temporal coverage and a higher proportion of samples representing safety-critical scenarios (such as sudden braking, cutting in, and pedestrian crossings), thereby improving the model's ability to learn dangerous dynamics.

[0133] A sample set refers to a basic unit in a training data sequence, corresponding to a complete data record of all moments within a continuous time segment. Each sample set contains three parts: image sequence samples (multi-view surround view images), motion parameter samples (vehicle trajectory or control variables), and subtask samples (high-level behavioral intentions). For example, a sample set may correspond to a 5-second real driving segment, which includes 7 camera images at each moment, motion parameters such as vehicle speed / acceleration / steering wheel angle, and the corresponding navigation instructions for that segment (such as "go straight through the intersection").

[0134] Image sequence samples refer to the visual portion of a sample set, specifically a sequence of raw image frames captured by multiple surround-view cameras on a moving vehicle over a continuous time period. This sequence covers the duration of real-world images and is typically sampled at a fixed frame rate (e.g., 15 fps). Image sequence samples are used to provide supervisory signals for environmental perception, including low-level visual details and high-level semantic information (vehicle and pedestrian motion patterns).

[0135] Motion parameter samples refer to the dynamic components of the sample set, i.e., the ground truth values ​​of the vehicle's motion state, time-aligned with the image sequence samples. These include longitudinal velocity, lateral velocity, yaw rate, absolute position (or relative displacement), steering wheel angle, acceleration, etc. These parameters serve as regression targets during training, used to supervise the future motion parameters output by the state decoder.

[0136] Subtask samples refer to high-level behavioral intent labels in a sample set, corresponding to the subtasks that the moving vehicle is expected to perform within that time segment (e.g., "change lanes," "stop and yield," "turn left"). Subtask samples can be obtained from navigation instructions, rules, or manual annotations. During training, subtask samples are used as conditional inputs to the visual language model, guiding the model to learn the scene evolution patterns under different intents.

[0137] The specific implementation is as follows: Obtain the constructed training data sequence. This sequence consists of multiple sample sets, each containing image sequence samples, motion parameter samples, and subtask samples. The training data sequence is arranged in chronological order, covering various driving scenarios (urban roads, highways, intersections, etc.) and subtasks of varying difficulty. The training data sequence is represented as a list D={(Ij,Pj,Tj)}j=1M, where M is the total number of sample sets, Ij is the image sequence sample of the j-th sample (including historical and future frames), Pj is the corresponding motion parameter sample, and Tj is the subtask sample (which can be a text description or a discrete ID).

[0138] During training, a batch of samples is randomly or selectively drawn from the training data sequence according to the course learning strategy and the hybrid sampling strategy. Each batch of samples is processed independently for subsequent training steps. Typically, image sequence samples need to be preprocessed (scaled to a fixed resolution, normalized), motion parameter samples are standardized, and subtask samples are encoded into embedding vectors.

[0139] Step S302: Based on the training data sequence, the duration of the first sample is divided into at least one first sample time block according to the first time interval in the short time domain prediction task, and the initial decision model is trained in the first stage. The initial decision model learns the correlation between the image sequence samples and the corresponding motion parameter samples of each first sample time block according to the sub-task samples until the first convergence condition is met, and a candidate decision model is obtained.

[0140] Specifically, the short temporal prediction task refers to the first stage of training objectives in the course. In this stage, the model only needs to learn to predict scene evolution within a short future time window, such as predicting image sequences and motion parameters within the next 1-2 seconds. By simplifying the task (short time span, small dynamic changes), the model can first master basic causal relationships and instantaneous dynamic capture capabilities, avoiding being overwhelmed by the complexity of long temporal prediction from the outset. Short temporal prediction corresponds to "intra-block dense image frame prediction" in the LARCH strategy, that is, focusing only on predictions within a single time block.

[0141] The first time interval refers to the fixed duration of each block when the duration of the first sample is divided into multiple first sample time blocks. For example, the first time interval can be set to 1 to 2 seconds (e.g., 2 seconds). This interval determines the number of dense image frames (frame rate × time interval) contained in the block, enabling the model to capture sufficiently fine-grained instantaneous dynamics.

[0142] The first sample duration refers to the total length of the future prediction window used for training in short-time-domain prediction tasks. This duration is typically short, such as 2 or 4 seconds, to allow the model to quickly learn reliable short-term predictions. The first sample duration can be equal to a first time interval (i.e., predicting only one block), or it can contain 2 to 3 blocks, but it is generally shorter than the first sample duration for subsequent long-time-domain tasks.

[0143] The first sample time block refers to the basic prediction unit divided according to the first time interval within the first sample duration. Each first sample time block contains consecutive frames of dense images (e.g., 30 frames within 2 seconds). In short temporal training, the model is trained to predict the image sequence and motion parameters of each time block sequentially (autoregression), but the total prediction length is relatively short.

[0144] The initial decision model refers to the original x-Foresight model (i.e., an untrained version of the target prediction model) that has not been trained or has only undergone basic pre-training. This model has a complete architecture (visual language model, VisionRenderer, FeatureAlignmentEncoder, state decoder), but its parameters are randomly initialized or loaded from the pre-trained basic visual language model. The initial decision model will be optimized during the first stage of training.

[0145] The first convergence criterion refers to the criteria for determining whether the training of the short-term prediction task is complete. These criteria may include, but are not limited to: the loss function (such as image reconstruction error or motion parameters) on the validation set not decreasing for several consecutive rounds; the prediction accuracy reaching a preset threshold (e.g., trajectory error < 0.2 meters in the next second); or the number of training rounds reaching a set upper limit. Once the first convergence criterion is met, the first stage of training stops, and the candidate decision model is obtained.

[0146] A candidate decision model refers to a model trained on a short-term prediction task. At this point, the model already possesses good short-term prediction capabilities, accurately predicting environmental changes and vehicle movement within the next 1-2 seconds. However, its stability in long-term (e.g., 6 seconds) predictions may be insufficient. The candidate decision model will serve as the starting point for the second stage of training, continuing to learn predictions for longer timeframes.

[0147] The first stage of training (the short-time domain phase of course learning) is implemented as follows: First, determine the duration of the first sample, Tshort (e.g., 4 seconds), and the first time interval, Δtchunk,short (e.g., 2 seconds). Calculate the number of time blocks for the first sample, Nchunks,short = Tshort / Δtchunk,short = 2. The number of frames within each time block is determined by the frame rate (e.g., 2 seconds corresponds to 60 frames at 30fps). Second, from the training data sequence, for each sample set, extract image sequence samples and motion parameter samples within its historical window and the future first sample duration, Tshort. Simultaneously, use sub-task samples as conditional input. If necessary, time alignment can be performed on each sample set to ensure clear start and end boundaries for each time block. Third, input the historical image sequences, real-time motion parameters (obtained from the last historical moment), and sub-task samples into the initial decision model. The model predicts the predicted image sequences and predicted motion parameters for Nchunks,short time blocks sequentially according to the LARCH strategy. Calculate separately: pixel-level reconstruction loss (e.g., L1 loss, perceptual loss, or noisy prediction loss of a diffusion model) between the predicted image output by the renderer and the true future image; and mean squared error (MSE) or smoothed L1 loss between the predicted motion parameters output by the state decoder and the true motion parameters. The total loss is a weighted sum of the two.

[0148] Backpropagation using an optimizer (such as AdamW) updates all trainable parameters of the initial decision model (including the visual language model, FeatureAlignmentEncoder, renderer, state decoder, and guided embeddings). During training, due to the short prediction window, the model converges easily; a small learning rate (e.g., 1e-4) and a large batch size (e.g., 32) can be set. After each epoch, the loss and metrics are evaluated on the validation set. The first convergence condition is met when the validation loss no longer decreases for five consecutive epochs, or reaches the preset maximum epoch (e.g., 100 epochs). At this point, training stops, the current model parameters are saved, and a candidate decision model is obtained. The candidate decision model is output for use in the second stage of training.

[0149] Step S303: Based on the training data sequence, the duration of the first sample is divided into at least one second sample time block according to the second time interval in the long-term prediction task, and the candidate decision model is trained in the second stage. The candidate decision model learns the correlation between the image sequence samples and the corresponding motion parameter samples of each second sample time block according to the sub-task samples until the second convergence condition is met, and the target prediction model is obtained.

[0150] Specifically, the long-term prediction task refers to the second-stage training objective of the course. In this stage, the model needs to learn to predict scene evolution over a longer time span (e.g., 6 seconds or more in the future). Compared to short-term tasks, long-term prediction requires the model to understand causal relationships across time blocks, avoid simple extrapolation, and handle accumulated errors. Long-term prediction corresponds to cross-block long-term prediction in the LARCH strategy, meaning the model needs to autoregressively generate content for multiple subsequent time blocks based on the prediction results of the previous time block.

[0151] The second time interval refers to the fixed duration of each block when the duration of the first sample (which is typically longer, such as 6 seconds, in long-time-domain tasks) is divided into multiple second sample time blocks. The second time interval can be the same as the first time interval (e.g., 2 seconds) or slightly longer. The difference between the second and first time intervals is that the model in the second stage needs to jointly optimize dense image prediction within a block and causal inference across blocks.

[0152] In long-term prediction tasks, a second sample time block is a basic unit that divides a relatively long first sample duration. Each second sample time block has the same structure as the first sample time block, but there are more of them (e.g., 6 seconds / 2 seconds = 3 blocks). During training, the model needs to start from historical inputs, autoregressively predict the content of all second sample time blocks, and obtain supervision signals from real data.

[0153] The second convergence criterion is a standard for determining whether training for a long-term prediction task is complete. It is usually more stringent than the first convergence criterion. For example, it may require that the long-term (6-second) trajectory prediction error be below a certain threshold (e.g., <1 meter), that the structural similarity of the future image be greater than a certain value, or that the verification loss no longer decreases over a longer period. Once the second convergence criterion is met, training ends, and the final target prediction model is obtained.

[0154] The target prediction model refers to the final model after a complete two-stage learning and training process, i.e., the pre-trained target prediction model. This model has the ability to capture instantaneous dynamics in the short time domain and to perform causal reasoning and prediction in the long time domain. It can provide forward-looking predictions of future scenarios for autonomous driving while maintaining real-time response performance.

[0155] The second stage of training (the long-term stage of course learning) is implemented as follows: First, determine the first sample duration Tlong (e.g., 6 seconds) and the second time interval Δtchunk,long (which can be equal to the first time interval, e.g., 2 seconds) used for long-term prediction. Then, calculate the number of second sample time blocks Nchunks,long = Tlong / Δtchunk,long (e.g., 3 blocks). In long-term tasks, the first sample duration refers to the total prediction window length, while the second time interval is the duration of each block.

[0156] Obtain the parameters of the candidate decision model as the initial weights for the second stage of training. The candidate decision model already possesses good short-term prediction capabilities, providing a solid foundation for long-term learning and avoiding training from scratch. From the training data sequence D, for each sample set, extract image sequence samples and motion parameter samples within the historical window (second duration) and the first sample duration Tlong. Note: The future portion must completely cover N chunks,long time blocks so that the model receives realistic supervision signals at each time step.

[0157] Forward training and loss calculation: Historical image sequences, real-time motion parameters (obtained from the last historical moment), and subtask samples are input into the candidate decision model. The model, following a LARCH strategy, autoregressively predicts all N chunks, long second sample time blocks sequentially. For each block, the model outputs a predicted image sequence and predicted motion parameters. During training, teacher forcing or scheduled sampling strategies can be used to mitigate the accumulation of autoregressive errors. Specifically, in early training rounds, more realistic previous block images and motion parameters are used as input for the next block; as training progresses, the proportion of using prediction results as input is gradually increased to adapt the model to the closed-loop conditions during inference. The total loss includes the weighted sum of image reconstruction loss and motion parameter regression loss for all blocks. For distant blocks (such as the 3rd block), the loss weight can be appropriately increased to emphasize the accuracy of long-term prediction.

[0158] Update all parameters of the candidate decision model using an optimizer (such as AdamW). Since the second-stage task is more difficult, the learning rate can be appropriately reduced (e.g., from 1e-4 to 5e-5), and the number of training epochs can be increased. Monitor long-term metrics on the validation set: such as the average displacement error over the next 6 seconds, the final displacement error, the peak signal-to-noise ratio of image frames, and structural similarity. The second convergence condition is met when these metrics no longer improve for several consecutive epochs, or when the preset maximum number of training epochs (e.g., 200 epochs) is reached.

[0159] After training is stopped, the final model parameters are saved to obtain the target prediction model. This model is the pre-trained model used, which can perform long-term block prediction on any input historical image sequence, real-time motion parameters and target sub-task during inference, and output the future image sequence and motion parameters for each time block.

[0160] Through a two-stage course, students can master short-term dense image prediction and long-term causal inference in the x-Foresight model, progressing from easy to difficult, significantly improving prediction stability and generalization ability.

[0161] In one embodiment of this application, before obtaining the training data sequence, the method further includes: Step C1: Obtain the original training data sequence, wherein the original training data sequence includes a set of original samples from multiple consecutive time segments, and each set of original samples includes original image sequence samples and corresponding original motion parameter samples and original subtask samples.

[0162] Specifically, the raw training data sequence refers to a continuous stream of raw data directly collected from real autonomous vehicles or simulation platforms without any sampling processing. This sequence consists of a large number of consecutive time segments, each containing high-frame-rate (e.g., 30fps) multi-view surround-view images, synchronized vehicle motion parameters, and corresponding subtask labels. Raw training data sequences are typically large in scale and highly redundant (e.g., numerous straight-line cruise segments), and contain relatively few long-tailed safety-critical scenarios (e.g., collisions, emergency braking). It is the fundamental raw material for constructing the final training data sequence. A time segment refers to a continuous time interval within the raw training data sequence, with a fixed length (e.g., 10 seconds) or variable length (divided by event). Each time segment corresponds to a complete driving scenario, such as "a vehicle merging from a ramp onto the main road" or "waiting to turn left at an intersection." The time segment is the basic unit constituting the raw sample set. The raw sample set corresponds to the data set of all moments within a time segment, containing multi-view images (raw image sequence samples), vehicle motion parameters (raw motion parameter samples), and high-level subtask labels (raw subtask samples) for each moment in the segment. The original sample set is the smallest unit for subsequent sampling operations. The original image sequence samples are unsampled image frame sequences continuously captured at the original frame rate by multiple surround-view cameras (e.g., 7) on a moving vehicle within a single time segment. This sequence contains rich spatiotemporal details and serves as the foundational data for training the model to perceive the environment. The original motion parameter samples are ground truth values ​​of the vehicle's motion state, time-aligned with the original image sequence samples, including position, velocity, acceleration, yaw rate, steering wheel angle, etc. The sampling frequency is typically consistent with the image frame rate. The original subtask samples are high-level behavioral intent labels annotated within this time segment, such as "go straight," "turn left," "change lanes," "stop and yield," etc. These can be obtained from navigation instructions or manual annotations.

[0163] The specific implementation is as follows: First, deploy multiple surround-view cameras (e.g., 7, covering 360°) on a real autonomous vehicle to continuously record video at a fixed frame rate (e.g., 30fps). Simultaneously record the vehicle's CAN bus signals (vehicle speed, steering angle, acceleration, etc.) and GPS / inertial navigation data. Data collection covers various scenarios (urban roads, highways, intersections, nighttime, rainy days, etc.), with a total duration of at least several thousand hours. Strictly align image sequences, motion parameters, and subtask labels according to timestamps. Then, segment the data into continuous time segments according to fixed durations (e.g., 10 seconds) or based on event boundaries (e.g., from the start to the end of a subtask). Each time segment is a raw sample set. The beginning and end timestamps of each segment are retained during segmentation for subsequent sampling. For each time segment, subtask labels are automatically generated based on navigation instructions or rules (e.g., "go straight through the intersection"). For complex scenarios, manual annotation or high-precision map information can be used to determine the subtask category. Subtask samples can be represented as text strings or discrete IDs. Each original sample set is stored as a separate file or database record, containing image frames (compressed format such as JPEG), motion parameters (numpy arrays), and subtask labels (text). All original sample sets are arranged in chronological order to form the original training data sequence.

[0164] Step C2: In the original training data sequence, the original sample set is sampled according to the third time interval to obtain the first training sequence, wherein the first training sequence includes: the first image sequence sample and the corresponding first motion parameter sample and the first subtask sample.

[0165] Specifically, the third time interval refers to the time step in which the original training data sequence is uniformly sampled during random sampling. For example, if the original sequence is continuous 30fps data, the third time interval can be set to 0.5 seconds, 1 second, or 2 seconds. By extracting a set of original samples at fixed third time intervals, the temporal redundancy between adjacent segments can be significantly reduced (because driving scenarios typically change slowly), while ensuring broad coverage of the training data in the temporal dimension. The specific value of the third time interval can be adjusted according to the data scale and computational resources, with a typical value of 1 to 2 seconds. The first training sequence refers to the subset of training data extracted from the original training data sequence through random sampling (actually uniform sampling). This sequence retains the temporal order of the original data but has a lower temporal density (i.e., the interval between segments is the third time interval). The first training sequence covers various common driving scenarios in the dataset and is mainly used for the basic training of the model, enabling it to learn general environmental evolution patterns.

[0166] To achieve uniform sampling, the specific process is as follows: Set a third time interval: Determine Δtsample (e.g., 1 second) based on requirements. Each original sample set in the original training data sequence corresponds to a time segment, and the segment itself has a fixed length (e.g., 10 seconds). Uniform sampling refers to selecting a starting time every Δtsample seconds along the time axis, and then extracting a fixed-length (e.g., 10 seconds) set of original samples starting from that time. If the segments are continuous and non-overlapping, one sample can also be extracted directly every k segments.

[0167] Assume the total duration of the original training data sequence is Ttotal seconds, with segments arranged consecutively. Starting from time 0, move Δtsample seconds each time, and retrieve the original sample set corresponding to the current time point (if the time point falls within a segment). To avoid sampling the same segment multiple times, record the IDs of the sampled segments.

[0168] Arrange all the original sample sets obtained from sampling in chronological order to form the first training sequence. The number of samples in this sequence is approximately Ttotal / Δtsample. Due to the uniform sampling interval, the first training sequence can represent the long-term distribution of various scenarios in the original data, but it may undersample safety-critical scenarios (such as emergency braking and collisions) because these scenarios account for a very small proportion of time. Save the first training sequence as a separate dataset file.

[0169] Step C3: Identify key time segments in the time segments, and perform weighted sampling on the original sample set associated with the key time segments in the original training data sequence to obtain the second training sequence. The second training sequence includes: second image sequence samples and corresponding second motion parameter samples and second subtask samples.

[0170] Specifically, critical time segments refer to time intervals in the original training data sequence that are particularly important for the model to learn safety, dynamics, and causal reasoning. These segments typically contain high-risk traffic events, such as: emergency braking of vehicles or adjacent vehicles, pedestrians crossing the road, vehicles cutting in, unprotected left turns, close following, and sudden trajectory changes. In these segments, the evolution of environmental states has high information entropy and strong causal relationships; the model can only accurately predict the future if it learns the correct physical laws. Critical time segments can be automatically identified through manual annotation, rule-based filtering (such as detecting absolute acceleration values ​​exceeding a threshold), or anomaly detection algorithms.

[0171] The second training sequence refers to a subset of training data extracted from the original training data sequence through weighted sampling (importance sampling). The proportion of key time segments in this sequence is significantly increased (e.g., from 1% to 20%). The second training sequence complements the first training sequence: the first sequence ensures broad temporal coverage, while the second sequence enhances key dynamic learning. The two are combined to generate the final training data sequence.

[0172] The specific process for implementing importance-weighted sampling is as follows: First, for each original sample set in the original training data sequence, calculate a series of dynamic indicators: such as absolute velocity, lateral acceleration, rate of change of yaw rate, minimum distance to obstacles ahead, and speed of oncoming vehicles. Second, if any indicator exceeds a threshold (e.g., deceleration > 3 m / s², or minimum distance < 5 meters), the segment is marked as a "critical time segment". Alternatively, a trained anomaly detection model (e.g., based on a variational autoencoder) can be used to calculate the reconstruction error; if the error exceeds the high percentile, it is marked as critical. Simultaneously, a batch of high-risk scenarios (e.g., accident videos, emergency takeover moments) can be manually labeled as positive samples. Assume there are a total of M original sample sets, of which K are critical time segments. Define basic weights: non-critical segment weight wnormal = 1, critical segment weight wcritical = α, where α > 1 (e.g., α = 20). Weights can be further subdivided according to criticality. Normalized weights: pj = wj / ∑i=1Mwi, such that the sum of the probabilities of all segments being selected is 1.

[0173] Secondly, based on the probability distribution {pj}, N2 original sample sets are drawn with or without replacement (N2 is the preset size of the second training sequence, which can be equal to or a certain proportion of the size of the first training sequence). During the sampling process, key segments are selected with a much higher probability than ordinary segments, thus achieving oversampling of key dynamics. The drawn original sample sets are arranged in their original chronological order (or randomly shuffled) to form the second training sequence. Note: Weighted sampling may result in key segments being sampled multiple times, which is allowed and can enhance the model's learning of such samples.

[0174] Step C4: Generate training data sequence using the first training sequence and the second training sequence.

[0175] Specifically, the first and second training sequences are merged to generate the final training data sequence. The implementation is as follows: Assume the first training sequence has N1 samples and the second training sequence has N2 samples. Ideally, the final training data sequence should have a balanced ratio of regular to critical scenarios. This can be achieved by directly merging the samples according to Nfinal = N1 + N2, or by mixing them according to a certain ratio λ (e.g., λ = 0.7 from the first sequence and 0.3 from the second sequence). The mixed sampling strategy combines random sampling with importance-weighted sampling; therefore, direct merging or splicing according to a set ratio is commonly used.

[0176] The sample sets from both the first and second training sequences are merged into a large list. To eliminate order bias, the merged list is typically shuffled so that each batch during training contains both regular and key samples. Note: The original temporal order can be preserved before shuffling, but shuffling is more conducive to training stability. If a key segment in the second training sequence is sampled multiple times, the duplicates can be retained (for reinforcement learning) or removed after merging (to avoid overfitting). Duplicates are usually retained because the number of key segments is finite. The merged and shuffled sample set is used as the final training data sequence. The total number of samples in this sequence is N1 + N2 (if duplicates are not removed). This training data sequence will be directly used for training data loading.

[0177] Before actual training, online data augmentation (such as random brightness adjustment, cropping, flipping, etc.) can be performed on the image sequence samples. The generated training data sequence is then stored as a file or a memory dataset. This completes the full training data preparation process.

[0178] Figure 6 This is a schematic diagram of the architecture of the driving scenario prediction model in an embodiment of the present invention, as shown below. Figure 6As shown, the core of this architecture is a multimodal fusion visual language model. The model takes guidance information (a preset query token to trigger future variable predictions and a text token to specify the prediction time range or time block) and historical image sequences (time-series images captured by multi-view surround-view cameras) as input. Task embeddings and visual features are extracted by the text encoder and visual encoder, respectively. Both types of features are input into the Transformer decoder of the visual language model, where a self-attention mechanism completes cross-modal information interaction and spatiotemporal context modeling, resulting in structured fusion features. Subsequently, based on the LARCH (Long-Term Block Autoregression) strategy, the model recursively generates visual implicit features and driving features for each time block in an autoregressive manner. The visual implicit features serve as a compact world representation directly for auxiliary supervision and are also input into the VisionRenderer to reconstruct a high-fidelity prediction image sequence conditioned on historical images. The driving features are input into the state decoder, and the output is specific action commands (such as future trajectory points), thus achieving a two-dimensional prediction of future driving scenarios based on both images and states.

[0179] This embodiment also provides a driving scenario prediction device, which is used to implement the above embodiments and preferred embodiments, and will not be repeated as already described. As used below, the term "module" can be a combination of software and / or hardware that implements a predetermined function. Although the device described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0180] This embodiment provides a driving scenario prediction device, such as... Figure 7 As shown, it includes: The first acquisition module 61 is used to acquire, from the driving task of the mobile vehicle, a subtask to be executed within a first duration after the current moment as a target subtask; wherein, the driving task includes at least two subtasks with a temporal relationship; the first duration includes at least two time blocks with a temporal relationship. The second acquisition module 62 is used to acquire the real-time motion parameters of the mobile vehicle at the current moment, and the historical image sequence collected within a second time period before the current moment; The prediction module 63 is used to predict the image sequence and motion parameters of the mobile vehicle in each time block of the first duration based on historical image sequences, real-time motion parameters and target sub-tasks, so as to obtain the driving scene prediction result.

[0181] In this embodiment of the application, the prediction module 63 includes: The first prediction submodule is used to generate a first baseline feature corresponding to the first time block based on visual features extracted from historical image sequences, driving features extracted from real-time motion parameters, and target sub-tasks; and based on the first baseline feature, to predict the first predicted image sequence and the first predicted motion parameters associated with the first time block.

[0182] The second prediction submodule is used to determine the preceding reference features corresponding to the preceding time block; wherein, the preceding time block is the time block before the target time block; if the preceding time block is the first time block, then the preceding reference features are the first reference features; if the preceding time block is any time block after the first time block, then the preceding reference features are the reference features corresponding to that arbitrary time block; based on the preceding reference features, the target reference features corresponding to the target time block are predicted; based on the target reference features, the target prediction image sequence associated with the target time block and the target prediction motion parameters are predicted.

[0183] In this embodiment, the first prediction submodule is specifically used to extract visual features from historical image sequences; perform spatiotemporal compression on the visual features to obtain visual implicit features; generate driving features based on real-time motion parameters; and obtain guidance information associated with the first time block, wherein the guidance information is used to guide the target prediction model to predict the image sequence and motion parameters of the first time block based on the visual features, visual implicit features, and driving features; and use the visual features, visual implicit features, driving features, target sub-task, and guidance information as the first baseline features.

[0184] In this embodiment, the second prediction submodule is specifically used as the target sub-task in the prior reference features as the basis for predicting the mobile vehicle in the target time block. It uses the visual features in the prior reference features to predict the target visual features of the mobile vehicle in the target time block, and uses the driving features in the prior reference features to predict the target driving features of the vehicle in the target time block. It performs spatiotemporal compression on the target visual features to obtain the target visual implicit features, and obtains the target guidance information corresponding to the target time block. The target guidance information is used to guide the target prediction model to predict the image sequence and motion parameters of the target time block based on the target visual features, the target visual implicit features, and the target driving features. The target visual features, the target visual implicit features, the target driving features, the target sub-task, and the target guidance information are used as target reference features.

[0185] In this embodiment of the application, the first prediction submodule is specifically used to input the visual implicit features in the first reference features into the visual renderer, construct the first predicted image sequence of the first time block using the historical image sequence as a condition, and input the driving features in the first reference features into the state decoder to obtain the first predicted motion parameters of the first time block.

[0186] In this embodiment of the application, the prediction module 63 is specifically used to input the historical image sequence, real-time motion parameters and target sub-task into the pre-trained target prediction model, so that the target prediction model analyzes the historical image sequence and real-time motion parameters according to the target sub-task, and outputs the image sequence and motion parameters associated with each time block in the first time period after the current moment to predict the driving scene prediction result.

[0187] In this embodiment, the apparatus further includes: a training module, configured to acquire a training data sequence, wherein the training data sequence includes at least one set of samples, the sample set including image sequence samples and corresponding motion parameter samples and subtask samples; based on the training data sequence, on a short-term domain prediction task, the duration of the first sample is divided into at least one first sample time block according to a first time interval, and a first-stage training is performed on the initial decision model, so that the initial decision model learns the correlation between the image sequence samples and the corresponding motion parameter samples of each first sample time block according to the subtask samples, until a first convergence condition is met, and a candidate decision model is obtained; based on the training data sequence, on a long-term domain prediction task, the duration of the first sample is divided into at least one second sample time block according to a second time interval, and a second-stage training is performed on the candidate decision model, so that the candidate decision model learns the correlation between the image sequence samples and the corresponding motion parameter samples of each second sample time block according to the subtask samples, until a second convergence condition is met, and a target prediction model is obtained.

[0188] In this embodiment, the apparatus further includes: a sampling module, configured to acquire an original training data sequence, wherein the original training data sequence includes a set of original samples from multiple consecutive time segments, each set of original samples including original image sequence samples and corresponding original motion parameter samples and original subtask samples; in the original training data sequence, the original sample sets are sampled according to a third time interval to obtain a first training sequence, wherein the first training sequence includes: a first image sequence sample and corresponding first motion parameter samples and first subtask samples; key time segments are identified in the time segments, and the original sample sets associated with the key time segments are weighted and sampled in the original training data sequence to obtain a second training sequence, wherein the second training sequence includes: a second image sequence sample and corresponding second motion parameter samples and second subtask samples; and a training data sequence is generated using the first training sequence and the second training sequence.

[0189] Please see Figure 8 , Figure 8 This is a schematic diagram of the structure of a computer device provided in an optional embodiment of the present invention, such as... Figure 8As shown, the computer device includes one or more processors 10, memory 20, and interfaces for connecting the components, including high-speed interfaces and low-speed interfaces. The components communicate with each other via different buses and can be mounted on a common motherboard or otherwise installed as needed. The processors can process instructions executed within the computer device, including instructions stored in or on memory to display graphical information of a GUI on external input / output devices (such as display devices coupled to the interfaces). In some alternative implementations, multiple processors and / or multiple buses can be used with multiple memories and multiple memory modules, if desired. Similarly, multiple computer devices can be connected, each providing some of the necessary operations (e.g., as a server array, a group of blade servers, or a multiprocessor system).

[0190] Processor 10 may be a central processing unit, a network processor, or a combination thereof. Processor 10 may further include a hardware chip. The hardware chip may be an application-specific integrated circuit (ASIC), a programmable logic device (PLD), or a combination thereof. The programmable logic device may be a complex programmable logic device (CAMP), a field-programmable gate array (FPGA), a general-purpose array logic (GDA), or any combination thereof.

[0191] The memory 20 stores instructions executable by at least one processor 10 to cause at least one processor 10 to perform the method shown in the above embodiments.

[0192] The memory 20 may include a program storage area and a data storage area. The program storage area may store the operating system and applications required for at least one function; the data storage area may store data created based on the use of the computer device as shown by a landing page for an app. Furthermore, the memory 20 may include high-speed random access memory and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some alternative embodiments, the memory 20 may optionally include memory remotely located relative to the processor 10, which can be connected to the computer device via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0193] The memory 20 may include volatile memory, such as random access memory; the memory may also include non-volatile memory, such as flash memory, hard disk or solid-state drive; the memory 20 may also include a combination of the above types of memory.

[0194] The computer device also includes a communication interface 30 for communicating with other devices or communication networks.

[0195] This invention also provides a computer-readable storage medium. The methods described above according to embodiments of the invention can be implemented in hardware or firmware, or implemented as computer code that can be recorded on a storage medium, or implemented as computer code downloaded via a network and originally stored on a remote storage medium or a non-transitory machine-readable storage medium and then stored on a local storage medium. Thus, the methods described herein can be processed by software stored on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. The storage medium can be a magnetic disk, optical disk, read-only memory, random access memory, flash memory, hard disk, or solid-state drive, etc.; further, the storage medium can also include combinations of the above types of memory. It is understood that computers, processors, microprocessor controllers, or programmable hardware include storage components capable of storing or receiving software or computer code, which, when accessed and executed by the computer, processor, or hardware, implements the methods shown in the above embodiments.

[0196] Although embodiments of the invention have been described in conjunction with the accompanying drawings, those skilled in the art can make various modifications and variations without departing from the spirit and scope of the invention, and such modifications and variations all fall within the scope defined by the appended claims.

Claims

1. A method for predicting driving scenarios, characterized in that, The method includes: Obtain the subtasks to be executed within a first duration after the current moment from the driving task of the mobile vehicle as target subtasks; wherein, the driving task includes at least two subtasks with a temporal relationship; the first duration includes time blocks with at least two temporal relationships; The real-time motion parameters of the mobile vehicle at the current moment, and the historical image sequence collected during the second time period prior to the current moment are obtained; Based on the historical image sequence, the real-time motion parameters, and the target sub-task, the image sequence and motion parameters of the mobile vehicle in each time block within the first duration are predicted to obtain the driving scene prediction result.

2. The method according to claim 1, characterized in that, The method for predicting the image sequence and motion parameters of the mobile vehicle in the first time block of the time series includes: Based on the visual features extracted from the historical image sequence, the driving features extracted from the real-time motion parameters, and the target sub-task, a first reference feature corresponding to the first time block is generated; Based on the first reference feature, a first predicted image sequence associated with the first time block and a first predicted motion parameter are predicted.

3. The method according to claim 2, characterized in that, The method for predicting the image sequence and motion parameters of the mobile vehicle in a target time block other than the first time block includes: Determine the preceding reference feature corresponding to the preceding time block; wherein, the preceding time block is the time block before the target time block; if the preceding time block is a first time block, then the preceding reference feature is the first reference feature; if the preceding time block is any time block after the first time block, then the preceding reference feature is the reference feature corresponding to that arbitrary time block. Based on the aforementioned prior reference features, predict the target reference features corresponding to the target time block; Based on the target baseline features, predict the target predicted image sequence and target predicted motion parameters associated with the target time block.

4. The method according to claim 2, characterized in that, The step of generating a first baseline feature corresponding to the first time block based on visual features extracted from the historical image sequence, driving features extracted from the real-time motion parameters, and the target sub-task includes: Extract visual features from the historical image sequence; Spatiotemporal compression is applied to the visual features to obtain visual implicit features; Based on the real-time motion parameters, driving features are generated, and guidance information associated with the first time block is obtained. The guidance information is used to guide the target prediction model to predict the image sequence and motion parameters of the first time block based on the visual features, the visual implicit features, and the driving features. The visual features, the visual implicit features, the driving features, the target sub-task, and the guidance information are used as the first reference features.

5. The method according to claim 3, characterized in that, The step of predicting the target reference features corresponding to the target time block based on the preceding reference features includes: Using the target sub-task in the preceding reference features as the basis for predicting the mobile vehicle in the target time block, the target visual features of the mobile vehicle in the target time block are predicted using the visual features in the preceding reference features, and the target driving features of the vehicle in the target time block are predicted using the driving features in the preceding reference features. Spatiotemporal compression is performed on the target visual features to obtain target visual implicit features, and target guidance information corresponding to the target time block is obtained. The target guidance information is used to guide the target prediction model to predict the image sequence and motion parameters of the target time block based on the target visual features, the target visual implicit features, and the target driving features. The target visual features, the target visual implicit features, the target driving features, the target sub-tasks, and the target guidance information are used as the target baseline features.

6. The method according to claim 2, characterized in that, The step of predicting the first predicted image sequence and the first predicted motion parameters associated with the first time block based on the first reference feature includes: The visual implicit features in the first reference features are input into the visual renderer, and the first predicted image sequence of the first time block is constructed using the historical image sequence as a condition. At the same time, the driving features in the first reference features are input into the state decoder to obtain the first predicted motion parameters of the first time block.

7. The method according to claim 1, characterized in that, The step of predicting the image sequence and motion parameters of the mobile vehicle for each time block within the first duration based on the historical image sequence, the real-time motion parameters, and the target sub-task, to obtain the driving scene prediction result, includes: The historical image sequence, the real-time motion parameters, and the target sub-task are input into a pre-trained target prediction model. The target prediction model analyzes the historical image sequence and the real-time motion parameters according to the target sub-task, and outputs the image sequence and motion parameters associated with each time block of the mobile vehicle in the first time period after the current moment to predict the driving scene prediction result.

8. The method according to claim 7, characterized in that, Before inputting the historical image sequence, the real-time motion parameters, and the subtask into the pre-trained target prediction model, the method further includes: Obtain a training data sequence, wherein the training data sequence includes at least one set of samples, the set of samples including image sequence samples and corresponding motion parameter samples and subtask samples; Based on the training data sequence, in the short time domain prediction task, the duration of the first sample is divided into at least one first sample time block according to the first time interval, and the initial decision model is trained in the first stage, so that the initial decision model learns the correlation between the image sequence samples and the corresponding motion parameter samples of each first sample time block according to the sub-task samples, until the first convergence condition is met, and a candidate decision model is obtained. Based on the training data sequence, in the long-term prediction task, the duration of the first sample is divided into at least one second sample time block according to the second time interval, and the candidate decision model is trained in the second stage. The candidate decision model learns the correlation between the image sequence samples and the corresponding motion parameter samples of each second sample time block according to the sub-task samples, until the second convergence condition is met, and the target prediction model is obtained.

9. The method according to claim 8, characterized in that, Before acquiring the training data sequence, the method further includes: Obtain the original training data sequence, wherein the original training data sequence includes a set of original samples from multiple consecutive time segments, and each set of original samples includes original image sequence samples and corresponding original motion parameter samples and original subtask samples; In the original training data sequence, the original sample set is sampled according to a third time interval to obtain a first training sequence, wherein the first training sequence includes: a first image sequence sample and corresponding first motion parameter samples and first subtask samples; Identify key time segments in the time segments, and perform weighted sampling on the original sample set associated with the key time segments in the original training data sequence to obtain a second training sequence, wherein the second training sequence includes: a second image sequence sample and corresponding second motion parameter samples and second subtask samples; The training data sequence is generated using the first training sequence and the second training sequence.

10. A driving scenario prediction device, characterized in that, The device includes: The first acquisition module is used to acquire, from the driving task of the mobile vehicle, a subtask to be executed within a first duration after the current moment as a target subtask; wherein, the driving task includes at least two subtasks with a temporal relationship; the first duration includes at least two time blocks with a temporal relationship. The second acquisition module is used to acquire the real-time motion parameters of the mobile vehicle at the current moment, and the historical image sequence collected within a second time period before the current moment; The prediction module is used to predict the image sequence and motion parameters of the mobile vehicle in each time block of the first duration based on the historical image sequence, the real-time motion parameters, and the target sub-task, so as to obtain the driving scene prediction result.

11. A mobile vehicle, characterized in that, The mobile vehicle includes a controller, which includes a memory and a processor, the memory and the processor being communicatively connected to each other, the memory storing computer instructions, and the processor executing the computer instructions to perform the method of any one of claims 1 to 9.

12. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions for causing a computer to perform the method of any one of claims 1 to 9.

Images