World model measurement system construction method and system based on generative adversarial discriminator, computer device and medium

Abstract

The application relates to a world model measurement system construction method and system based on generative adversarial discrimination, a computer device and a medium. The method comprises the following steps: constructing a strategy network, a world model simulation system and a virtual-real discriminator; freezing the strategy network and the world model simulation system, generating a first virtual trajectory through offline rolling of the strategy network and the world model simulation system, taking a real trajectory and the first virtual trajectory as positive samples and negative samples respectively, and pre-training the virtual-real discriminator; unfreezing the strategy network and the world model simulation system, generating a second virtual trajectory through online rolling of the strategy network and the world model simulation system, and performing joint adversarial training on the pre-trained virtual-real discriminator, the strategy network and the world model simulation system by using the real trajectory and the second virtual trajectory; freezing the virtual-real discriminator, generating a third virtual trajectory through online rolling of the strategy network and the world model simulation system, and performing fine-tuning training on the strategy network by using the third virtual trajectory. The method can realize closed-loop training and quantitative evaluation of an automatic driving strategy under a condition without true value supervision.

Description

Technical Field

[0001] This application relates to the field of autonomous driving technology, and in particular to a method, system, computer device and medium for constructing a world model metric system based on generative adversarial discrimination. Background Technology

[0002] Currently, the training and evaluation of autonomous driving strategies mainly employ three methods: manual metric design, behavioral cloning evaluation, and rule-based verification. Manual metric design involves defining handcrafted features such as trajectory smoothness and lateral offset based on expert experience; for example, jerk should be less than 2.0 m / s³ and lateral error less than 0.3 m. Behavioral cloning evaluation calculates strategy similarity based on expert demonstration data, such as DTW distance. Rule-based verification determines scenario feasibility by setting preset safety thresholds, such as a time-to-collision (TTC) greater than 3 seconds. However, these methods still have some shortcomings. For example, manually designed metrics may have a certain degree of subjectivity, leading to significant differences between hand-designed features and the distribution of real driving behavior, such as a KL divergence greater than 0.45; the methods mentioned above are all non-differentiable, so the evaluation metrics cannot backpropagate gradients, resulting in decoupling between training and evaluation operations; the above methods are not sensitive enough to low-probability scenarios, which may lead to a high false negative rate in abnormal scenario detection, such as a false negative rate greater than 40%; in addition, the trajectories generated in the virtual environment deviate significantly from the distribution of real trajectories. Summary of the Invention

[0003] To address the aforementioned shortcomings and drawbacks, this application provides a method, system, computer equipment, and medium for constructing a world model metric system based on generative adversarial discrimination. This method enables closed-loop training and quantitative evaluation of autonomous driving strategies under unsupervised ground truth conditions, resolving the distribution bias and non-differentiability issues in manually designed metrics.

[0004] This application provides a method for constructing a world model metric system according to a first aspect. The method includes: constructing a policy network, a world model simulation system, and a virtual-real discriminant; the policy network is used to generate vehicle control actions for the current time step based on the system state at the current time step; the world model simulation system is used to generate the system state for the next time step based on the system state at the current time step and the vehicle control actions; the virtual-real discriminant is used to calculate the probability of the authenticity of an input trajectory, where the input trajectory is a real trajectory or a virtual trajectory; freezing the policy network and the world model simulation system, and offline rolling generation of a first virtual trajectory through the policy network and the world model simulation system, and comparing the real trajectory with the first virtual trajectory. Virtual trajectories are used as positive and negative samples respectively to pre-train the virtual-real discriminator. The policy network and world model simulation system are unfrozen, and a second virtual trajectory is generated online through the policy network and world model simulation system. The pre-trained virtual-real discriminator, policy network, and world model simulation system are jointly trained using the real trajectory and the second virtual trajectory. The virtual-real discriminator is frozen, and a third virtual trajectory is generated online through the policy network and world model simulation system. The policy network is fine-tuned using the third virtual trajectory. During fine-tuning, the authenticity probability output by the frozen virtual-real discriminator or its corresponding embedding vector is used as the reward signal.

[0005] In some embodiments, the first virtual trajectory is generated offline in a rolling manner through a policy network and a world model simulation system, including: cyclically calling the frozen policy network and world model simulation system outputs of vehicle control actions and system states; generating the first virtual trajectory based on the multi-frame vehicle control actions corresponding to different time steps output by the policy network and the multi-frame system states corresponding to different time steps output by the world model simulation system; in each cyclic call, if this is the first cyclic call, the input to the policy network is a single-frame system state randomly selected from the real driving record; if this is not the first cyclic call, the system state output by the world model simulation system in the previous cyclic call; in each cyclic call, the input to the world model simulation system is the vehicle control action output by the policy network in this cyclic call.

[0006] In some embodiments, the second virtual trajectory is generated online in a rolling manner through a policy network and a world model simulation system, including: before each round of iterative training, cyclically calling the unfrozen policy network and the world model simulation system to output vehicle control actions and system states, and generating the second virtual trajectory based on the multi-frame vehicle control actions output by the policy network corresponding to different time steps and the multi-frame system states output by the world model simulation system corresponding to different time steps.

[0007] In some embodiments, a third virtual trajectory is generated online in a rolling manner through a policy network and a world model simulation system, including: detecting whether the trajectory generation conditions are met; the trajectory generation conditions refer to the completion of parameter updates of the policy network and the failure to meet the training termination conditions corresponding to fine-tuning training; if the trajectory generation conditions are met, the policy network and the world model simulation system output vehicle control actions and system states in a loop, and the third virtual trajectory is generated based on the multi-frame vehicle control actions corresponding to different time steps output by the policy network and the multi-frame system states corresponding to different time steps output by the world model simulation system.

[0008] In some embodiments, the following loss function is used during pre-training: ; D represents the virtual / real discriminant; Indicates both real and virtual losses; This indicates that the virtual / real discriminator distinguishes between the real and virtual trajectories. The probability of the output being true; This indicates that the virtual / real discriminator distinguishes between virtual trajectories. The probability of the output being true; The following loss function is used when conducting joint adversarial training: ; This indicates a dynamic measure of learning loss; This indicates that the virtual / real discriminator distinguishes between the real and virtual trajectories. The probability of the output being true; This represents the expectation of the distribution of the true trajectory; This represents the virtual trajectory generated by the world model simulation system; This indicates that the virtual / real discriminator distinguishes between virtual trajectories. The probability of the output being true; This represents the expectation of the distribution of the virtual trajectory; The following loss function is used during fine-tuning training: ;in, ; Indicate the strategy optimization objective; Representing the trajectory Embedded vectors in a differentiable metric space; This represents the system state at time step t; This represents the vehicle control action corresponding to time step t; Represents the action value function; Indicates the trajectory Calculate the expected system state for all time steps within the time interval; The feature weights represent the output of the frozen virtual / real discriminator for the system state at time step t. This represents the behavioral feature vector output by the feature extractor for the system state at time step t.

[0009] In some embodiments, when performing joint adversarial training on a pre-trained virtual-real discriminator, a policy network, and a world model simulation system using real trajectories and second virtual trajectories, the parameters of the virtual-real discriminator and the world model simulation system are updated alternately; while updating the parameters of the world model simulation system, the parameters of the policy network are updated synchronously.

[0010] In some embodiments, during fine-tuning training, the world model simulation system is frozen or kept thawed.

[0011] This application provides a world model measurement system construction system according to the second aspect, the system comprising: The building module is used to construct the policy network, the world model simulation system, and the virtual-real discriminator. The policy network is used to generate the vehicle control action for the current time step based on the system state at the current time step. The world model simulation system is used to generate the system state for the next time step based on the system state at the current time step and the vehicle control action. The virtual-real discriminator is used to calculate the probability of the authenticity of the input trajectory, which can be either a real trajectory or a virtual trajectory. The phased training module is used to freeze the policy network and the world model simulation system. It generates a first virtual trajectory offline using the policy network and the world model simulation system, and uses the real trajectory and the first virtual trajectory as positive and negative samples, respectively, to pre-train the virtual-real discriminator. Next, it unfreezes the policy network and the world model simulation system, and generates a second virtual trajectory online using the policy network and the world model simulation system. It then uses the real trajectory and the second virtual trajectory to perform joint adversarial training on the pre-trained virtual-real discriminator, the policy network, and the world model simulation system. Finally, it freezes the virtual-real discriminator, and generates a third virtual trajectory online using the policy network and the world model simulation system. It uses the third virtual trajectory to fine-tune the policy network, using the real probability output by the frozen virtual-real discriminator or its corresponding embedding vector as the reward signal during fine-tuning.

[0012] According to a third aspect, this application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of any of the world model measurement system construction methods described above.

[0013] According to a fourth aspect, this application provides a computer device including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executed, implements the steps of any of the world model measurement system construction methods described above.

[0014] The above embodiments of this application are applicable to closed-loop training and performance evaluation of autonomous driving systems under conditions lacking truth supervision. They can realize closed-loop training and quantitative evaluation of autonomous driving strategies under conditions without truth supervision, and solve the problems of distribution bias and non-differentiability in manual measurement design.

[0015] Specifically, after constructing the policy network, the world model simulation system, and the virtual-real discriminator, the above embodiments perform three stages of training in sequence: First, the policy network and the world model simulation system are frozen. The first virtual trajectory is generated offline by the policy network and the world model simulation system. The real trajectory and the first virtual trajectory are used as positive samples and negative samples, respectively, to pre-train the virtual-real discriminator. This allows the virtual-real discriminator to establish an initial decision boundary on the real trajectory and the virtual trajectory, providing a cold start surface for the subsequent metric space and avoiding training oscillations caused by random initialization. Next, the policy network and world model simulation system are unfrozen. A second virtual trajectory is generated online through the policy network and world model simulation system. The pre-trained virtual-real discriminator, policy network, and world model simulation system are then subjected to joint adversarial training using the real and second virtual trajectories. During joint adversarial training, the virtual-real discriminator outputs the probability of the trajectory's authenticity during forward propagation. The intermediate features naturally form an embedding vector, which evolves synchronously with the joint adversarial training, constituting a differentiable trajectory metric space. This metric space can convert the quality of the trajectory into vector distance and can be backpropagated, thus achieving the effect of an evaluation metric, i.e., a loss function. Furthermore, it can integrate training operations and... The evaluation operations are linked together; finally, the virtual-real discriminator is frozen, and a third virtual trajectory is generated online through the policy network and the world model simulation system. The policy network is then fine-tuned using the third virtual trajectory. During fine-tuning, the real probability output by the frozen virtual-real discriminator or its corresponding embedding vector is used as the reward signal. Since the reward signal is from the same source as the metric space, there is no need to manually design heuristic indicators such as jerk or time-to-collision (TTC). The policy (i.e., the parameters of the policy network) can be continuously optimized in the virtual environment. At the same time, since the metric space is kept in the same distribution as the real domain, the Fréchet distance between the virtual trajectory and the real trajectory can be significantly reduced, that is, the degree of deviation between the distribution of the virtual trajectory and the real trajectory is reduced. Attached Figure Description

[0016] Figure 1 This is a flowchart illustrating a method for constructing a world model metric system according to one or more embodiments of this application; Figure 2 This is a schematic diagram of the adversarial metric learning framework in one or more embodiments of this application; Figure 3This is a schematic diagram of an adversarial gradient propagation path in one or more embodiments of this application; Figure 4 This is a schematic diagram of the alternating update process in one or more embodiments of this application; Figure 5 This is a schematic diagram illustrating the distribution optimization of virtual and real trajectories in one or more embodiments of this application; Figure 6 This is a schematic diagram of the structure of the world model metric system construction system in another embodiment of this application; Figure 7 This is a schematic diagram of the internal structure of a computer device according to one or more embodiments of this application. Detailed Implementation

[0017] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings. It should be understood that the described embodiments are merely some embodiments of this application, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the scope of protection of this application.

[0018] In the following description, when referring to the accompanying drawings, the same numbers in different drawings denote the same or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of systems and methods consistent with some aspects of this application as detailed in the appended claims.

[0019] In the description of this application, it should be understood that the terms "first," "second," "third," etc., are used only to distinguish similar objects and are not necessarily used to describe a specific order or sequence, nor should they be construed as indicating or implying relative importance. Those skilled in the art can understand the specific meaning of the above terms in this application according to the specific circumstances. Furthermore, in the description of this application, unless otherwise stated, "multiple" refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. The character " / " generally indicates that the preceding and following related objects have an "or" relationship.

[0020] To address the shortcomings or defects of related technologies, this application provides a method for constructing a world model metric system. This method is applicable to the closed-loop training and performance evaluation of autonomous driving systems under conditions lacking truth supervision. It can realize the closed-loop training and quantitative evaluation of autonomous driving strategies under conditions without truth supervision, and solve the problems of distribution bias and non-differentiability in manual metric design.

[0021] In some exemplary embodiments of this application, the method includes as follows: Figure 1 The steps shown are explained in detail below.

[0022] S110: Construct a policy network, a world model simulation system, and a virtual-real discriminator.

[0023] The policy network is used to generate vehicle control actions for the current time step based on the system state at the current time step.

[0024] The policy network can be constructed using an Actor-Critic architecture based on the Proximal Policy Optimization (PPO) algorithm. This Actor-Critic architecture is a core algorithm in reinforcement learning. The Actor represents the policy network, responsible for selecting actions based on the current policy, while the Critic represents the value network, responsible for evaluating the policy and providing a value function estimate. The Actor-Critic framework achieves efficient decision-making through the collaborative optimization of the policy network (Actor) and the value network (Critic).

[0025] In some embodiments, the Actor and Critic each contain at least one fully connected layer (which may have a dimension of 256) and a nonlinear activation (such as the ReLU activation function).

[0026] Actor's autonomous driving system state at time step t As input, after passing through at least one fully connected layer and nonlinear activation, the output distributed parameters of the vehicle control action can be expressed as follows: , It refers to the mean of a normal distribution. This refers to the standard deviation of a normal distribution. Sampling the above distribution parameters yields the vehicle control action at time step t, which can be expressed as: Critic, on the other hand, uses the same system state. As input, after passing through at least one fully connected layer and nonlinear activation, the output is the value estimate of this state, which can be expressed as: The weights of the Actor and Critic can be independent or shared. These weights are used to simultaneously generate vehicle control actions and corresponding state values ​​in the same forward process to support subsequent policy gradient calculations. The autonomous driving system state, or simply system state, refers to the minimum set of observations at a given time step (e.g., time step t) that fully describes the autonomous vehicle's own kinematics and the surrounding static and dynamic scene. This set can include the vehicle's state, bird's-eye view perception data, map semantic data, and the states of traffic participants.

[0027] The autonomous vehicle state refers to the set of kinematic and dynamic parameters describing an autonomous driving vehicle (ADV). This can include the autonomous vehicle's position (such as the three-dimensional coordinates of the vehicle's center in the global coordinate system x, y, z), velocity (such as the linear velocity of the vehicle along the x, y, and z axes v_x, v_y, v_z), acceleration (such as the linear acceleration of the vehicle along the x, y, and z axes a_x, a_y, a_z), steering angle (such as the front wheel steering angle), heading angle (the angle between the vehicle's longitudinal axis and the global x-axis), and control variables (such as accelerator or throttle opening, brake pressure, gear position, and turn signal).

[0028] Bird's-eye view perception data refers to the top-down grid or vector features formed by projecting vehicle sensor data (such as LiDAR, camera, and / or radar sensors) onto the XY plane of the vehicle coordinate system. It is used to uniformly describe drivable areas, obstacles, and road boundaries. For example, bird's-eye view perception data can be a tensor of size 128×128×3, with its three channels representing point cloud density, maximum height, and semantic classification (e.g., a semantic classification value of 0 indicates background, 1 indicates a vehicle, and 2 indicates a pedestrian).

[0029] Map semantic data refers to static semantic elements extracted from high-precision maps that are related to the vehicle's current location within a preset range (e.g., 100 meters). These elements can be in vector or raster form. For example, elements such as ① discrete points of the center line (x, y, z, curvature, speed limit), ② polylines of the left and right lane boundaries, ③ unique identifiers, phases, and remaining durations of traffic lights, and ④ bitmaps of intersections, speed bumps, and parking spaces can all be converted to the vehicle's center coordinates to obtain map semantic data, which can have up to 7 channels.

[0030] Traffic participant status refers to the real-time status of all dynamic traffic participants, excluding the vehicle itself, at the current moment (e.g., time t), including but not limited to position (e.g., the three-dimensional coordinates of the traffic participant's center in the global coordinate system x, y, z), speed (e.g., the linear velocity of the traffic participant along the x and y axes), orientation, and behavioral intentions (e.g., going straight, turning left, turning right, standing still, changing lanes, accelerating or decelerating, etc.).

[0031] The world model simulation system (or simply simulation system) is a key component of the world model. It generates the system state for the next time step based on the current system state and vehicle control actions. The world model simulation system is a differentiable dynamics model. It is assumed that the current system state and vehicle control actions are... and ,Will and By inputting the world model simulation system, the system state at the next time step can be predicted, which can be represented as: .

[0032] The real / virtual discriminator is used to calculate the probability of an input trajectory being real, and the result of the discriminator's calculation can be used as a metric. The input trajectory refers to the trajectory whose probability of being real needs to be calculated by the real / virtual discriminator; the input trajectory can be a real trajectory or a virtual trajectory.

[0033] The actual trajectory refers to the time series of state, action, and environment extracted from real road test data.

[0034] Real-world road test data consists of raw records collected by multiple sensors over one or more time periods, including but not limited to sensor data, vehicle status, environmental information, and information on other road users. These sensors may include LiDAR (Light Laser Detection and Ranging), cameras, millimeter-wave radar, IMU (Inertial Measurement Unit), and GPS (or RTK). GPS stands for Global Positioning System, and RTK stands for Real-time kinematic. Sensor data can include 3D point cloud sequences acquired by LiDAR (sampling frequency can be 10 to 20 Hz, which can be used for high-precision geometric modeling and obstacle detection), multi-view RGB (Red-Green-Blue) image sequences acquired by cameras (sampling frequency can be 30 to 60 Hz, which can be used for semantic segmentation and traffic sign recognition), 4D point clouds acquired by millimeter-wave radar (sampling frequency can be 10 to 25 Hz, which can be used for moving object tracking and speed measurement), 6-axis inertial measurement data acquired by IMU (sampling frequency can be 100 to 1000 Hz, which can be used for vehicle pose estimation and motion compensation), and high-precision global positioning data acquired by GPS or RTK (sampling frequency can be 5 to 10 Hz, which can be used for global path planning and map matching).

[0035] In the state-action-environment time series, "state" refers to the system state, "action" refers to the vehicle control actions, and "environment" refers to environmental observation information. Environmental observation information refers to the set of external static and dynamic environmental parameters that affect vehicle decisions, including traffic signals (such as red, yellow, and green lights and countdown seconds), road topology (such as lane centerline coordinates, lane width, curvature, and speed limits), weather conditions (such as discrete categories like sunny, rainy, snowy, and foggy, and visibility in meters), lighting conditions (such as discrete categories like daytime, nighttime, dusk, and tunnel conditions), and road surface adhesion coefficient.

[0036] Virtual trajectory refers to a state-action environment time sequence generated frame by frame within a world model simulation system, driven by the vehicle control action sequence output by the policy network, without the participation of human driving data throughout the process.

[0037] The architecture of the virtual-real discriminator may include an input layer, a first spatiotemporal convolutional layer, a pooling layer, a second spatiotemporal convolutional layer, a global average pooling layer, a fully connected layer, and a sigmoid activation function, connected in sequence.

[0038] The input layer is used to receive the spatiotemporal cube, which is obtained by stacking multiple frames of system state, vehicle control actions and environmental observation information in the trajectory. The first spatiotemporal convolutional layer is used to perform convolution operations on the spatiotemporal cube, such as the Conv3D(32, kernel=(3,5,5)) operation, and apply spectral normalization constraints to the convolution kernel, finally outputting the first spatiotemporal feature map; Pooling layers are used to perform max pooling operations, such as MaxPool3D(2), on the first spatiotemporal feature map to downsample the first feature map; The second spatiotemporal convolutional layer is used to perform convolution operations on the downsampled first spatiotemporal feature map, such as the Conv3D(64, kernel=(3,5,5)) operation, and continues to apply spectral normalization constraints, finally outputting the second spatiotemporal feature map; The global average pooling layer is used to perform global average pooling operations on the second spatiotemporal feature map, such as the GlobalAvgPool3D operation, and finally obtains a vector of fixed length. A fully connected layer is used to map this vector to a one-dimensional output; The Sigmoid activation function is used to convert a one-dimensional output into a probability of the trajectory's authenticity, with the probability of authenticity ranging from [0,1].

[0039] The real / virtual discriminator is also connected in series with a feature extractor. The intermediate features of the real / virtual discriminator serve as the input to the feature extractor. The real / virtual discriminator calculates the probability of a trajectory being real, while the feature extractor converts the trajectory into a behavioral feature vector. After calculating the behavioral feature vectors between the real and virtual trajectories, the Fréchet distance between the two trajectories can be calculated. After the original trajectory is input into the real / virtual discriminator, the data flow is: original trajectory → real / virtual discriminator → feature extractor → differentiable metric space.

[0040] The architecture of the feature extractor includes an input interface, a spatiotemporal Transformer module, and a feature projection layer.

[0041] The input interface is used to receive the spatiotemporal feature map (T'×H'×W'×C') output by the intermediate layer of the virtual-real discriminator; The Spatiotemporal Transformer module is used to perform 8-head self-attention computation on the spatiotemporal feature map and add positional encoding, and finally output spatiotemporal correlation features; The feature projection layer is used to map spatiotemporal correlation features into d-dimensional behavioral feature vectors to form a differentiable metric space.

[0042] The S120 process consists of three training phases executed sequentially: the first phase is the discriminator pre-training phase, the second phase is the joint adversarial training phase, and the third phase is the policy fine-tuning training phase. The training process of each phase is explained in detail below.

[0043] The first stage is to freeze the policy network and the world model simulation system. The first virtual trajectory is generated offline by the policy network and the world model simulation system. The real trajectory and the first virtual trajectory are used as positive samples and negative samples, respectively, to pre-train the virtual-real discriminator.

[0044] Before training begins, all trainable parameters of the policy network and the world model simulation system can be set to require_grad=False to ensure that these parameters are not updated during this training phase.

[0045] The first virtual trajectory refers to the virtual trajectory generated in the first stage. There are multiple first virtual trajectories. All first virtual trajectories are generated at once before training begins using the frozen policy network and world model simulation system, and stored in a local cache file. During training, the first virtual trajectories are read directly from the local cache file, without calling the policy network and world model simulation system again, to ensure the stability of the pre-training boundary.

[0046] The operation of generating the first virtual trajectory offline using the policy network and world model simulation system can include: cyclically calling the frozen policy network and world model simulation system outputs of vehicle control actions and system states; generating the first virtual trajectory based on the multi-frame vehicle control actions corresponding to different time steps output by the policy network and the multi-frame system states corresponding to different time steps output by the world model simulation system; in each cyclic call, if this is the first cyclic call, the input to the policy network is a single-frame system state randomly selected from the real driving record (i.e., real road test data); if this is not the first cyclic call, the system state output by the world model simulation system in the previous cyclic call; in each cyclic call, the input to the world model simulation system is the vehicle control action output by the policy network in this cyclic call.

[0047] For example, performing a loop to call the frozen policy network and world model simulation system outputs vehicle control actions and system states may include: setting the system state at the current time step (e.g., time step t). Input the policy network to obtain the action distribution parameters for the current time step. Based on these action distribution parameters, sample the vehicle control actions for the current time step. ; System status and vehicle control actions Input the world model simulation system to obtain the system state at time step t+1. ; Determine whether the number of loop calls has reached the preset number. If the preset number has been reached, stop executing the loop calls and generate the first virtual trajectory based on the system state and vehicle control actions corresponding to all time steps. If the preset number has not been reached, execute the loop calls using the system state of the current time step.

[0048] The preset number of times is the number of times the loop is executed. It is related to the range of values ​​of t, which can be [1, T]. If T is 300 steps, then the number of times the loop is executed is 300. If the sampling frequency of the vehicle sensor is 10 Hz (i.e., sampling once every 0.1 seconds), then when the number of loop calls reaches 300, a first virtual trajectory of 30 seconds can be generated.

[0049] Through this stage of training, the virtual-real discriminator can establish initial decision boundaries on real and virtual trajectories. This provides a cold-start surface for the subsequent metric space and avoids training oscillations caused by random initialization.

[0050] The second stage is to unfreeze the policy network and world model simulation system, generate a second virtual trajectory online through the policy network and world model simulation system, and use the real trajectory and the second virtual trajectory to conduct joint adversarial training on the pre-trained virtual-real discriminator, policy network and world model simulation system.

[0051] Figure 2 The diagram shown is a schematic of the adversarial metric learning framework. Figure 3 This is a schematic diagram of the adversarial gradient propagation path. The following will combine... Figure 2 and Figure 3 The training process for the second phase will be explained.

[0052] After completing the first phase of training, all trainable parameters of the policy network and the world model simulation system are set to require_grad=True to allow the parameters of the policy network and the world model simulation system to be updated via backpropagation.

[0053] The second virtual trajectory refers to the virtual trajectory generated in the second stage. There are multiple second virtual trajectories, each generated in real-time before each round of training iterations, rather than pre-generated offline. The distribution of the first virtual trajectory generated offline is fixed, while the distribution of the second virtual trajectory generated online changes in real-time with the parameters of the policy network and the world model simulation system, thus forming an adversarial game.

[0054] The operation of generating a second virtual trajectory online using a policy network and a world model simulation system includes: before each round of iterative training, repeatedly calling the unfrozen policy network and the world model simulation system to output vehicle control actions and system states, and generating a second virtual trajectory based on the multi-frame vehicle control actions output by the policy network corresponding to different time steps and the multi-frame system states output by the world model simulation system corresponding to different time steps.

[0055] For example, the process for each training iteration (epoch) can be as follows: (1) Initial state sampling, i.e., randomly sampling a single frame of system state from the actual driving record, can be represented as: ; (2) Perform online rolling operation (assuming the range of t is [1,T], T=300 steps, and the sensor sampling frequency is 10 Hz), that is: ① Set the system state at the current time step (e.g., time step t). Input the policy network to obtain the action distribution parameters for the current time step. Based on these action distribution parameters, sample the vehicle control actions for the current time step. ; ②System status and vehicle control actions Input the world model simulation system to obtain the system state at time step t+1. ; ③ Put Pushed into the GPU (Graphics Processing Unit) cache, it is immediately used in the current batch.

[0056] When performing joint adversarial training on the pre-trained virtual-real discriminator, policy network, and world model simulation system using real and second virtual trajectories, the parameters of the virtual-real discriminator and the world model simulation system are updated alternately; when updating the parameters of the world model simulation system, the parameters of the policy network are updated synchronously.

[0057] For example, please see Figure 4 The diagram shows the alternating update process, which includes steps D and G.

[0058] Step D involves freezing the policy network and the world model simulation system, and only updating the virtual-real discriminator. The G-step involves freezing the real / virtual discriminator, unfreezing the policy network and the world model simulation system, not updating the real / virtual discriminator, updating the world model simulation system, and synchronously updating the policy network.

[0059] Repeat the D-step and G-step process until the training termination condition for this phase is met, such as the loss has converged.

[0060] In this training phase, the virtual-real discriminator outputs the probability of the trajectory's authenticity during forward propagation. The intermediate features naturally form an embedding vector, which evolves synchronously with joint adversarial training, forming a differentiable trajectory metric space. This metric space can convert the quality of the trajectory into vector distance and can be backpropagated, thus achieving the effect of an evaluation metric, i.e., a loss function. Furthermore, it can link the training and evaluation operations, avoiding decoupling between the two.

[0061] The third stage is: freezing the real / virtual discriminator, generating a third virtual trajectory online through the policy network and world model simulation system, using the third virtual trajectory to fine-tune the policy network, and using the real probability output by the frozen real / virtual discriminator or its corresponding embedding vector as the reward signal during fine-tuning.

[0062] The world model includes not only a simulation system but also a metric system. After the first and second training phases are completed, the virtual-real discriminator can be used as a metric system within the world model. This metric system can be used for the quantitative evaluation of autonomous driving strategies and as a reward signal during the closed-loop training process of the autonomous driving system. Specifically, the virtual-real discriminator is used to calculate the probability of the input trajectory being real, and the calculation result of the virtual-real discriminator can serve as a metric.

[0063] Before starting the third phase of training, all trainable parameters of the virtual-real discriminator can be set to `requires_grad=False` to ensure that these parameters are not updated during this phase of training. During this phase, you can choose to freeze the world model simulation system or keep it thawed.

[0064] The third virtual trajectory refers to the virtual trajectory generated in the third stage. There are multiple third virtual trajectories, each generated in real time before each round of training iterations.

[0065] The operation of generating a third virtual trajectory online using a policy network and a world model simulation system includes: checking whether the trajectory generation conditions are met; the trajectory generation conditions refer to the completion of the update of the policy network parameters and the failure to meet the training termination conditions corresponding to the fine-tuning training; if the trajectory generation conditions are met, the policy network and the world model simulation system output vehicle control actions and system states are called in a loop, and the third virtual trajectory is generated based on the multi-frame vehicle control actions corresponding to different time steps output by the policy network and the multi-frame system states corresponding to different time steps output by the world model simulation system.

[0066] For example, the process of each training iteration (epoch) is similar to the process of generating the second virtual trajectory, and can be specifically as follows: (1) Initial state sampling, i.e., randomly sampling a single frame of system state from the actual driving record, can be represented as: ; (2) Perform online rolling operation (assuming the range of t is [1,T], T=300 steps, and the sensor sampling frequency is 10 Hz), that is: ① Set the system state at the current time step (e.g., time step t). Input the policy network to obtain the action distribution parameters for the current time step. Based on these action distribution parameters, sample the vehicle control actions for the current time step. ; ②System status and vehicle control actions Input the world model simulation system to obtain the system state at time step t+1. ; ③ Put Pushed into the GPU (Graphics Processing Unit) cache, it is immediately used in the current batch.

[0067] In this training phase, the real probability output by the frozen virtual-real discriminator or its corresponding embedding vector is used as the reward signal. Since the reward signal is from the same source as the metric space, there is no need to manually design heuristic metrics such as jerk or time-to-collision (TTC) to continuously optimize the policy (i.e., the parameters of the policy network) in the virtual environment. Simultaneously, because the metric space maintains the same distribution as the real domain, the Fréchet distance between the virtual and real trajectories can be significantly reduced (e.g., ...). Figure 5 As shown, the Frescher distance between the virtual trajectory and the real trajectory can be reduced from 12.7 to 2.3, which reduces the degree of deviation between the distribution of the virtual trajectory and the real trajectory.

[0068] In some embodiments, the following loss function is used during pre-training: ; D represents the virtual / real discriminant; Indicates both real and virtual losses; This indicates that the virtual / real discriminator distinguishes between the real and virtual trajectories. The probability of the output being true; This indicates that the virtual / real discriminator distinguishes between virtual trajectories. The probability of the output being true.

[0069] The following loss function is used when conducting joint adversarial training: ; This indicates a dynamic measure of learning loss; This indicates that the virtual / real discriminator distinguishes between the real and virtual trajectories. The probability of the output being true; This represents the expectation of the distribution of the true trajectory; This represents the virtual trajectory generated by the world model simulation system; This indicates that the virtual / real discriminator distinguishes between virtual trajectories. The probability of the output being true; This represents the expectation of the distribution of the virtual trajectory.

[0070] The following loss function is used during fine-tuning training: ;in, ; Indicate the strategy optimization objective; Representing the trajectory Embedded vectors in a differentiable metric space; This represents the system state at time step t; This represents the vehicle control action corresponding to time step t; Represents the action value function; Indicates the trajectory Calculate the expected system state for all time steps within the time interval; The feature weights represent the output of the frozen virtual / real discriminator for the system state at time step t. This represents the behavioral feature vector output by the feature extractor for the system state at time step t.

[0071] In this embodiment, the first stage uses a binary cross-entropy function as the loss function, allowing the real / virtual discriminator to quickly establish the initial boundary between real and virtual trajectories. The second stage employs a dynamic metric learning loss function, which only calculates the gradient of the trajectory's authenticity probability. However, since the gradient must pass through the behavioral feature vector (embedding vector) output by the feature extractor, this behavioral feature vector evolves synchronously with alternating optimization, naturally forming a differentiable trajectory metric space M. This reduces the Fréchet distance between virtual and real trajectories, decreasing the deviation in their distributions. The third stage freezes the real / virtual discriminator and directly uses the trajectory's embedding vector in the metric space as the external reward, thus completing ground-value-free reinforcement learning. This effectively improves the autonomous driving system's recognition rate of long-tailed abnormal scenarios, and eliminates the need for manually designed heuristic rewards such as jerk or TTC throughout the process.

[0072] In this embodiment, by using loss functions of the same origin but different applications in the three training stages, the autonomous driving system can still complete the construction of metric space, distribution alignment and continuous reward generation simultaneously even in the absence of truth labels, thus achieving end-to-end closed-loop training and quantitative evaluation.

[0073] It should be noted that, unless otherwise explicitly stated herein, the execution order of the steps included in the world model metric system construction method provided in any of the above embodiments is not strictly limited, and these steps can be executed in other orders. Moreover, at least some of these steps may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least a portion of the sub-steps or stages of other steps.

[0074] Based on the same inventive concept, this application also provides a system for constructing a world model measurement system. In some embodiments, such as Figure 6 As shown, the system includes the following modules: Module 110 is used to construct a policy network, a world model simulation system, and a virtual-real discriminator. The policy network is used to generate vehicle control actions for the current time step based on the system state at the current time step. The world model simulation system is used to generate the system state for the next time step based on the system state at the current time step and the vehicle control actions. The virtual-real discriminator is used to calculate the probability of the authenticity of the input trajectory, which can be either a real trajectory or a virtual trajectory. The phased training module 120 is used to freeze the policy network and the world model simulation system. It generates a first virtual trajectory offline using the policy network and the world model simulation system, and uses the real trajectory and the first virtual trajectory as positive and negative samples, respectively, to pre-train the virtual-real discriminator. It then unfreezes the policy network and the world model simulation system, generates a second virtual trajectory online using the policy network and the world model simulation system, and uses the real trajectory and the second virtual trajectory to perform joint adversarial training on the pre-trained virtual-real discriminator, the policy network, and the world model simulation system. Finally, it freezes the virtual-real discriminator, generates a third virtual trajectory online using the policy network and the world model simulation system, and uses the third virtual trajectory to fine-tune the policy network. During fine-tuning, the real probability output by the frozen virtual-real discriminator or its corresponding embedding vector is used as the reward signal.

[0075] In some embodiments, when the phased training module 120 performs offline rolling generation of the first virtual trajectory through the policy network and the world model simulation system, it is specifically used to cyclically call the frozen vehicle control actions and system states output by the policy network and the world model simulation system, and generate the first virtual trajectory based on the multi-frame vehicle control actions output by the policy network corresponding to different time steps and the multi-frame system states output by the world model simulation system corresponding to different time steps; wherein, in each cyclic call, if this cyclic call is the first cyclic call, the input of the policy network is a single frame system state randomly selected from the real driving record, and if this cyclic call is not the first cyclic call, the system state output by the world model simulation system in the previous cyclic call; in each cyclic call, the input of the world model simulation system is the vehicle control action output by the policy network in this cyclic call.

[0076] In some embodiments, when the phased training module 120 performs online rolling generation of the second virtual trajectory through the policy network and the world model simulation system, it is used to cyclically call the unfrozen policy network and the world model simulation system output vehicle control actions and system states before each round of iterative training, and generate the second virtual trajectory based on the multi-frame vehicle control actions output by the policy network corresponding to different time steps and the multi-frame system states output by the world model simulation system corresponding to different time steps.

[0077] In some embodiments, when the phased training module 120 performs online rolling generation of the third virtual trajectory through the policy network and the world model simulation system, it is specifically used to detect whether the trajectory generation conditions are met. The trajectory generation conditions refer to the completion of the parameter update of the policy network and the failure to meet the training termination condition corresponding to the fine-tuning training. If the trajectory generation conditions are met, the vehicle control actions and system states output by the policy network and the world model simulation system are called in a loop. The third virtual trajectory is generated based on the multi-frame vehicle control actions corresponding to different time steps output by the policy network and the multi-frame system states corresponding to different time steps output by the world model simulation system.

[0078] In some embodiments, the staged training module 120 uses the following loss function during pre-training: ; D represents the virtual / real discriminant; Indicates both real and virtual losses; This indicates that the virtual / real discriminator distinguishes between the real and virtual trajectories. The probability of the output being true; This indicates that the virtual / real discriminator distinguishes between virtual trajectories. The probability of the output being true.

[0079] The following loss function is used when performing joint adversarial training in the phased training module 120: ; This indicates a dynamic measure of learning loss; This indicates that the virtual / real discriminator distinguishes between the real and virtual trajectories. The probability of the output being true; This represents the expectation of the distribution of the true trajectory; This represents the virtual trajectory generated by the world model simulation system; This indicates that the virtual / real discriminator distinguishes between virtual trajectories. The probability of the output being true; This represents the expectation of the distribution of the virtual trajectory.

[0080] The following loss function is used when fine-tuning training module 120 in the phased training stage: ;in, ; Indicate the strategy optimization objective; Representing the trajectory Embedded vectors in a differentiable metric space; This represents the system state at time step t; This represents the vehicle control action corresponding to time step t; Represents the action value function; Indicates the trajectory Calculate the expected system state for all time steps within the time interval; The feature weights represent the output of the frozen virtual / real discriminator for the system state at time step t. This represents the behavioral feature vector output by the feature extractor for the system state at time step t.

[0081] In some embodiments, when the phased training module 120 performs joint adversarial training on the pre-trained virtual-real discriminator, policy network, and world model simulation system using real trajectories and second virtual trajectories, it alternately updates the parameters of the virtual-real discriminator and the world model simulation system; and, when updating the parameters of the world model simulation system, it synchronously updates the parameters of the policy network.

[0082] In some embodiments, when the phased training module 120 performs fine-tuning training, it freezes the world model simulation system or keeps the world model simulation system in a thawed state.

[0083] Specific limitations regarding the construction system of the world model metric system can be found in the limitations on the construction method of the world model metric system above, and will not be repeated here. Each module in the aforementioned world model metric system construction system can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device in hardware form, or stored in the memory of a computer device in software form, so that the processor can call and execute the corresponding operations of each module.

[0084] This application also provides a computer device. In some embodiments, the computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it can implement the steps of the world model measurement system construction method provided in any of the above embodiments.

[0085] In some embodiments, the internal structure diagram of a computer device may be as follows: Figure 7 As shown, the computer device includes a processor, memory, and a network interface connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores an operating system, computer programs, and a database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The database stores data related to autonomous driving training, such as real-world road test data; the specific data stored can also be found in the limitations defined in the above method embodiments. The network interface is used to communicate with external terminals via a network connection. When the computer program is executed by the processor, it implements a method for constructing a world model metric system.

[0086] Those skilled in the art will understand that Figure 7 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0087] This application also provides a computer-readable storage medium, in some embodiments of which a computer program is stored on the computer-readable storage medium, and when the computer program is executed by a processor, it implements the steps of the world model measurement system construction method provided in any of the above embodiments.

[0088] In the above embodiments of this application, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0089] Those skilled in the art will understand that implementing all or part of the processes in the above method embodiments can be accomplished by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), Synchlink, DRAM (SLDRAM), memory bus, direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0090] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0091] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

Claims

1. A method for constructing a world model metric system, characterized in that, The method includes: A policy network, a world model simulation system, and a virtual-real discriminant are constructed. The policy network is used to generate vehicle control actions for the current time step based on the system state at the current time step. The world model simulation system is used to generate the system state for the next time step based on the system state at the current time step and the vehicle control actions. The virtual-real discriminant is used to calculate the probability of the authenticity of the input trajectory, which can be a real trajectory or a virtual trajectory. The policy network and the world model simulation system are frozen. A first virtual trajectory is generated offline using the policy network and the world model simulation system. The real trajectory and the first virtual trajectory are used as positive and negative samples, respectively, to pre-train the virtual-real discriminator. The policy network and the world model simulation system are unfrozen. A second virtual trajectory is generated online using the policy network and the world model simulation system. The real trajectory and the second virtual trajectory are used to perform joint adversarial training on the pre-trained virtual-real discriminator, the policy network, and the world model simulation system. The virtual-real discriminator is frozen. A third virtual trajectory is generated online using the policy network and the world model simulation system. The policy network is fine-tuned using the third virtual trajectory. During the fine-tuning, the real probability output by the frozen virtual-real discriminator or its corresponding embedding vector is used as the reward signal.

2. The method as described in claim 1, characterized in that, The first virtual trajectory is generated offline via the policy network and the world model simulation system, including: The frozen policy network and the world model simulation system output vehicle control actions and system states are called in a loop. Based on the multi-frame vehicle control actions corresponding to different time steps output by the policy network and the multi-frame system states corresponding to different time steps output by the world model simulation system, a first virtual trajectory is generated. In each loop call, if this loop call is the first loop call, the input of the policy network is a single frame of system state randomly selected from the real driving record; if this loop call is not the first loop call, the system state output by the world model simulation system in the previous loop call is used. In each loop call, the input of the world model simulation system is the vehicle control action output by the policy network in this loop call.

3. The method as described in claim 1, characterized in that, The second virtual trajectory is generated online in a rolling manner through the policy network and the world model simulation system, including: Before each round of iterative training, the unfrozen policy network and the world model simulation system output vehicle control actions and system states are called in a loop. Based on the multi-frame vehicle control actions output by the policy network corresponding to different time steps and the multi-frame system states output by the world model simulation system corresponding to different time steps, a second virtual trajectory is generated.

4. The method as described in claim 1, characterized in that, The third virtual trajectory is generated online in a rolling manner through the policy network and the world model simulation system, including: Check whether the trajectory generation conditions are met; the trajectory generation conditions refer to the completion of the parameter update of the policy network and the failure to meet the training termination condition corresponding to the fine-tuning training. If the trajectory generation conditions are met, the vehicle control actions and system states output by the policy network and the world model simulation system are called in a loop. Based on the multi-frame vehicle control actions corresponding to different time steps output by the policy network and the multi-frame system states corresponding to different time steps output by the world model simulation system, a third virtual trajectory is generated.

5. The method as described in claim 1, characterized in that, The following loss function is used during the pre-training process: ； D represents the virtual / real discriminant; Indicates both real and virtual losses; This indicates that the virtual / real discriminator distinguishes between the real and virtual trajectories. The probability of the output being true; This indicates that the virtual / real discriminator distinguishes between virtual trajectories. The probability of the output being true; The following loss function is used when performing the joint adversarial training: ； This indicates a dynamic measure of learning loss; This indicates that the virtual / real discriminator distinguishes between the real and virtual trajectories. The probability of the output being true; This represents the expectation of the distribution of the true trajectory; This represents the virtual trajectory generated by the world model simulation system; This indicates that the virtual / real discriminator distinguishes between virtual trajectories. The probability of the output being true; This represents the expectation of the distribution of the virtual trajectory; The following loss function is used during the fine-tuning training: ;in, ; Indicate the strategy optimization objective; Representing the trajectory Embedded vectors in a differentiable metric space; This represents the system state at time step t; This represents the vehicle control action corresponding to time step t; Represents the action value function; Indicates the trajectory Calculate the expected system state for all time steps within the time interval; The feature weights represent the output of the frozen virtual / real discriminator for the system state at time step t. This represents the behavioral feature vector output by the feature extractor for the system state at time step t.

6. The method as described in claim 1, characterized in that, When performing joint adversarial training on the pre-trained virtual-real discriminator, the policy network, and the world model simulation system using the real trajectory and the second virtual trajectory, the parameters of the virtual-real discriminator and the world model simulation system are updated alternately; when updating the parameters of the world model simulation system, the parameters of the policy network are updated synchronously.

7. The method as described in claim 1, characterized in that, During the fine-tuning training, the world model simulation system is either frozen or kept thawed.

8. A world model metric system construction system, characterized in that, The system includes: The module is used to construct a policy network, a world model simulation system, and a virtual-real discriminator. The policy network is used to generate vehicle control actions for the current time step based on the system state at the current time step. The world model simulation system is used to generate the system state for the next time step based on the system state at the current time step and the vehicle control actions. The virtual-real discriminator is used to calculate the probability of the authenticity of the input trajectory, which can be a real trajectory or a virtual trajectory. The phased training module is used to freeze the policy network and the world model simulation system, generate a first virtual trajectory offline using the policy network and the world model simulation system, and use the real trajectory and the first virtual trajectory as positive and negative samples, respectively, to pre-train the virtual-real discriminator; unfreeze the policy network and the world model simulation system, generate a second virtual trajectory online using the policy network and the world model simulation system, and use the real trajectory and the second virtual trajectory to perform joint adversarial training on the pre-trained virtual-real discriminator, the policy network, and the world model simulation system; freeze the virtual-real discriminator, generate a third virtual trajectory online using the policy network and the world model simulation system, and use the third virtual trajectory to fine-tune the policy network, using the real probability output by the frozen virtual-real discriminator or its corresponding embedding vector as the reward signal during the fine-tuning.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 7.

10. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 7.

Images