Virtual-real fusion digital twinborn simulation method and system for incomplete data perception

Abstract

The invention belongs to the field of virtual-real fusion digital twinning, and particularly relates to a virtual-real fusion digital twinning simulation method and system based on incomplete data perception. The method specifically comprises the following steps: if an obtained historical track of a certain entity lacks track data from a current time point to a certain historical time point, taking the segment as a track segment to be predicted; and inputting environment information including a known trajectory segment of the entity and historical trajectory data of other agents around the entity within the period of time as noise distribution for generating a trajectory initial value into the trained diffusion network, determining a noiseless predicted value of a trajectory segment to be predicted in the iteration according to output, and combining with the last generated trajectory to obtain a noiseless predicted value of the trajectory segment to be predicted. Obtaining a generated track after current denoising; repeatedly iterating to obtain a generation track of the last iteration; and carrying out the above processes at least twice, and obtaining a complete historical track of the entity through the motion area corresponding to at least two generated tracks of the last iteration in combination with known track segment data.

Description

Technical Field

[0001] This invention belongs to the field of virtual-real fusion digital twins, specifically relating to a virtual-real fusion digital twin simulation method and system with incomplete data perception. Background Technology

[0002] In existing virtual-real fusion scene simulation platforms, intelligent agents can be divided into two categories: physical agents and virtual agents. Physical agents refer to individuals with actual physical models in the scene, such as unmanned vehicle models, drone models, and robots. These entities typically have a decisive influence on the dynamic evolution of the scene. Virtual agents, on the other hand, refer to individuals that exist only within the current scene's virtual twin platform, such as virtual individuals that follow or guide the movement of physical agents (e.g., vehicles, aircraft), or virtual individuals that follow groups of robots.

[0003] In existing simulation platforms, physical entities are typically unaware of the existence of virtual entities, while virtual entities can acquire information about other virtual entities and physical entities in real time and need to dynamically adjust their own decisions based on changes in the state of physical entities. To ensure that the simulation of virtual entities can adapt to the dynamic evolution of the scene, virtual entities need to acquire the actual data of physical entities (such as trajectory data) in a timely manner. However, since physical entity data needs to be transmitted to the data platform via a data link, information transmission delays or periodic data loss often occur, affecting the generation of virtual entity motion trajectories. Summary of the Invention

[0004] The purpose of this invention is to provide a virtual-real fusion digital twin simulation method and system with incomplete data perception, which is used to solve the problem that the information transmission delay or phased data loss of entity data in the existing virtual-real fusion scene simulation platform affects the generation of virtual body motion trajectory.

[0005] To achieve the above objectives, the present invention provides a virtual-real fusion digital twin simulation method with incomplete data perception, comprising: if the historical trajectory of an entity is missing trajectory data between the current time point and a certain historical time point, then the trajectory segment with missing trajectory data is taken as the trajectory segment to be predicted; environmental information including the known trajectory segment of the entity and the historical trajectory data of other intelligent agents within a set distance around the entity during the time of the known trajectory segment, and noise distribution as the initial value for generating the trajectory are all input into a trained diffusion network; and the noise-free prediction value of the trajectory segment to be predicted in this iteration is determined based on the output.

[0006] The generated trajectory after denoising is obtained by comparing the predicted value of this iteration with the generated trajectory of the previous iteration; in the next iteration, the generated trajectory and environmental information are input into the diffusion network; the iteration is repeated to obtain the generated trajectory of the last iteration; the process of obtaining the generated trajectory of the last iteration is performed at least twice, and the corresponding motion region is determined by the at least two generated trajectories of the last iteration; the complete historical trajectory of the entity is obtained by combining the motion region with the known trajectory segment data.

[0007] The complete historical trajectory of the entity is included as part of the environmental information of virtual objects within a set distance around the entity, and is used in the calculation when simulating the trajectory of the virtual object.

[0008] Furthermore, the complete historical trajectory of the entity is included as part of the environmental information of virtual bodies within a set distance around the entity, and its calculation is performed when simulating the trajectory of the virtual body.

[0009] For a virtual body whose trajectory is to be simulated, its future trajectory is taken as the trajectory segment to be predicted. The environmental information, including the known trajectory segment of the virtual body and the historical trajectory data of other intelligent agents within a set distance around the virtual body during the time of the known trajectory segment data, as well as the noise distribution as the initial value for generating the trajectory, are all input into the trained diffusion network. The process is repeated iteratively, and the future trajectory data of the virtual body is determined based on the generated trajectory obtained in the last iteration.

[0010] Furthermore, it also includes: if, within the time window of the acquired historical trajectory of an entity, only the trajectory data of time points from the start of the time window to a certain historical time point is missing, then the trajectory data of the missing time point with the set missing value is taken as the complete historical trajectory data of the entity.

[0011] The complete historical trajectory of the entity is included as part of the environmental information of virtual objects within a set distance around the entity, and is used in the calculation when simulating the trajectory of the virtual object.

[0012] Furthermore, the environmental information, which includes the known trajectory segment of the entity and the historical trajectory data of other intelligent agents within a set distance around the entity during the time of the known trajectory segment, also includes the data state corresponding to each time point in the known trajectory segment. The data state includes the state with no missing trajectory data, the state with missing trajectory data, and the state corresponding to the motion area.

[0013] Furthermore, the trajectory data at each time point includes the target location corresponding to that trajectory;

[0014] The methods for obtaining the denoised generated trajectory by comparing the predicted value of this iteration with the generated trajectory of the previous iteration include:

[0015] Using the total guided loss of the predicted values ​​in this iteration, the gradients corresponding to the predicted values ​​in this iteration and the generated trajectory in the previous iteration are calculated respectively. Based on the predicted values ​​and the gradients corresponding to the generated trajectory, the predicted values ​​in this iteration and the generated trajectory in the previous iteration are updated respectively. Using the updated predicted values ​​and the generated trajectory in the previous iteration, the denoised generated trajectory is obtained.

[0016] The total guidance loss includes the guidance loss term of the target consistency loss; the target consistency loss in this iteration is obtained by the difference between the target position at each time point in the noiseless prediction value of the trajectory segment to be predicted in this iteration and the corresponding target position setting value in the known trajectory segment.

[0017] Furthermore, the total guiding loss also includes at least one of the following guiding loss items: the guiding loss item of grouping loss, the guiding loss item of following loss, and the guiding loss item of obstacle avoidance loss.

[0018] Furthermore, the methods for determining the corresponding motion region using at least two generated trajectories from the last iteration include:

[0019] Based on the position information in the trajectory data of at least two generated trajectories from the last iteration at each time point, the corresponding position points are determined; based on the minimum adjacency graph of each position point, the possible motion region of the entity in the missing part of the historical trajectory is determined as the corresponding motion region; the shape of the adjacency graph is a set planar shape.

[0020] Furthermore, it also includes: during the training of the diffusion network, the environmental information used includes the historical trajectory data of other agents within a set distance around the agent during the time period of the known trajectory segment data; if there is a missing trajectory data between the current time point and a certain historical time point in the historical trajectory data of an entity, the current diffusion model is first used to obtain at least two generated trajectories from the last iteration for the missing part, and the corresponding motion region is determined; the motion region is combined with the known trajectory segment data to obtain the complete historical trajectory of the entity; and the complete historical trajectory of the entity is then used as the historical trajectory data of the entity to construct the environmental information.

[0021] The above-described technical solution of this invention provides a novel method for virtual-real fusion digital twin simulation with incomplete data perception, the beneficial effects of which include:

[0022] When an entity's historical trajectory data is missing in the latter part, making its current position and motion information unknown, guided by environmental information including the entity's known trajectory segment and the historical trajectory data of other intelligent agents around the entity within the known trajectory segment, the trajectory segment with missing trajectory data is used as the prediction object. By using a diffusion model to make more than two predictions, two or more predicted trajectory data are obtained, which determines the possible motion area of ​​the entity within the time corresponding to that trajectory segment. This is equivalent to obtaining the entity's missing trajectory information relatively accurately through prediction. By using this motion area to fill in the missing part of the entity's historical trajectory, more complete entity data can be obtained for the trajectory simulation of nearby virtual objects, thereby effectively reducing the impact of missing entity trajectory data on the simulation of virtual object trajectories.

[0023] When an entity's historical trajectory lacks data for the latter part, making its current position and motion information unknown, guided by environmental information including the entity's known trajectory segment and the historical trajectory data of other intelligent agents surrounding the entity within that known trajectory segment, the missing trajectory segment is used as the prediction object. Through two or more predictions by the diffusion model, two or more trajectory data points are obtained, determining the entity's possible motion region within the corresponding time period. This motion region is then used to complete the missing portion of the entity's historical trajectory, thus obtaining more complete entity data. Furthermore, when a virtual entity generates a scene, if the historical trajectory of nearby entities lacks the latter half of its data, the completed entity historical trajectory data can be used as one of the environmental information to guide the diffusion model to obtain the virtual entity's future trajectory data, effectively reducing the impact of missing entity trajectory data on the generation of the virtual entity's trajectory. Moreover, the principles of entity trajectory completion and virtual entity scene generation are the same, allowing the use of the same trained diffusion model, saving training costs.

[0024] The present invention also provides a virtual-real fusion digital twin simulation system with incomplete data perception, including a processor, the processor being used to execute program instructions, the program instructions being used to implement the above-described virtual-real fusion digital twin simulation method with incomplete data perception.

[0025] The technical solution of the virtual-real fusion digital twin simulation system with incomplete data perception described above in this invention can achieve the same beneficial effects as the virtual-real fusion digital twin simulation method with incomplete data perception described above. Attached Figure Description

[0026] Figure 1 This is a flowchart illustrating the non-complete data-aware virtual-real fusion digital twin simulation method in the implementation of the non-complete data-aware virtual-real fusion digital twin simulation method of the present invention.

[0027] Figure 2This is a schematic diagram illustrating the principle of the virtual-real fusion digital twin simulation method with incomplete data perception, as described in the implementation of the incomplete data perception method of the present invention. Detailed Implementation

[0028] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments.

[0029] Implementation of a Virtual-Real Fusion Digital Twin Simulation Method with Incomplete Data Awareness

[0030] This embodiment presents a technical solution for a virtual-real fusion digital twin simulation method with incomplete data perception. When the latter part of the historical trajectory data of an entity is missing, resulting in the current position and motion information being unknown, this virtual-real fusion digital twin simulation method uses a diffusion model to predict the missing latter half of the historical trajectory based on the unmissing first half of the historical trajectory to fill in the missing data. Then, the filled-in data is used as environmental information for predicting the future trajectory of nearby virtual entities, thereby avoiding the impact of missing entity data on the generation of virtual entity trajectories.

[0031] Reference Figure 1 The virtual-real fusion digital twin simulation method specifically includes: if the historical trajectory of an entity is missing trajectory data between the current time point and a certain historical time point, then the trajectory segment with missing trajectory data is taken as the trajectory segment to be predicted. The environmental information containing the known trajectory segment of the entity and the historical trajectory data of other intelligent agents within a set distance around the entity during the time of the known trajectory segment, as well as the noise distribution as the initial value for generating the trajectory, are all input into the trained diffusion network. Based on the output, the noise-free prediction value of the trajectory segment to be predicted in this iteration is determined.

[0032] The generated trajectory after denoising is obtained by comparing the predicted value of this iteration with the generated trajectory of the previous iteration; in the next iteration, the generated trajectory and environmental information are input into the diffusion network; the iteration is repeated to obtain the generated trajectory of the last iteration; the process of obtaining the generated trajectory of the last iteration is performed at least twice, and the corresponding motion region is determined by the at least two generated trajectories of the last iteration; the complete historical trajectory of the entity is obtained by combining the motion region with the known trajectory segment data; the complete historical trajectory of the entity is used as the historical trajectory data of the entity.

[0033] The complete historical trajectory of the entity is included as part of the environmental information of virtual objects within a set distance around the entity, and is used in the calculation when simulating the trajectory of the virtual object.

[0034] Therefore, this virtual-real fusion digital twin simulation method addresses the situation where the historical trajectory of an entity is missing a segment, resulting in an unknown current position and motion information. Guided by environmental information including the entity's known trajectory segment and the historical trajectory data of other intelligent agents within a set distance around the entity during the known trajectory segment, the method uses the missing trajectory segment as the prediction object and iteratively obtains predicted trajectory data through a diffusion model prediction. This process is repeated more than twice to obtain two or more possible trajectories. By using two or more possible trajectory data, the possible motion area of ​​the entity within the corresponding time segment of the trajectory is determined, which can accurately compensate for the entity's missing trajectory information. Using this motion area to complete the missing part of the entity's historical trajectory yields more complete entity data. Based on this, when a virtual entity generates its trajectory, if the historical trajectory of a nearby entity is missing a segment, the completed historical trajectory data of the entity can be used as one of the environmental information elements. Using the virtual entity's predicted trajectory segment as the prediction object, the diffusion model is guided to obtain the virtual entity's future trajectory data, thereby effectively reducing the impact of missing entity trajectory data on the generation of the virtual entity's trajectory.

[0035] The complete historical trajectory of the entity is included in the environmental information of virtual bodies within a set distance around the entity. The methods used in simulating the trajectory of this virtual body include:

[0036] For a virtual body whose trajectory is to be simulated, its future trajectory is taken as the trajectory segment to be predicted. The environmental information, including the known trajectory segment of the virtual body and the historical trajectory data of other intelligent agents within a set distance around the virtual body during the time of the known trajectory segment data, as well as the noise distribution as the initial value for generating the trajectory, are all input into the trained diffusion network. The process is repeated iteratively, and the future trajectory data of the virtual body is determined based on the generated trajectory obtained in the last iteration.

[0037] It is evident that trajectory completion for entities with incomplete data (i.e., obtaining the complete historical trajectory of the entity) and trajectory generation for virtual entities (i.e., determining the future trajectory data of the virtual entity) are based on the same principle and can share the same trained diffusion model, thus saving training costs.

[0038] Furthermore, in this embodiment, the method for determining the corresponding motion region using at least two generated trajectories from the last iteration includes:

[0039] Based on the position information in the trajectory data of at least two generated trajectories from the last iteration at each time point, the corresponding position points are determined; based on the minimum adjacency graph of each position point, the possible motion region of the entity (agent) in the missing part of the historical trajectory is determined as the corresponding motion region; the shape of the adjacency graph here is a predefined planar shape.

[0040] In one embodiment, if the shape of the adjacency graph is a rectangle, then firstly, based on the position information in the trajectory data of at least two generated trajectories from the last iteration at each time point, the corresponding position points are determined, and then the minimum adjacency rectangle of each position point is obtained (these position points contain the position information in the trajectory data of all generated trajectories from the last iteration at each time point; for example, if three generated trajectories from the last iteration are obtained, the determined position points contain the position information in the trajectory data of these three generated trajectories from the last iteration at each time point, and then the minimum adjacency rectangle is obtained through all position points). This minimum adjacency rectangle is used as the motion area corresponding to the generated trajectory. In other embodiments, other adjacency polygons or adjacency circles can also be used as the shape of the adjacency graph.

[0041] Under normal circumstances, the trajectory data at each time point mainly contains the information of position p, velocity v, yaw angle yaw, acceleration acc, and angular velocity yawvel; however, in order to ensure that the predicted trajectory can still accurately reflect the expected target corresponding to the predicted trajectory as time goes by, in this embodiment, the trajectory data at each time point also includes the target position corresponding to the trajectory.

[0042] The methods for obtaining the denoised generated trajectory by comparing the predicted value of this iteration with the generated trajectory of the previous iteration include:

[0043] Using the total guided loss of the predicted values ​​in this iteration, the gradients corresponding to the predicted values ​​in this iteration and the generated trajectory in the previous iteration are calculated respectively. Based on the gradients corresponding to the predicted values ​​in this iteration and the generated trajectory in the previous iteration, the predicted values ​​in this iteration and the generated trajectory in the previous iteration are updated respectively. Using the updated predicted values ​​in this iteration and the generated trajectory in the previous iteration, the denoised generated trajectory is obtained.

[0044] The total guidance loss includes the guidance loss term of the target consistency loss; the target consistency loss in this iteration is obtained by the difference between the target position at each time point in the noiseless prediction value of the trajectory segment to be predicted in this iteration and the corresponding target position setting value in the known trajectory segment.

[0045] Therefore, the target position corresponding to the trajectory is added to the trajectory data, serving as a noise-free prediction value for the trajectory segment to be predicted obtained through the diffusion network. Part of this, the target location setting is used as the target location label (for a single trajectory of a single agent, the target location setting is the same at each time point, but the predicted value obtained through the diffusion network is different). The target location may differ at different times (the target location may vary), and is determined by the target location label and... The difference between the target positions corresponding to the mid-trajectory can drive the predicted value. Optimize in a direction that accurately reflects the corresponding expected goals over time; the specific optimization method is to combine the target location label with... The guided loss term, derived from the target consistency loss obtained by considering the difference between the target positions corresponding to the mid-trajectory, is used as the guided loss term in the total guided loss function (i.e., the total guided loss of the predicted values ​​in this iteration), and is further used in optimizing and updating the predicted trajectory. At the same time, it also incorporates target consistency loss to generate trajectories. Optimizations were made to effectively improve the guidance accuracy of scene guidance and the diffusion network's ability to perceive targets.

[0046] It is important to note that the "guidance" mentioned in the total guidance loss refers to the reverse diffusion process of guidance, that is, sampling, or inference, or the process of using the model to perform a prediction task. Therefore, the total guidance loss is not the loss of training the model during training.

[0047] In other implementations, the total guidance loss function is determined based on the actual guidance requirements of the scenario. It may also include at least one of the following: a guiding loss term for grouping loss, a guiding loss term for following loss, and a guiding loss term for obstacle avoidance loss. Since these guiding loss terms are already existing in the prior art, they will not be described in detail here.

[0048] In summary, in one embodiment of this implementation, for the time window (i.e., the time window w corresponding to the historical trajectory of the entity used as one of the environmental information to determine the future trajectory data of the virtual body), h In cases where an entity's historical trajectory lacks data between the current time point and a certain historical time point (i.e., the latter part of the trajectory is missing), the process of obtaining the entity's complete historical trajectory is as follows:

[0049] First, using the time window w h The trajectory segments with missing data are the trajectory segments to be predicted; random Gaussian noise τ is used. T ~N(0,1) is used as the initial value for generating the trajectory (other noise distributions, such as salt-and-pepper noise, can be used in other embodiments). This, along with environmental information including the known trajectory segment containing the entity and the historical trajectory data of other agents within a set distance around the entity during the time of the known trajectory segment, is input into the trained diffusion network. Based on the output of the diffusion network, the noise-free prediction value of the trajectory segment to be predicted in this iteration is obtained. The formula corresponding to the principle of the generated trajectory after denoising is as follows:

[0050]

[0051] Where, τ tThat is, the input trajectory containing t steps of noise, τ t-1 The trajectory obtained after one step of inverse diffusion denoising process contains t-1 steps of noise, where t∈[1,T]; β t This characterizes the degree of noise introduced at each step in the forward diffusion process; we define α as... t =1-β t Then there is I is the identity matrix that participates in the formation of the covariance matrix; in this embodiment, the initial value for generating the trajectory is Gaussian noise τ. T ~N(0,1); The noise-free prediction value of the trajectory segment to be predicted in this iteration can be obtained from the output of the diffusion network. The predicted value of this iteration can also be called the predicted trajectory. Specifically, the methods to obtain the noise-free predicted value of the trajectory segment to be predicted in this iteration based on the output of the diffusion network are mainly divided into two types: directly predicting the noise-free trajectory through the diffusion network, and predicting the noise of the trajectory through the diffusion network and then denoising based on the predicted noise. Both of these are existing technologies and will not be elaborated here.

[0052] In one specific embodiment, the predicted noiseless trajectory is used during training. The loss is calculated with the label τ0, backpropagated to the diffusion model parameters, and the parameters are optimized.

[0053] During the inference phase (i.e., when using the trained diffusion model), a guidance mechanism is needed to continuously optimize the model's output, gradually reducing noise in the desired direction. Specifically, the total guidance loss function is used to calculate the noise-free trajectory values ​​predicted in each iteration. (i.e., the predicted value in this iteration), the trajectory prediction value containing t steps of noise obtained in the previous denoising process. (i.e., the gradient corresponding to the generated trajectory obtained in the previous iteration, which can be simply referred to as the generated trajectory of the previous iteration), and the gradient corresponding to the predicted trajectory value without noise. Gradient of trajectory prediction with t-step noise The trajectory predictions without noise and those containing t-step noise are updated to obtain the updated predictions for this iteration. The generated trajectory obtained from the updated previous iteration This process can be represented by the following formula:

[0054]

[0055] Using the updated and As τ0 and τ respectively t Substituting p(τ) into the above... t-1 |τ t In the formula, the distribution is calculated. The mean and variance of the distribution can be used to sample the generated trajectory after denoising. Will The environmental information, including the known trajectory segment containing the entity and the historical trajectory data of other agents within a set distance around the entity during the time of the known trajectory segment, is then input into the trained diffusion network. Based on the output of the diffusion network, the noise-free prediction value of the trajectory segment to be predicted in the next iteration is obtained. Finally, the optimized value for the next iteration is obtained through the total guided loss function. and The value is obtained by iterating in this way, repeating t-1 times, until the generated trajectory of the last iteration is obtained, which is used as the final predicted trajectory τ0. i .

[0056] Repeat this process multiple times to obtain n (n≥2) possible final predicted trajectories. (That is, at least two generated trajectories from the last iteration are obtained). By merging the corresponding motion region determined by the sequence of the final predicted trajectory with the known trajectory segment data of the entity, the complete historical trajectory of the entity can be obtained.

[0057] In one embodiment of this implementation, the training method of the diffusion network is as follows:

[0058] For each iteration step of training the diffusion network, Gaussian noise ∈ ~N(0,1) is first randomly sampled during the forward diffusion process, and then superimposed onto the clean future trajectory τ0 set in the training samples of this iteration using the following formula:

[0059]

[0060] Where, β t This characterizes the degree of noise introduced at each step in the forward diffusion process; let α t =1-β t Then there is This represents the cumulative effect after the t-th addition of noise. By adjusting the value of t, trajectories τ with varying degrees of added noise can be generated. t This can be referred to as a noise trajectory.

[0061] Subsequently, during the reverse diffusion process, the noise trajectory τ after superimposing noise from the forward diffusion process is... t The environmental information input in the training samples of this iteration uses a one-dimensional temporal UNet as the backbone network of the diffusion network to be trained (in other embodiments, the network structure of other existing diffusion models can be used). Based on the output of the diffusion network to be trained, the predicted noiseless future trajectory is obtained, that is, the noiseless prediction value of the trajectory segment to be predicted in this iteration is determined. In this embodiment, during the training of the diffusion network, the environmental information includes the historical trajectory data of other agents within a set distance around the agent during the time period when the trajectory segment data is known. If there is a missing historical trajectory data of an entity between the current time point and a certain historical time point, the currently trained diffusion model will first be used to predict possible motion regions for the missing parts (i.e., obtain at least two generated trajectories from the last iteration for the missing parts and determine the corresponding motion regions) to complete the historical trajectory data. That is, the complete historical trajectory of the entity is obtained by combining the motion region with the known trajectory segment data. Then, the complete historical trajectory of the entity is used as the historical trajectory data of the entity to participate in the construction of the environmental information.

[0062] Then you can use the predicted trajectory The loss is calculated by comparing the difference between the predicted result corresponding to the sample and the set clean future trajectory τ0 (equivalent to the sample label). Therefore, based on this loss Optimize the network parameters of the diffusion network. Since the principles of the forward diffusion and reverse diffusion processes described above are existing technologies, they will not be elaborated upon here. Furthermore, during the training process of the diffusion network, apart from completing the historical trajectory data for entities with missing trajectory data between the current time point and a certain historical time point, it is unnecessary to adjust the predicted values ​​using the total guiding loss. Perform the update operation.

[0063] Additionally, refer to Figure 2 In this embodiment, for the time window (i.e., the time window w corresponding to the historical trajectory of the entity used as one of the environmental information to determine the future trajectory data of the virtual body), h In the case of missing first half of the trajectory, the virtual-real fusion digital twin simulation method in this embodiment further includes: if the trajectory data of the time point from the start time of the time window to a certain historical time point is missing within the time window of the acquired historical trajectory of a certain entity (i.e., the first half of the trajectory is missing), then the trajectory data of the missing time point is taken as the historical trajectory after setting the missing value as the complete historical trajectory data of the entity.

[0064] The complete historical trajectory of the entity is included as part of the environmental information of virtual objects within a set distance around the entity, and is used in the calculation when simulating the trajectory of the virtual object.

[0065] In one embodiment, missing values ​​are set to all zeros, meaning all trajectory data at missing time points are set to 0; in other embodiments, different missing values ​​can be set according to other requirements.

[0066] That is, for an entity in the past time window w hIn cases where historical trajectories within a given time window are missing, the virtual-real fusion digital twin simulation method of this implementation employs classification processing. If the trajectory exists in the latter part of the time window but the trajectory in the former part is missing, the latter part of the trajectory is used to describe the motion state of the entity. However, if the trajectory in the latter part of the time window is missing, resulting in the entity's current position and motion information being unknown, the missing part is sampled at least twice based on a pre-trained diffusion model to generate a possible motion region A. h (That is, the process of generating the trajectory of the last iteration is performed at least twice, and the corresponding motion region is determined by the at least two generated trajectories of the last iteration.) Subsequently, the motion region is merged with the normal trajectory to form the complete historical information of the entity.

[0067] Therefore, this virtual-real fusion digital twin simulation method fully considers the possibility that the first half of the historical data of one or more entities may be missing (i.e., the first half of the trajectory is missing) during the actual simulation process, which has little impact on the trajectory generation of the virtual entity. Therefore, for this situation, a processing method that does not require data prediction and completion is selected, which simplifies the processing method of entity trajectories with missing historical data in some working conditions and saves processing costs.

[0068] Since the motion area belongs to the historical data that is supplemented by prediction, and the missing historical data in the first half is not supplemented by prediction, but the missing part is directly set with a missing value, in order to distinguish the historical data of different situations in the historical trajectory of the entity, the environmental information includes the known trajectory segment of the entity and the historical trajectory data of other intelligent agents within a set distance around the entity during the time of the known trajectory segment. It also includes the data status corresponding to each time point in the known trajectory segment. The data status includes the status of the trajectory data without missing data, the status of the trajectory data with missing data, and the status corresponding to the motion area.

[0069] In one embodiment of this implementation, the data state corresponding to each time point is added to the environmental information by superimposing a historical mask corresponding to the data state at each time point. The historical masks for the state with no missing trajectory data, the state with missing trajectory data, and the state corresponding to the movement area are respectively assigned different values, for example, 1, 0, and -1. In other embodiments, other forms can also be used to represent the data state corresponding to each time point.

[0070] In one embodiment of this implementation, after processing the historical trajectory data of each entity that has missing information, the specific process for simulating the trajectory of a virtual entity whose trajectory is to be simulated is as follows:

[0071] The future trajectory of the virtual entity is taken as the trajectory segment to be predicted. The environmental information of the known trajectory segment containing the virtual entity and the historical trajectory data of other intelligent entities within a set distance around the virtual entity during the time period of the known trajectory segment data, and the noise distribution (in this embodiment, random Gaussian noise τ) are used as the initial value for generating the trajectory. T ~N(0,1) is used as the initial value for generating the trajectory. In other embodiments, other noise distributions can be used, such as salt-and-pepper noise, etc., and all of them are input into the trained diffusion network.

[0072] The noise-free prediction value of the trajectory segment to be predicted in this iteration is obtained based on the output of the diffusion network. Then, using the overall guiding loss function, the noise-free trajectory values ​​predicted in this iteration are calculated respectively. (i.e., the predicted value in this iteration), the trajectory prediction value containing t steps of noise obtained in the previous denoising process. (i.e., the gradient corresponding to the generated trajectory obtained in the previous iteration, which can be simply referred to as the generated trajectory of the previous iteration), and the gradient corresponding to the predicted trajectory value without noise. Gradient of trajectory prediction with t-step noise The trajectory predictions without noise and those containing t-step noise are updated to obtain the updated predictions for this iteration. The generated trajectory obtained from the updated previous iteration Using the updated predicted value from this iteration and the previously generated trajectory, substitute it into the above p(τ) t-1 |τ t The formula is used to obtain the generated trajectory after denoising.

[0073] Will The environmental information, including the known trajectory segment containing the virtual object and the historical trajectory data of other intelligent objects within a set distance around the virtual object during the time of the known trajectory segment, is then input into the trained diffusion network. Based on the output of the diffusion network, the noise-free prediction value of the trajectory segment to be predicted in the next iteration is obtained. Then, the prediction value of the current iteration and the generated trajectory value relative to the previous iteration are updated by the total loss function. This process is repeated t-1 times until the generated trajectory of the last iteration is obtained. The future trajectory data of the virtual object is determined based on the generated trajectory obtained in the last iteration.

[0074] It is evident that this virtual-real fusion digital twin simulation method with incomplete data perception can address the practical problems encountered by virtual-real fusion digital twin platforms related to missing entity data, achieve complete closed-loop operation, and improve the matching degree between the trajectory data of the generated virtual body and the environment.

[0075] Implementation of a Virtual-Real Fusion Digital Twin Simulation System with Incomplete Data Awareness

[0076] This embodiment provides a technical solution for a virtual-real fusion digital twin simulation system with incomplete data perception. The solution specifically includes a processor, which executes program instructions to implement the virtual-real fusion digital twin simulation method with incomplete data perception as described above.

[0077] Since the specific working principle of the virtual-real fusion digital twin simulation system with incomplete data perception in this embodiment has been described in detail in the above-described virtual-real fusion digital twin simulation method with incomplete data perception, it will not be repeated here.

[0078] It should be understood that the above-described specific embodiments of the present invention are merely illustrative or explanatory of the principles of the present invention, and do not constitute a limitation thereof.

Claims

1. A virtual-real fusion digital twin simulation method with incomplete data perception, characterized in that, include: If the historical trajectory of an entity is missing trajectory data between the current time point and a certain historical time point, then the trajectory segment with missing trajectory data is taken as the trajectory segment to be predicted. The environmental information, including the known trajectory segment of the entity and the historical trajectory data of other intelligent agents within a set distance around the entity during the time of the known trajectory segment, as well as the noise distribution used as the initial value for generating the trajectory, are all input into the trained diffusion network. Based on the output, the noise-free prediction value of the trajectory segment to be predicted in this iteration is determined. The generated trajectory after denoising is obtained by comparing the predicted value of this iteration with the generated trajectory of the previous iteration; in the next iteration, the generated trajectory and environmental information are input into the diffusion network; the iteration is repeated to obtain the generated trajectory of the last iteration; the process of obtaining the generated trajectory of the last iteration is performed at least twice, and the corresponding motion region is determined by the at least two generated trajectories of the last iteration; the complete historical trajectory of the entity is obtained by combining the motion region with the known trajectory segment data. The complete historical trajectory of the entity is included as part of the environmental information of virtual objects within a set distance around the entity, and is used in the calculation when simulating the trajectory of the virtual object.

2. The virtual-real fusion digital twin simulation method with incomplete data perception according to claim 1, characterized in that, The complete historical trajectory of the entity is included in the environmental information of virtual bodies within a set distance around the entity. The methods used in simulating the trajectory of this virtual body include: For a virtual body whose trajectory is to be simulated, its future trajectory is taken as the trajectory segment to be predicted. The environmental information, including the known trajectory segment of the virtual body and the historical trajectory data of other intelligent agents within a set distance around the virtual body during the time of the known trajectory segment data, as well as the noise distribution as the initial value for generating the trajectory, are all input into the trained diffusion network. The process is repeated iteratively, and the future trajectory data of the virtual body is determined based on the generated trajectory obtained in the last iteration.

3. The virtual-real fusion digital twin simulation method with incomplete data perception according to claim 1, characterized in that, Also includes: If, within the time window of the historical trajectory of an entity, only the trajectory data of time points from the start of the time window to a certain historical time point is missing, then the historical trajectory data of the missing time point with the set missing value is taken as the complete historical trajectory data of the entity. The complete historical trajectory of the entity is included as part of the environmental information of virtual objects within a set distance around the entity, and is used in the calculation when simulating the trajectory of the virtual object.

4. The virtual-real fusion digital twin simulation method with incomplete data perception according to claim 3, characterized in that, The environmental information, which includes the known trajectory segment of the entity and the historical trajectory data of other intelligent agents within a set distance around the entity during the time of the known trajectory segment, also includes the data status corresponding to each time point in the known trajectory segment. The data status includes the status of the trajectory data without missing data, the status of the trajectory data missing data, and the status corresponding to the motion area.

5. The virtual-real fusion digital twin simulation method with incomplete data perception according to any one of claims 1-4, characterized in that, The trajectory data at each time point includes the target location corresponding to that trajectory; The methods for obtaining the denoised generated trajectory by comparing the predicted value of this iteration with the generated trajectory of the previous iteration include: Using the total guided loss of the predicted values ​​in this iteration, the gradients corresponding to the predicted values ​​in this iteration and the generated trajectory in the previous iteration are calculated respectively. Based on the predicted values ​​and the gradients corresponding to the generated trajectory, the predicted values ​​in this iteration and the generated trajectory in the previous iteration are updated respectively. Using the updated predicted values ​​and the generated trajectory in the previous iteration, the denoised generated trajectory is obtained. The total guidance loss includes the guidance loss term of the target consistency loss; the target consistency loss in this iteration is obtained by the difference between the target position at each time point in the noiseless prediction value of the trajectory segment to be predicted in this iteration and the corresponding target position setting value in the known trajectory segment.

6. The virtual-real fusion digital twin simulation method with incomplete data perception according to claim 5, characterized in that, The total guiding loss also includes at least one of the following guiding loss items: the guiding loss item of grouping loss, the guiding loss item of following loss, and the guiding loss item of obstacle avoidance loss.

7. The virtual-real fusion digital twin simulation method with incomplete data perception according to any one of claims 1-4, characterized in that, The methods for determining the corresponding motion region by obtaining at least two generated trajectories from the last iteration include: Based on the position information in the trajectory data of at least two generated trajectories from the last iteration at each time point, the corresponding position points are determined; based on the minimum adjacency graph of each position point, the possible motion region of the entity in the missing part of the historical trajectory is determined as the corresponding motion region; the shape of the adjacency graph is a set planar shape.

8. The virtual-real fusion digital twin simulation method with incomplete data perception according to any one of claims 1-4, characterized in that, Also includes: During the training of the diffusion network, the environmental information used includes the historical trajectory data of other agents within a set distance around the agent during the time period of the known trajectory segment data. If there is a missing trajectory data between the current time point and a certain historical time point in the historical trajectory data of an entity, the current diffusion model is first used to obtain at least two generated trajectories from the last iteration for the missing part, and the corresponding motion region is determined. The motion region is then combined with the known trajectory segment data to obtain the complete historical trajectory of the entity. The complete historical trajectory of the entity is then used as the historical trajectory data of the entity to construct the environmental information.

9. A virtual-real fusion digital twin simulation system with incomplete data perception, comprising a processor for executing program instructions, characterized in that, The program instructions are used to implement the virtual-real fusion digital twin simulation method with incomplete data perception as described in any one of claims 1-8.

Images