Rail scene prediction method, device, equipment and medium
By combining a lightweight diffusion model with multimodal data training, the problems of slow prediction speed and low accuracy of high-resolution video data are solved, and efficient and accurate orbit scene prediction is achieved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- BEIJING AINIBABY HEALTH MANAGEMENT CO LTD
- Filing Date
- 2025-08-22
- Publication Date
- 2026-06-09
Smart Images
Figure CN120766086B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of track sensing technology, and in particular to a method, apparatus, device and medium for predicting track scenes. Background Technology
[0002] In recent years, multimodal data fusion technology has gradually attracted attention. By integrating multimodal data such as video, depth maps, and normal maps, a more comprehensive perception of rail transit scenes can be achieved. However, in terms of model building, traditional deep learning models such as Convolutional Neural Networks (CNNs) struggle to capture the spatiotemporal dependencies in long-term time series when processing video data. Although Recurrent Neural Networks (RNNs) and their variants can model time-series data to some extent, complex models are often required for high-resolution video data to achieve rail transit scene prediction. These complex models require significant resources and time, thus potentially reducing the accuracy and speed of rail transit scene prediction. Summary of the Invention
[0003] This invention provides a method, apparatus, device, and medium for predicting rail transit scenes, which addresses the shortcomings of existing technologies that often require complex models to predict rail transit scenes for high-resolution video data. These complex models require significant resources and time, which may reduce the accuracy and speed of rail transit scene prediction. The invention implements a lightweight diffusion model trained using multimodal data at key time steps, which reduces the computational parameters of the model and accelerates the prediction speed and accuracy.
[0004] This invention provides a method for predicting orbital scenes, comprising the following steps.
[0005] Obtain the RGB image corresponding to the current frame in the orbit scene, the depth map corresponding to the RGB image, the normal map corresponding to the RGB image, and the task generation instruction;
[0006] The RGB image, along with its corresponding depth map, normal map, and the generated task instruction, are input into the lightweight diffusion model to obtain the orbital scene prediction video corresponding to the generated task instruction output by the lightweight diffusion model. The lightweight diffusion model is trained on the initial lightweight diffusion model based on the sample depth map, sample normal map, and sample text prompts corresponding to each sample image at key time steps.
[0007] According to a method for predicting orbital scenes provided by the present invention, the lightweight diffusion model includes an encoding module, a diffusion module, and a decoding module. The step of inputting the RGB image, its corresponding depth map, normal map, and the generation task instruction into the lightweight diffusion model to obtain the orbital scene prediction video corresponding to the generation task instruction output by the lightweight diffusion model includes: inputting the RGB image, its corresponding depth map, normal map, and the generation task instruction into the encoding module to obtain a first latent representation corresponding to the current frame output by the encoding module; inputting the first latent representation into the diffusion module to obtain a second latent representation corresponding to the next frame output by the diffusion module; continuously using the latent representation currently output by the diffusion module as input to the diffusion module to obtain new latent representations until a video length threshold is met; and then inputting all latent representations except the first latent representation into the decoding module to obtain the orbital scene prediction video corresponding to the generation task instruction output by the decoding module.
[0008] According to a method for predicting orbital scenes provided by the present invention, the lightweight diffusion model is trained based on the following steps: obtaining the sample depth map, sample normal map, sample text prompts, and sample generation task instructions corresponding to the sample images in the real sample videos corresponding to each key time step; inputting the sample images, corresponding sample depth maps, sample normal maps, and sample generation task instructions corresponding to the real sample videos in each key time step into an initial lightweight diffusion model to obtain the orbital scene sample prediction videos corresponding to each key time step output by the initial lightweight diffusion model; determining the loss information between the sample text prompts corresponding to the real sample videos and the text prompts in each of the orbital scene sample prediction videos based on the objective loss function; and continuously updating the model parameters of the initial lightweight diffusion model based on the loss information to obtain the lightweight diffusion model.
[0009] According to a method for predicting orbital scenes provided by the present invention, the initial lightweight diffusion model includes an initial encoding module, an initial diffusion module, and an initial decoding module. The step of inputting the sample images, corresponding sample depth maps, sample normal maps, and sample generation task instructions corresponding to the real sample videos at each key time step into the initial lightweight diffusion model to obtain the orbital scene sample prediction videos corresponding to each key time step output by the initial lightweight diffusion model includes: inputting the sample depth maps, sample normal maps, and sample generation task instructions corresponding to the sample images at each key time step into the initial encoding module to obtain the sample latent representations corresponding to each key time step output by the initial encoding module; the dimension of the sample latent representations is smaller than the dimension of the initial features corresponding to the key time steps; inputting each of the sample latent representations into the initial diffusion module to obtain the predicted latent representations corresponding to each key time step; and inputting each of the predicted latent representations into the initial decoding module to obtain the orbital scene sample prediction videos corresponding to each key time step.
[0010] According to a trajectory scene prediction method provided by the present invention, the target loss function includes a trajectory few-step guidance loss function, a consistency loss function, and a regularization loss function. The step of determining the loss information between the sample text prompts corresponding to the real sample videos and the text prompts in each of the trajectory scene sample prediction videos based on the target loss function includes: determining a first loss value between each of the trajectory scene sample prediction videos and the original prediction video based on the trajectory few-step guidance loss function; the original prediction video is a video predicted based on a pre-trained diffusion model; determining a second loss value corresponding to the spatiotemporal consistency between adjacent frames in each of the trajectory scene sample prediction videos based on the consistency loss function and the regularization loss function; and determining the loss information between the sample text prompts corresponding to the real sample videos and the text prompts in each of the trajectory scene sample prediction videos based on each of the first loss value and each of the second loss value.
[0011] According to a method for predicting orbital scenes provided by the present invention, the step of obtaining sample depth maps, sample normal maps, and sample text prompts corresponding to sample images in real sample videos corresponding to each key time step includes: obtaining an initial RGB video related to the orbital scene; performing enhancement processing on each frame of the initial RGB video to obtain an enhanced RGB video; determining the enhanced depth map and enhanced normal map corresponding to each frame of the enhanced RGB video; inputting each frame of the enhanced image into a multimodal large language model to obtain the text prompts corresponding to each frame of the enhanced image output by the multimodal large language model; and determining the sample depth map, sample normal map, and sample text prompts corresponding to each sample image within a first time interval based on the enhanced RGB video, the enhanced depth map, the enhanced normal map, and the text prompts.
[0012] According to a trajectory scene prediction method provided by the present invention, the step of determining the sample depth map, sample normal map, and sample text prompts corresponding to each sample image within a first time interval based on the enhanced RGB video, the enhanced depth map, the enhanced normal map, and the text prompts includes: standardizing each enhanced image, enhanced depth map, and enhanced normal map in the enhanced RGB video to obtain a standard RGB video, a standard depth map, and a standard normal map; inputting the standard RGB video, standard depth map, and standard normal map into a video diffusion model to obtain a video denoising trajectory; extracting multiple key time steps from the video denoising trajectory based on the quality of the RGB video; using each frame image of each key time step as a sample image, using the depth map and normal map corresponding to each frame image of the key time step as the sample depth map and sample normal map, respectively, and using the text prompts corresponding to each frame image of the key time step as sample text prompts.
[0013] The present invention also provides an orbital scene prediction device, comprising the following modules:
[0014] The acquisition module is used to acquire the RGB image corresponding to the current frame in the orbit scene, the depth map corresponding to the RGB image, the normal map corresponding to the RGB image, and the task generation instruction;
[0015] The prediction module is used to input the RGB image, the corresponding depth map, normal map, and the generation task instruction into the lightweight diffusion model to obtain the orbital scene prediction video corresponding to the generation task instruction output by the lightweight diffusion model; the lightweight diffusion model is trained on the initial lightweight diffusion model based on the sample depth map, sample normal map, and sample text prompts corresponding to each sample image at key time steps.
[0016] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the orbital scene prediction method as described above.
[0017] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the orbital scene prediction method as described above.
[0018] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the orbital scene prediction method as described above.
[0019] The orbital scene prediction method, apparatus, device, and medium provided by this invention acquire the RGB image corresponding to the current frame of the orbital scene, the depth map corresponding to the RGB image, the normal map corresponding to the RGB image, and the generation task instruction. The RGB image, along with the corresponding depth map, normal map, text prompt, and generation task instruction, are input into a lightweight diffusion model to obtain the orbital scene prediction video corresponding to the generation task instruction output by the lightweight diffusion model. The lightweight diffusion model is trained on an initial lightweight diffusion model using sample depth maps, sample normal maps, and sample text prompts corresponding to sample images at key time steps in the denoised trajectory generated by the video diffusion model. Thus, by using multimodal data trained on key time steps, the lightweight diffusion model reduces the model's computational parameters, accelerating the prediction speed and accuracy. Attached Figure Description
[0020] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.
[0021] Figure 1 A flowchart illustrating the orbital scene prediction method provided by this invention.
[0022] Figure 2 This is a schematic diagram of the training process for the lightweight diffusion model provided by the present invention.
[0023] Figure 3 This is a schematic diagram of the orbital scene prediction device provided by the present invention.
[0024] Figure 4 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation
[0025] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.
[0026] The following section describes existing methods for predicting orbital scenarios.
[0027] Traditional methods for predicting orbital scenarios often rely on single-modal data, such as video surveillance or sensor data. These methods have limitations when facing complex and dynamic environments. On the one hand, single-modal data cannot comprehensively capture the spatiotemporal information in the orbital scenario, resulting in insufficient accuracy and robustness of the prediction results. On the other hand, traditional prediction models are usually computationally complex and difficult to meet real-time requirements.
[0028] In recent years, multimodal data fusion technology has gradually attracted attention. By integrating multimodal data such as video, depth maps, and normal maps, a more comprehensive understanding of orbital scenes can be achieved. However, how to effectively fuse these multimodal data and utilize their spatiotemporal consistency for efficient prediction remains a challenging problem. Furthermore, existing prediction methods often lack effective mechanisms to fully leverage the spatiotemporal correlation of the data, thus affecting the reliability and practicality of the prediction results.
[0029] In addition, although recurrent neural networks (RNNs) and their variants can model time series data to some extent, complex models are often required to predict rail transit scenes for high-resolution video data. Complex models require a lot of resources and time. Therefore, using complex models may reduce the accuracy and speed of rail transit scene prediction.
[0030] To address the aforementioned issues, the orbital scene prediction method provided in this application reduces the computational parameters of the model by using a lightweight diffusion model trained with multimodal data at key time steps, thereby accelerating the prediction speed and accuracy of the model.
[0031] The following is combined Figure 1 The present invention describes an orbital scene prediction method applicable to the prediction of any orbital scene. The subject executing this method can be an electronic device or an orbital scene prediction method installed in the electronic device. The orbital scene prediction device can be implemented by software, hardware, or a combination of both.
[0032] Figure 1 This is a flowchart illustrating the orbital scene prediction method provided by the present invention, as shown below. Figure 1 As shown, the method includes the following:
[0033] Step 101: Obtain the RGB image corresponding to the current frame in the orbit scene, the depth map corresponding to the RGB image, the normal map corresponding to the RGB image, and the task generation instruction.
[0034] Here, a depth map is a grayscale image where the value of each pixel represents the distance (depth) from the scene point corresponding to that pixel to the camera; a normal map is a texture image where each pixel stores the direction of the normal to the surface at that point (usually represented in tangent space). Normal maps are used to simulate the details of high-polygon models on low-polygon models.
[0035] Optionally, the depth map can be obtained using depth data provided by the simulator, or it can be obtained using advanced video depth estimation methods such as RollingDepth.
[0036] Optionally, a depth normal function can be used to calculate the normal direction of each pixel in the depth map by analyzing the depth value changes, thereby generating a normal map. Alternatively, methods in the optical character recognition engine (TesserAct) can be used to generate high-quality normal maps using tools such as the Temporal-Consistent Marigold-LCM-normal model.
[0037] It should be noted that the RGB image corresponding to the current frame can be data that has undergone normalization, standardization, and noise sampling processing.
[0038] Typically, noise is sampled from a standard normal distribution. Therefore, noise sampling refers to sampling noise from a Gaussian distribution using a random number generator.
[0039] Optionally, the form of generating task instructions may include, but is not limited to, text descriptions or voice input.
[0040] Step 102: Input the RGB image, the corresponding depth map, normal map, and the generation task instruction into the lightweight diffusion model to obtain the orbital scene prediction video corresponding to the generation task instruction output by the lightweight diffusion model.
[0041] The lightweight diffusion model is obtained by training the initial lightweight diffusion model based on the sample depth map, sample normal map, and sample text prompts corresponding to each sample image at key time steps.
[0042] Optionally, the trajectory scene prediction video can be of any length.
[0043] Optionally, the process of generating a track scene prediction video can be to generate the RGB image, depth map, and normal map of the next frame based on the RGB image, depth map, and normal map of the current frame, and then continue this process until the preset video length is met to obtain the track scene prediction video; alternatively, the track scene prediction video can be generated directly based on the RGB image of the current frame.
[0044] In this embodiment of the invention, the RGB image corresponding to the current frame in the orbital scene, the depth map corresponding to the RGB image, the normal map corresponding to the RGB image, and the generation task instruction are obtained. The RGB image, along with the corresponding depth map, normal map, text prompt, and generation task instruction, are input into a lightweight diffusion model to obtain the orbital scene prediction video corresponding to the generation task instruction output by the lightweight diffusion model. The lightweight diffusion model is trained on an initial lightweight diffusion model based on the sample depth map, sample normal map, and sample text prompt corresponding to each sample image in the key time step of the denoised trajectory generated by the video diffusion model. Thus, by using multimodal data trained on key time steps, the lightweight diffusion model reduces the computational parameters of the model, accelerating the prediction speed and accuracy.
[0045] For example, the lightweight diffusion model includes an encoding module, a diffusion module, and a decoding module. The step of inputting the RGB image, its corresponding depth map, normal map, and the generation task instruction into the lightweight diffusion model to obtain the orbital scene prediction video corresponding to the generation task instruction output by the lightweight diffusion model includes: inputting the RGB image, its corresponding depth map, normal map, and the generation task instruction into the encoding module to obtain a first latent representation corresponding to the current frame output by the encoding module; inputting the first latent representation into the diffusion module to obtain a second latent representation corresponding to the next frame output by the diffusion module; continuously using the latent representation currently output by the diffusion module as input to the diffusion module to obtain new latent representations until a video length threshold is met; and then inputting all latent representations except the first latent representation into the decoding module to obtain the orbital scene prediction video corresponding to the generation task instruction output by the decoding module.
[0046] Here, the encoding module can be a 3D Variational Autoencoder (3DVAE) used to encode the RGB image and its corresponding depth map and normal map separately, converting them into latent representations. Furthermore, compression encoding can compress the high-dimensional data of the RGB image and its corresponding depth map and normal map into a low-dimensional latent space while preserving the main features and structural information of the data. In other words, the dimension of the first latent representation is smaller than the dimensions of the corresponding RGB image, depth map, and normal map.
[0047] Here, the diffusion module can be Diffusion in Transformer (DiT), a powerful sequence modeling tool that captures long-term dependencies and complex patterns in data. In DiT, the latent representation is unfolded into a sequence and processed through a self-attention mechanism. This self-attention mechanism allows the model to establish connections between different time steps and spatial locations, thereby capturing spatiotemporal information in the video data.
[0048] It should be noted that the second latent representation is the corresponding feature representation of the next frame image predicted based on the first latent representation.
[0049] For example, at each time frame, the lightweight diffusion model predicts the latent representation for the next time step based on the current sampled noise and latent representation. This process is iterative until a complete video sequence is generated. Specifically, at each time step, the model first integrates the latent representation as input, then uses its own architecture (such as the self-attention mechanism in DiT) to extract input features and capture spatiotemporal information. Based on these features, the model predicts the latent representation for the next time step. Subsequently, this latent representation is used as input for the next time step to continue the generation process until the preset video length is reached. This requires the model to have fully learned the complex patterns and long-term dependencies in the data during the training phase to ensure accurate predictions at each time step, thereby generating videos that are consistent in time and space.
[0050] Finally, in the decoding and output stage, the generated latent representation needs to be restored to the original data, i.e., the track scene video, by a decoding module (such as a 3D VAE decoder). The decoder has learned how to map the latent representation back to the original data space during model training. The decoding process involves operations such as feature mapping and data reconstruction. After the latent representation is input into the decoder, the resolution and details of the feature map are gradually restored through a series of convolution and upsampling operations. Then, at the decoder's output layer, convolution operations map the feature map to the dimensions and structure of the original data, generating RGB video, depth maps, and normal maps. Finally, these different modalities of data are fused together to form a complete track scene video. In this process, alignment and fusion techniques may be needed to ensure that the generated video has visual consistency and coherence.
[0051] In this embodiment of the invention, a lightweight diffusion model is constructed by an encoding module, a diffusion module, and a decoding module. The lightweight diffusion model is used to extract complex patterns and long-term dependencies to ensure accurate prediction at each time frame, thereby generating a video that maintains consistency in time and space and improving the accuracy, speed, and efficiency of orbit scene prediction.
[0052] In another embodiment, after generating the scene trajectory prediction video, its spatiotemporal consistency can be verified according to spatiotemporal standards, its performance can be evaluated by comparing it with real data, and the model can be optimized based on the results to further improve the prediction quality.
[0053] For example, Figure 2 This is a schematic diagram of the training process for the lightweight diffusion model provided by the present invention, as shown below. Figure 2 As shown, the lightweight diffusion model is trained based on the following steps:
[0054] Step 201: Obtain the sample depth map, sample normal map, sample text prompt, and sample generation task instruction corresponding to each sample image in the real sample video corresponding to each key time step.
[0055] It should be noted that different time steps add different degrees of noise to the real sample video. A time step includes multiple image frames. The initial lightweight diffusion model can predict the image of the next frame based on each image frame. Then, the predicted image is compared with the real image to obtain the difference value. Based on the difference value, the model parameters of the initial lightweight diffusion model are continuously trained to obtain the lightweight diffusion model.
[0056] It should be noted that sample text prompts can be understood as real labels in RGB images. Sample text prompts are descriptive text generated based on image content. The RGB image of the track scene can be input into a multimodal large language model (MLLM) to obtain a text describing the objects, actions and environment in the image. This text is the sample text prompt, such as the InternVL2.5-8B model.
[0057] Here, each sample image, the corresponding sample depth map, and the sample normal map can be preprocessed data, including but not limited to normalization, standardization, and noise sampling.
[0058] For example, the step of obtaining the sample depth map, sample normal map, and sample text prompts corresponding to each sample image in the real sample video corresponding to each key time step includes: obtaining an initial RGB video related to the track scene; performing enhancement processing on each frame of the initial RGB video to obtain an enhanced RGB video; determining the enhanced depth map and enhanced normal map corresponding to each frame of the enhanced RGB video; inputting each frame of the enhanced image into a multimodal large language model to obtain the text prompts corresponding to each frame of the enhanced image output by the multimodal large language model; and determining the sample depth map, sample normal map, and sample text prompts corresponding to each sample image within a first time interval based on the enhanced RGB video, the enhanced depth map, the enhanced normal map, and the text prompts.
[0059] Optionally, the initial RGB video can be a composite video generated using a simulator such as RLBench, or it can be video captured by sensors and other devices in a real orbital scene.
[0060] Here, enhancement processing includes, but is not limited to, altering the fusion of different background images with the original orbital scene image to simulate different environmental conditions; adjusting the light intensity and angle to simulate different lighting conditions; and performing operations such as random cropping, rotation, and scaling on the image to increase the diversity of the data.
[0061] Here, you can directly select sample images and their corresponding sample depth maps, sample normal maps, and sample text prompts from enhanced RGB video, enhanced depth map, enhanced normal map, and sample text prompts; or you can obtain sample images and their corresponding sample depth maps, sample normal maps, and sample text prompts by performing noise sampling on enhanced RGB video, enhanced depth map, and enhanced normal map.
[0062] In this embodiment of the invention, multimodal data of the orbital scenario are integrated and a high-quality dataset is constructed. Appropriate data augmentation is then performed to provide a rich and effective data foundation for training the lightweight diffusion model, thereby improving the model's predictive performance in the orbital scenario.
[0063] For example, determining the sample depth map, sample normal map, and sample text prompt corresponding to each sample image within a first time interval based on the enhanced RGB video, the enhanced depth map, the enhanced normal map, and the text prompt includes: standardizing each enhanced image, enhanced depth map, and enhanced normal map in the enhanced RGB video to obtain a standard RGB video, a standard depth map, and a standard normal map; inputting the standard RGB video, standard depth map, and standard normal map into a video diffusion model to obtain a video denoising trajectory; extracting multiple key time steps from the video denoising trajectory based on the quality of the RGB video; using each frame image of each key time step as a sample image, using the depth map and normal map corresponding to each frame image of the key time step as the sample depth map and sample normal map, respectively, and using the text prompt corresponding to each frame image of the key time step as the sample text prompt.
[0064] For example, the value range of the depth map and normal map can be normalized to the [0, 1] interval. For RGB images, pixel value normalization can be performed, mapping pixel values from the [0, 255] interval to the [0, 1] interval.
[0065] It should be noted that the video diffusion model converts the data into a Gaussian distribution by gradually adding noise, and then generates data through a denoising process. Here, the noise added at different time steps is different. The frames corresponding to the time steps with good quality are extracted from the denoised trajectory generated by the video diffusion model, and the corresponding depth map, normal map and text prompts are extracted for each frame as data samples. The data samples include sample images, sample depth maps, sample normal maps and sample text prompts.
[0066] In this embodiment of the invention, the data is further processed through normalization, standardization, and noise sampling to ensure the quality and validity of the final dataset, providing solid data support for subsequent model training and optimization, and laying the foundation for achieving efficient orbital scene prediction.
[0067] Step 202: Input the sample image, corresponding sample depth map, sample normal map and sample generation task instruction corresponding to the real sample video in each key time step into the initial lightweight diffusion model to obtain the orbit scene sample prediction video corresponding to each key time step output by the initial lightweight diffusion model.
[0068] For example, the initial lightweight diffusion model includes an initial encoding module, an initial diffusion module, and an initial decoding module. The step of inputting the sample images, corresponding sample depth maps, sample normal maps, and sample generation task instructions corresponding to the real sample videos at each key time step into the initial lightweight diffusion model to obtain the orbital scene sample prediction videos output by the initial lightweight diffusion model at each key time step includes: inputting the sample depth maps, sample normal maps, and sample generation task instructions corresponding to the real sample videos at each key time step into the initial encoding module to obtain the sample latent representations output by the initial encoding module at each key time step; the dimension of the sample latent representations is smaller than the dimension of the initial features corresponding to the key time steps; inputting each of the sample latent representations into the initial diffusion module to obtain the predicted latent representations corresponding to each key time step; and inputting each of the predicted latent representations into the initial decoding module to obtain the orbital scene sample prediction videos corresponding to each key time step.
[0069] Here, the initial encoding module is used to encode the sample image, the corresponding sample depth map, and the sample normal map to obtain the compressed sample latent representation. The initial diffusion model is used to generate the predicted latent representation based on the compressed sample latent representation. The predicted latent representation is then input into the initial decoding module to obtain the orbital sample prediction video.
[0070] It should be noted that the initial diffusion module may include a self-attention mechanism. After the latent representation is expanded into a sequence, it is processed through the self-attention mechanism. The self-attention mechanism allows the initial lightweight diffusion model to establish connections between different time steps and spatial locations, thereby capturing the spatiotemporal information in the video data.
[0071] In this embodiment of the invention, the key features of the data are captured by the initial encoding module, reducing the computational complexity of the model; the initial diffusion module captures long-term dependencies and complex patterns in the data; and the initial decoder restores the image, thus realizing an efficient, accurate, and lightweight scene prediction model.
[0072] Step 203: Based on the target loss function, determine the loss information between the sample text prompts corresponding to the real sample videos and the text prompts in the predicted videos of each of the track scene samples.
[0073] It should be noted that the loss information is used to represent the difference between the sample text prompts corresponding to the real sample videos and the text prompts in the predicted videos of each track scene. The larger the loss information, the larger the difference; the smaller the loss information, the smaller the difference, and the better the model.
[0074] The sample text prompts corresponding to real sample videos refer to the sample text prompts corresponding to each sample image in the real sample video.
[0075] For example, the target loss function includes a trajectory few-step guidance loss function, a consistency loss function, and a regularization loss function. The step of determining the loss information between the sample text prompts corresponding to the real sample videos and the text prompts in each of the predicted track scene sample videos based on the target loss function includes: determining a first loss value between each of the predicted track scene sample videos and the original predicted video based on the trajectory few-step guidance loss function; the original predicted video is a video predicted based on a pre-trained diffusion model; determining a second loss value corresponding to the spatiotemporal consistency between adjacent frames in each of the predicted track scene sample videos based on the consistency loss function and the regularization loss function; and determining the loss information between the sample text prompts corresponding to the real sample videos and the text prompts in each of the predicted track scene sample videos based on each of the first loss value and each of the second loss values.
[0076] It should be noted that the sample text prompts can be understood as the real labels in the RGB images. After predicting the video based on the orbital scene samples generated at each key time step, it is necessary to compare the predicted video of the orbital scene samples with the text prompts of each sample image in the real sample video to obtain the difference information between the two. Based on the difference information, backpropagation is performed to optimize the parameters of the initial lightweight diffusion model to obtain the lightweight diffusion model.
[0077] Here, the trajectory few-step guidance loss function is used to train the initial diffusion model to learn the parameters of the diffusion model.
[0078] It's important to note that the original diffusion model lacks a lightweight diffusion model. However, this model requires denoising at each time step to generate the video, necessitating a large number of model parameters, resulting in long prediction times and slow speed. Therefore, a trajectory-based few-step guided loss function is needed to measure the difference between key data points in the predicted video from the trajectory scene samples and those generated by the original diffusion model. Backpropagation is then used to optimize the parameters of the initial lightweight diffusion model. In this way, the initial lightweight diffusion model can generate videos of comparable quality to the teacher model in fewer steps. A larger first loss value indicates a greater difference between the initial lightweight diffusion model and the original diffusion model, while a smaller first loss value indicates a smaller difference.
[0079] Here, the consistency between adjacent frames can be distinguished from the background and dynamic regions by the optical flow between adjacent frames. In other words, the generated video needs to ensure spatiotemporal consistency. The larger the loss value, the worse the spatiotemporal consistency, and the smaller the loss value, the better the spatiotemporal consistency.
[0080] The regularization loss function imposes consistency constraints on these regions. Specifically, the model uses optical flow calculation tools (such as RAFT) to calculate the optical flow field between adjacent frames, thereby determining the temporal correspondence of pixels. Based on this correspondence, the model defines consistency losses in dynamic and background regions respectively to ensure the temporal consistency of the depth map.
[0081] The regularization loss function constrains the depth map generation process by incorporating prior knowledge. For example, it can assume that the surfaces in the scene are smooth, thus imposing a smoothness constraint on the depth map. By minimizing the regularization loss function, the model can generate depth maps that are more physically accurate, improving the accuracy of scene prediction.
[0082] In this embodiment of the invention, after selecting data points at key time steps, a trajectory short-step guidance mechanism is introduced to construct a shorter noise-to-video mapping path, thereby reducing the steps required for generation. Consistency loss and regularization loss functions are introduced to ensure that the generated video is consistent in time and space, thereby improving the accuracy of scene prediction.
[0083] Step 204: Based on the loss information, continuously update the model parameters of the initial lightweight diffusion model to obtain the lightweight diffusion model.
[0084] The update process typically employs optimization algorithms such as stochastic gradient descent or Adam. By iteratively updating the model parameters, the model can gradually converge to a stable state, resulting in a lightweight diffusion model, thereby achieving spatiotemporal consistent predictions of orbital scenarios.
[0085] In this embodiment of the invention, a lightweight diffusion model is constructed through an initial encoding module, an initial diffusion module, and an initial decoding module. The model introduces a trajectory few-step guidance mechanism, the definition of consistency loss and regularization loss function, as well as model training and optimization, to ensure that the model can efficiently process orbit scene data and generate spatiotemporally consistent prediction results, thereby achieving an efficient and accurate lightweight scene prediction model.
[0086] The orbital scene prediction device provided by the present invention is described below. The orbital scene prediction device described below can be referred to in correspondence with the orbital scene prediction method described above.
[0087] Figure 3 This is a schematic diagram of the orbital scene prediction method provided by the present invention, as shown below. Figure 3 As shown, the orbital scene prediction device 300 includes:
[0088] The acquisition module 310 is used to acquire the RGB image corresponding to the current frame in the track scene, the depth map corresponding to the RGB image, the normal map corresponding to the RGB image, and the task generation instruction;
[0089] The prediction module 320 is used to input the RGB image, the corresponding depth map, normal map, and the generation task instruction into the lightweight diffusion model to obtain the track scene prediction video corresponding to the generation task instruction output by the lightweight diffusion model; the lightweight diffusion model is trained on the initial lightweight diffusion model based on the sample depth map, sample normal map, and sample text prompts corresponding to each sample image at key time steps.
[0090] In another embodiment, the lightweight diffusion model includes an encoding module, a diffusion module, a decoding module, and a prediction module 320, specifically configured to: input the RGB image, its corresponding depth map, normal map, and the generation task instruction into the encoding module to obtain a first latent representation corresponding to the current frame output by the encoding module; input the first latent representation into the diffusion module to obtain a second latent representation corresponding to the next frame output by the diffusion module; continuously use the latent representation currently output by the diffusion module as input to the diffusion module to obtain new latent representations until a video length threshold is met; and input all latent representations except the first latent representation into the decoding module to obtain the track scene prediction video corresponding to the generation task instruction output by the decoding module.
[0091] In another embodiment, the orbital scene prediction device 300 further includes a training module, specifically configured to: acquire sample depth maps, sample normal maps, sample text prompts, and sample generation task instructions corresponding to the real sample videos in each key time step; input each sample image, corresponding sample depth map, sample normal map, and sample generation task instructions corresponding to the real sample videos in each key time step into an initial lightweight diffusion model to obtain the orbital scene sample prediction videos corresponding to each key time step output by the initial lightweight diffusion model; determine the loss information between the sample text prompts corresponding to the real sample videos and the text prompts in each of the orbital scene sample prediction videos based on the target loss function; and continuously update the model parameters of the initial lightweight diffusion model based on the loss information to obtain the lightweight diffusion model.
[0092] In another embodiment, the initial lightweight diffusion model includes an initial encoding module, an initial diffusion module, and an initial decoding module. The training module is further specifically used for: inputting the sample depth map, sample normal map, and sample generation task instruction corresponding to the sample image in each key time step into the initial encoding module to obtain the sample latent representation corresponding to each key time step output by the initial encoding module; the dimension of the sample representation is smaller than the dimension of the initial feature corresponding to the key time step; inputting each of the sample latent representations into the initial diffusion module to obtain the predicted latent representation corresponding to each key time step; and inputting each of the predicted latent representations into the initial decoding module to obtain the track scene sample prediction video corresponding to each key time step.
[0093] In another embodiment, the target loss function includes a trajectory few-step guidance loss function, a consistency loss function, and a regularization loss function. The training module is further specifically used for: determining a first loss value between the predicted video of each orbital scene sample and the original predicted video based on the trajectory few-step guidance loss function; the original predicted video is a video predicted based on a pre-trained diffusion model; determining a second loss value corresponding to the degree of spatiotemporal consistency between adjacent frames in each predicted video of each orbital scene sample based on the consistency loss function and the regularization loss function; and determining loss information between the sample text prompts corresponding to the real sample videos and the text prompts in each predicted video of each orbital scene sample based on the first loss value and the second loss value.
[0094] In another embodiment, the training module is further specifically configured to: acquire an initial RGB video related to the track scene; perform enhancement processing on each frame of the initial RGB video to obtain an enhanced RGB video; determine the enhanced depth map and enhanced normal map corresponding to each frame of the enhanced RGB video; input the enhanced images of each frame into a multimodal large language model to obtain the text prompts corresponding to each frame of the enhanced images output by the multimodal large language model; and determine the sample depth map, sample normal map, and sample text prompts corresponding to each sample image within a first time interval based on the enhanced RGB video, the enhanced depth map, the enhanced normal map, and the text prompts.
[0095] In another embodiment, the training module is further specifically configured to: standardize each of the enhanced images, enhanced depth maps, and enhanced normal maps in the enhanced RGB video to obtain standard RGB video, standard depth maps, and standard normal maps; input the standard RGB video, standard depth maps, and standard normal maps into a video diffusion model to obtain a video denoising trajectory; extract multiple key time steps from the video denoising trajectory based on the quality of the RGB video; use each frame image of each key time step as a sample image, use the depth map and normal map corresponding to each frame image of the key time step as sample depth maps and sample normal maps, respectively, and use the text prompts corresponding to each frame image of the key time step as sample text prompts.
[0096] Figure 4 This is a schematic diagram of the structure of the electronic device provided by the present invention, such as... Figure 4 As shown, the electronic device may include: a processor 410, a communication interface 420, a memory 430, and a communication bus 440, wherein the processor 410, the communication interface 420, and the memory 430 communicate with each other through the communication bus 440. The processor 410 can call logical instructions in the memory 430 to execute an orbital scene prediction method, which includes: acquiring the RGB image corresponding to the current frame in the orbital scene, the depth map corresponding to the RGB image, the normal map corresponding to the RGB image, and a generation task instruction; inputting the RGB image, the corresponding depth map, the normal map, and the generation task instruction into a lightweight diffusion model to obtain the orbital scene prediction video corresponding to the generation task instruction output by the lightweight diffusion model; the lightweight diffusion model is trained on an initial lightweight diffusion model based on the sample depth map, sample normal map, and sample text prompts corresponding to each sample image at key time steps.
[0097] Furthermore, the logical instructions in the aforementioned memory 430 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.
[0098] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the orbital scene prediction method provided by the above methods. The method includes: acquiring an RGB image corresponding to the current frame in the orbital scene, a depth map corresponding to the RGB image, a normal map corresponding to the RGB image, and a generation task instruction; inputting the RGB image, the corresponding depth map, the normal map, and the generation task instruction into a lightweight diffusion model to obtain an orbital scene prediction video corresponding to the generation task instruction output by the lightweight diffusion model; the lightweight diffusion model is trained on an initial lightweight diffusion model based on sample depth maps, sample normal maps, and sample text prompts corresponding to sample images at key time steps.
[0099] In another aspect, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the orbital scene prediction method provided by the above methods. The method includes: acquiring an RGB image corresponding to the current frame in the orbital scene, a depth map corresponding to the RGB image, a normal map corresponding to the RGB image, and a generation task instruction; inputting the RGB image, the corresponding depth map, the normal map, and the generation task instruction into a lightweight diffusion model to obtain an orbital scene prediction video corresponding to the generation task instruction output by the lightweight diffusion model; wherein the lightweight diffusion model is trained on an initial lightweight diffusion model based on sample depth maps, sample normal maps, and sample text prompts corresponding to sample images at key time steps.
[0100] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.
[0101] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.
[0102] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.
Claims
1. A method for predicting orbital scenes, characterized in that, include: Obtain the RGB image corresponding to the current frame in the orbit scene, the depth map corresponding to the RGB image, the normal map corresponding to the RGB image, and the task generation instruction; The RGB image, the corresponding depth map, the normal map, and the generation task instruction are input into the encoding module in the lightweight diffusion model to obtain the first latent representation of the current frame output by the encoding module. The encoding module is a three-dimensional variational autoencoder, which compresses the high-dimensional data of the RGB image and the corresponding depth map and normal map into a low-dimensional latent space through compression encoding. The first latent representation is input into the diffusion module in the lightweight diffusion model to obtain the second latent representation corresponding to the next frame output by the diffusion module. The second latent representation is the feature representation of the next frame image predicted based on the first latent representation. The latent representation currently output by the diffusion module is continuously used as input to the diffusion module to obtain new latent representations until the video length threshold is met. Then, all latent representations except the first latent representation are input to the decoding module to obtain the track scene prediction video corresponding to the generation task instruction output by the decoding module. The lightweight diffusion model is trained on the initial lightweight diffusion model by the sample depth map, sample normal map, and sample text prompts corresponding to each sample image in the key time step of the denoised trajectory generated by the video diffusion model. The sample text prompts are the real labels in the RGB image and the sample text prompts are descriptive text generated based on the image content. The lightweight diffusion model is trained based on the following steps: Obtain the sample depth map, sample normal map, sample text prompt, and sample generation task instruction corresponding to the real sample video in each key time step; Input the sample image, the corresponding sample depth map, the sample normal map, and the sample generation task instruction corresponding to the real sample video in each key time step into the initial lightweight diffusion model to obtain the orbit scene sample prediction video corresponding to each key time step output by the initial lightweight diffusion model. Based on the trajectory few-step guidance loss function, a first loss value is determined between the text prompts of the predicted video of each track scene sample and the text prompts of the original predicted video. The trajectory few-step guidance loss function is used to measure the difference between the key data points in the predicted video of the track scene sample and the key data points generated by the original diffusion model. The original predicted video is a video predicted based on a pre-trained diffusion model. Based on the consistency loss function and the regularization loss function, a second loss value corresponding to the degree of spatiotemporal consistency between adjacent frames in the predicted video of each track scene sample is determined; Based on each of the first loss values and each of the second loss values, the loss information between the sample text prompts corresponding to the real sample videos and the text prompts in each of the track scene sample prediction videos is determined, wherein the sample text prompts corresponding to the real sample videos refer to the sample text prompts corresponding to each sample image in the real sample videos. Based on the loss information, the model parameters of the initial lightweight diffusion model are continuously updated to obtain the lightweight diffusion model.
2. The orbital scene prediction method according to claim 1, characterized in that, The initial lightweight diffusion model includes an initial encoding module, an initial diffusion module, and an initial decoding module. The initial lightweight diffusion model is obtained by inputting the sample images, corresponding sample depth maps, sample normal maps, and sample generation task instructions corresponding to the real sample videos at each key time step into the initial lightweight diffusion model, resulting in the output of the initial lightweight diffusion model for the predicted orbital scene samples at each key time step, including: The sample depth map, sample normal map, and sample generation task instruction corresponding to the real sample video at each key time step are input into the initial encoding module to obtain the sample latent representation corresponding to each key time step output by the initial encoding module; the dimension of the sample latent representation is smaller than the dimension of the initial feature corresponding to the key time step. The latent representations of each sample are input into the initial diffusion module to obtain the predicted latent representations corresponding to each key time step; Each of the predicted latent representations is input into the initial decoding module to obtain the predicted video of the orbital scene sample corresponding to each key time step.
3. The orbital scene prediction method according to claim 2, characterized in that, The acquisition of sample depth maps, sample normal maps, and sample text prompts corresponding to each sample image in the real sample video at each key time step includes: Acquire the initial RGB video related to the track scene; The initial images of each frame in the initial RGB video are enhanced to obtain an enhanced RGB video; Determine the enhancement depth map and enhancement normal map corresponding to each frame of the enhanced RGB video; The enhanced images of each frame are input into the multimodal large language model to obtain the text prompts corresponding to the enhanced images of each frame output by the multimodal large language model. Based on the enhanced RGB video, the enhanced depth map, the enhanced normal map, and the text prompt, the sample depth map, sample normal map, and sample text prompt corresponding to each sample image within the first time interval are determined.
4. The orbital scene prediction method according to claim 3, characterized in that, The step of determining the sample depth map, sample normal map, and sample text prompt corresponding to each sample image within a first time interval based on the enhanced RGB video, the enhanced depth map, the enhanced normal map, and the text prompt includes: The enhanced images, enhanced depth maps, and enhanced normal maps in the enhanced RGB video are standardized to obtain standard RGB video, standard depth maps, and standard normal maps. The standard RGB video, standard depth map, and standard normal map are input into the video diffusion model to obtain the video denoising trajectory. Based on the quality of the RGB video, multiple key time steps are extracted from the video denoising trajectory. Each frame image at each key time step is used as a sample image, and the depth map and normal map corresponding to each frame image at each key time step are used as sample depth map and sample normal map, respectively. The text prompts corresponding to each frame image at each key time step are used as sample text prompts.
5. A trajectory scene prediction device, characterized in that, include: The acquisition module acquires the RGB image corresponding to the current frame in the orbit scene, the depth map corresponding to the RGB image, the normal map corresponding to the RGB image, and the task generation instruction; The prediction module inputs the RGB image, its corresponding depth map, normal map, and the generation task instruction into the encoding module of the lightweight diffusion model to obtain the first latent representation corresponding to the current frame output by the encoding module. The encoding module is a 3D variational autoencoder that compresses the high-dimensional data of the RGB image and its corresponding depth map and normal map into a low-dimensional latent space through compression encoding. The first latent representation is input into the diffusion module of the lightweight diffusion model to obtain the second latent representation corresponding to the next frame output by the diffusion module. The second latent representation is the feature representation of the next frame image predicted based on the first latent representation. The current latent representation output by the diffusion module is continuously used as input to the diffusion module to obtain new latent representations until a video length threshold is met. Then, all latent representations except the first latent representation are input into the decoding module to obtain the track scene prediction video corresponding to the generation task instruction output by the decoding module. The lightweight diffusion model is trained on an initial lightweight diffusion model using sample depth maps, sample normal maps, and sample text prompts corresponding to sample images at key time steps in the denoised trajectory generated by the video diffusion model. The sample text prompts are the ground truth labels in the RGB image and are descriptive text generated based on the image content. The training module acquires the sample depth map, sample normal map, sample text prompts, and sample generation task instructions corresponding to the real sample videos for each key time step. It then inputs the sample images, corresponding sample depth maps, sample normal maps, and sample generation task instructions from each key time step into an initial lightweight diffusion model to obtain the track scene sample prediction videos for each key time step output by the initial lightweight diffusion model. Based on the trajectory few-step guidance loss function, it determines the first loss value between the text prompts of each track scene sample prediction video and the text prompts of the original prediction video. The trajectory few-step guidance loss function is used to measure the key data points in the track scene sample prediction videos. The original predicted video, which differs from the key data points generated by the original diffusion model, is a video predicted based on a pre-trained diffusion model. A second loss value is determined based on the consistency loss function and the regularization loss function, corresponding to the degree of spatiotemporal consistency between adjacent frames in each of the predicted videos of the track scene samples. Based on each of the first loss value and each of the second loss values, loss information is determined between the sample text prompts corresponding to the real sample videos and the text prompts in each of the predicted videos of the track scene samples, wherein the sample text prompts corresponding to the real sample videos refer to the sample text prompts corresponding to each sample image in the real sample videos. Based on the loss information, the model parameters of the initial lightweight diffusion model are continuously updated to obtain the lightweight diffusion model.
6. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the computer program, it implements the orbital scene prediction method as described in any one of claims 1 to 4.
7. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the orbital scene prediction method as described in any one of claims 1 to 4.