Track scene prediction method and device, equipment and medium

Abstract

The invention provides a track scene prediction method and device, equipment and a medium, and relates to the technical field of track perception, and the track scene prediction method comprises the steps: obtaining an RGB image corresponding to a current frame in a track scene, a depth map corresponding to the RGB image, a normal map corresponding to the RGB image, a text prompt corresponding to the RGB image, and a generation task instruction; inputting the RGB image, the corresponding depth map, the normal map, the text prompt and the generation task instruction into a lightweight diffusion model to obtain a track scene prediction video corresponding to the generation task instruction output by the lightweight diffusion model; the lightweight diffusion model is obtained by training an initial lightweight diffusion model based on a sample depth map, a sample normal map and a sample text prompt corresponding to each sample image of the key time step. Through the lightweight diffusion model trained by using the multi-modal data of the key time step, the calculation parameters of the model are reduced, and the prediction speed and accuracy of the model are accelerated.

Description

Technical Field

[0001] The present invention relates to the field of track perception technology, and in particular to a track scene prediction method, device, equipment and medium. Background Art

[0002] In recent years, multimodal data fusion technology has gained increasing attention. By integrating multimodal data such as video, depth maps, and normal maps, it can provide a more comprehensive understanding of rail transit scenarios. However, when it comes to model construction, traditional deep learning models such as convolutional neural networks (CNNs) struggle to capture the spatiotemporal dependencies within long time series when processing video data. While recurrent neural networks (RNNs) and their variants can model time series data to a certain extent, high-resolution video data often requires complex models to achieve rail transit scenario prediction. These complex models require significant resources and are time-consuming. Consequently, using these models can reduce the accuracy and speed of rail transit scenario prediction. Summary of the Invention

[0003] The present invention provides a track scene prediction method, device, equipment and medium to solve the problem in the prior art that high-resolution video data often requires complex models to achieve track transit scene prediction. Complex models require a large amount of resources and a long time. Therefore, the use of complex models may reduce the accuracy and speed of track scene prediction. The lightweight diffusion model trained with multimodal data at key time steps reduces the model's calculation parameters and accelerates the model's prediction speed and accuracy.

[0004] The present invention provides a track scene prediction method, comprising the following steps.

[0005] Obtain an RGB image corresponding to a current frame in a track scene, a depth map corresponding to the RGB image, a normal map corresponding to the RGB image, and generate a task instruction; The RGB image and the corresponding depth map, normal map and the generation task instruction are input into a lightweight diffusion model to obtain a track scene prediction video corresponding to the generation task instruction output by the lightweight diffusion model; the lightweight diffusion model is obtained by training an initial lightweight diffusion model based on the sample depth map, sample normal map and sample text prompt corresponding to each sample image at a key time step.

[0006] According to a track scene prediction method provided by the present invention, the lightweight diffusion model includes an encoding module, a diffusion module, and a decoding module. The RGB image and the corresponding depth map, normal map and the generation task instruction are input into the lightweight diffusion model to obtain a track scene prediction video corresponding to the generation task instruction output by the lightweight diffusion model, including: inputting the RGB image and the corresponding depth map, normal map and the generation task instruction into the encoding module to obtain a first potential representation corresponding to the current frame output by the encoding module; inputting the first potential representation into the diffusion module to obtain a second potential representation corresponding to the next frame output by the diffusion module; continuously using the potential representation currently output by the diffusion module as the input of the diffusion module to obtain a new potential representation until a video length threshold is met, and then inputting all potential representations except the first potential representation into the decoding module to obtain a track scene prediction video corresponding to the generation task instruction output by the decoding module.

[0007] According to a track scene prediction method provided by the present invention, the lightweight diffusion model is trained based on the following steps: obtaining a sample depth map, a sample normal map, a sample text prompt corresponding to each sample image in a real sample video corresponding to each key time step, and a sample generation task instruction corresponding to the real sample video; inputting each sample image, the corresponding sample depth map, the sample normal map, and the sample generation task instruction corresponding to the real sample video in each key time step into an initial lightweight diffusion model to obtain a track scene sample prediction video corresponding to each key time step output by the initial lightweight diffusion model; based on a target loss function, determining the loss information between the sample text prompt corresponding to the real sample video and the text prompt in each track scene sample prediction video; based on each loss information, continuously updating the model parameters of the initial lightweight diffusion model to obtain the lightweight diffusion model.

[0008] According to a track scene prediction method provided by the present invention, the initial lightweight diffusion model includes an initial encoding module, an initial diffusion module and an initial decoding module, and the sample generation task instructions corresponding to each sample image, the corresponding sample depth map, the sample normal map and the real sample video in each key time step are input into the initial lightweight diffusion model to obtain the track scene sample prediction video corresponding to each key time step output by the initial lightweight diffusion model, including: inputting the sample depth map, the sample normal map and the sample generation task instructions corresponding to each sample image in each key time step and the real sample video into the initial encoding module to obtain the sample potential representation corresponding to each key time step output by the initial encoding module; the dimension of the sample potential representation is smaller than the dimension of the initial feature corresponding to the key time step; inputting each of the sample potential representations into the initial diffusion module to obtain the predicted potential representation corresponding to each key time step; and inputting each of the predicted potential representations into the initial decoding module to obtain the track scene sample prediction video corresponding to each key time step.

[0009] According to a track scene prediction method provided by the present invention, the target loss function includes a track few-step guidance loss function, a consistency loss function and a regularization loss function. The method of determining the loss information between the sample text prompt corresponding to the real sample video and the text prompt in each track scene sample prediction video based on the target loss function includes: determining a first loss value between each track scene sample prediction video and an original prediction video based on the track few-step guidance loss function; the original prediction video is a video predicted based on a pre-trained diffusion model; based on the consistency loss function and the regularization loss function, determining a second loss value corresponding to the degree of spatiotemporal consistency between adjacent frames in each track scene sample prediction video; based on each of the first loss values ​​and each of the second loss values, determining the loss information between the sample text prompt corresponding to the real sample video and the text prompt in each of the track scene sample prediction videos.

[0010] According to a track scene prediction method provided by the present invention, the method of obtaining a sample depth map, a sample normal map, and a sample text prompt corresponding to each sample image in a real sample video corresponding to each key time step includes: obtaining an initial RGB video related to the track scene; enhancing each initial image frame in the initial RGB video to obtain an enhanced RGB video; determining an enhanced depth map and an enhanced normal map corresponding to each enhanced image frame in the enhanced RGB video; inputting each enhanced image frame into a multimodal large language model to obtain a text prompt corresponding to each enhanced image frame output by the multimodal large language model; and determining a sample depth map, a sample normal map, and a sample text prompt corresponding to each sample image in a first time interval based on the enhanced RGB video, the enhanced depth map, the enhanced normal map, and the text prompt.

[0011] According to a track scene prediction method provided by the present invention, based on the enhanced RGB video, the enhanced depth map, the enhanced normal map and the text prompt, the sample depth map, the sample normal map and the sample text prompt corresponding to each sample image in the first time interval are determined, including: standardizing each of the enhanced images, enhanced depth maps and enhanced normal maps in the enhanced RGB video to obtain a standard RGB video, a standard depth map and a standard normal map; inputting the standard RGB video, the standard depth map and the standard normal map into a video diffusion model to obtain a video denoising trajectory; based on the quality of the RGB video, extracting multiple key time steps from the video denoising trajectory; using each frame image of each key time step as a sample image, using the depth map and normal map corresponding to each frame image of the key time step as a sample depth map and sample normal map, respectively, and using the text prompt corresponding to each frame image of the key time step as a sample text prompt.

[0012] The present invention also provides a track scene prediction device, comprising the following modules: An acquisition module is used to acquire an RGB image corresponding to a current frame in a track scene, a depth map corresponding to the RGB image, a normal map corresponding to the RGB image, and generate a task instruction; The prediction module is used to input the RGB image and the corresponding depth map, normal map and the generation task instruction into a lightweight diffusion model to obtain a track scene prediction video corresponding to the generation task instruction output by the lightweight diffusion model; the lightweight diffusion model is obtained by training an initial lightweight diffusion model based on the sample depth map, sample normal map and sample text prompt corresponding to each sample image at a key time step.

[0013] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the orbital scene prediction method as described above is implemented.

[0014] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements any of the above-described orbital scene prediction methods.

[0015] The present invention also provides a computer program product, comprising a computer program, which, when executed by a processor, implements any of the above-described orbital scene prediction methods.

[0016] The track scene prediction method, apparatus, device, and medium provided by the present invention obtain an RGB image, a depth map, a normal map, and a generation task instruction corresponding to the current frame of the track scene; input the RGB image, the corresponding depth map, the normal map, and the generation task instruction into a lightweight diffusion model to obtain a track scene prediction video corresponding to the generation task instruction output by the lightweight diffusion model; the lightweight diffusion model is trained based on the sample depth maps, sample normal maps, and sample text prompts corresponding to each sample image at a key time step in the denoised trajectory generated by the video diffusion model. In this way, the lightweight diffusion model, trained using multimodal data at the key time step, reduces the model's computational parameters and accelerates the model's prediction speed and accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in the present invention or the prior art, a brief introduction is given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0018] Figure 1 A schematic flow chart of the orbit scene prediction method provided by the present invention.

[0019] Figure 2 It is a flowchart of the lightweight diffusion model training provided by the present invention.

[0020] Figure 3 It is a structural schematic diagram of the orbit scene prediction device provided by the present invention.

[0021] Figure 4 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0022] To make the objectives, technical solutions, and advantages of the present invention more clear, the technical solutions of the present invention will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the embodiments described are only some of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts shall fall within the scope of protection of the present invention.

[0023] The following describes the existing orbit scenario prediction methods.

[0024] Traditional orbit scene prediction methods often rely on single-modality data, such as video surveillance or sensor data. These methods have certain limitations when faced with complex and dynamic environments. On the one hand, single-modality data cannot fully capture the spatiotemporal information of the orbit scene, resulting in insufficient accuracy and robustness of the prediction results. On the other hand, traditional prediction models are often computationally complex and cannot meet real-time requirements.

[0025] In recent years, multimodal data fusion technology has gained increasing attention. By integrating multimodal data such as video, depth maps, and normal maps, a more comprehensive perception of track scenes can be achieved. However, effectively fusing this multimodal data and leveraging its spatiotemporal consistency for efficient prediction remains a challenging problem. Furthermore, existing prediction methods often lack effective mechanisms to fully exploit the spatiotemporal correlations of the data, which impacts the reliability and practicality of the prediction results.

[0026] In addition, although Recurrent Neural Networks (RNNs) and their variants can model time series data to a certain extent, for high-resolution video data, complex models are often required to achieve rail transit scene prediction. Complex models require a lot of resources and a long time. Therefore, the use of complex models may reduce the accuracy and speed of rail scene prediction.

[0027] Based on the above problems, the orbit scenario prediction method provided in this application reduces the model's calculation parameters and accelerates the model's prediction speed and accuracy by using a lightweight diffusion model trained with multimodal data at key time steps.

[0028] The following combination Figure 1 The present invention describes a method for predicting an orbital scenario. The method is applicable to the prediction of any orbital scenario. The execution subject of the method can be an electronic device or a orbital scenario prediction method set in the electronic device. The orbital scenario prediction device can be implemented by software, hardware, or a combination of the two.

[0029] Figure 1 FIG. 1 is a flow chart of the orbit scene prediction method provided by the present invention, such as Figure 1 As shown, the method includes the following: Step 101: Obtain an RGB image corresponding to a current frame in a track scene, a depth map corresponding to the RGB image, a normal map corresponding to the RGB image, and generate a task instruction.

[0030] Here, a depth map is a grayscale image where the value of each pixel represents the distance (depth) from the corresponding scene point to the camera; a normal map is a texture image where each pixel stores the normal direction of the surface at that point (usually expressed in tangent space). Normal maps are used to simulate the details of high-polygon models on low-polygon models.

[0031] Optionally, the depth map can be obtained using depth data provided by the simulator, or by using advanced video depth estimation methods such as rolling depth estimation.

[0032] Optionally, a depth normal function can be used to analyze the change in depth value of each pixel in the depth map to calculate the normal direction of the point, thereby generating a normal map. Alternatively, methods in the optical character recognition engine (TesserAct) can be used to generate high-quality normal maps using tools such as the Temporal-Consistent Marigold (LCM-normal).

[0033] It should be noted that the RGB image corresponding to the current frame may be data that has been normalized, standardized, and noise sampled.

[0034] Typically, noise is sampled from a standard normal distribution, so noise sampling refers to sampling noise from a Gaussian distribution using a random number generator.

[0035] Optionally, the form of generating task instructions includes but is not limited to text description or voice input.

[0036] Step 102: Input the RGB image and the corresponding depth map, normal map, and the generation task instruction into a lightweight diffusion model to obtain a track scene prediction video corresponding to the generation task instruction output by the lightweight diffusion model.

[0037] The lightweight diffusion model is obtained by training an initial lightweight diffusion model based on a sample depth map, a sample normal map, and a sample text prompt corresponding to each sample image at a key time step.

[0038] Optionally, the track scene prediction video can be a video of any length.

[0039] Optionally, the process of generating a track scene prediction video can be to generate the RGB image, depth map and normal map of the next frame based on the RGB image, depth map and normal map of the current frame, and then generate the track scene prediction video based on the RGB image, depth map and normal map of the next frame until the preset length of the video is met; or the track scene prediction video can be generated directly based on the RGB image of the current frame.

[0040] In the embodiment of the present application, the RGB image corresponding to the current frame in the track scene, the depth map corresponding to the RGB image, the normal map corresponding to the RGB image, and the generation task instruction are obtained; the RGB image and the corresponding depth map, normal map, text prompt, and generation task instruction are input into a lightweight diffusion model to obtain a track scene prediction video corresponding to the generation task instruction output by the lightweight diffusion model; the lightweight diffusion model is trained based on the sample depth map, sample normal map, and sample text prompt of each sample image in the denoised track generated by the video diffusion model to an initial lightweight diffusion model. In this way, the lightweight diffusion model trained using the multi-modal data of the key time step reduces the calculation parameters of the model and accelerates the prediction speed and accuracy of the model.

[0041] For example, the lightweight diffusion model includes an encoding module, a diffusion module, and a decoding module. The RGB image and the corresponding depth map, normal map, and generation task instruction are input into the lightweight diffusion model to obtain the track scene prediction video corresponding to the generation task instruction output by the lightweight diffusion model, which includes: inputting the RGB image and the corresponding depth map, normal map, and generation task instruction into the encoding module to obtain the first latent representation corresponding to the current frame output by the encoding module; inputting the first latent representation into the diffusion module to obtain the second latent representation corresponding to the next frame output by the diffusion module; continuously inputting the latent representation currently output by the diffusion module into the diffusion module as the input of the diffusion module to obtain a new latent representation until the video length threshold is met; inputting all latent representations except the first latent representation into the decoding module to obtain the track scene prediction video corresponding to the generation task instruction output by the decoding module.

[0042] Here, the encoding module can be a three-dimensional variational autoencoder (3DVAE), which is used to encode the RGB image and the corresponding depth map and normal map into latent representations, and the compression encoding can compress the high-dimensional data of the RGB image and the corresponding depth map and normal map into a low-dimensional latent space while retaining the main features and structural information of the data. That is, the dimension of the first latent representation is less than that of the corresponding RGB image, depth map, and normal map.

[0043] Here, the diffusion module can be introduced by the Diffusion in Transformer (DiT) mechanism. DiT is a powerful sequence modeling tool that can capture long-term dependencies and complex patterns in data. In DiT, the latent representation is expanded into a sequence and processed through a self-attention mechanism. The self-attention mechanism allows the model to establish connections between different time steps and spatial positions, thereby capturing the spatiotemporal information in video data.

[0044] It should be noted that the second latent representation is the corresponding feature representation of the next frame image predicted based on the first latent representation.

[0045] For example, at each time frame, the lightweight diffusion model predicts the potential representation of the next time step based on the current sampling noise and potential representation. This process is iterative until the complete video sequence is generated. Specifically, at each time step, the model first integrates the potential representation as input, and then uses its own architecture (such as the self-attention mechanism in DiT) to extract input features and capture the spatiotemporal information therein. Based on these features, the model predicts the potential representation of the next time step. Subsequently, the potential representation will be used as input for the next time step, and the generation process will continue until the preset video length is reached. This requires that the model has fully learned the complex patterns and long-term dependencies in the data during the training phase to ensure accurate predictions at each time step, thereby generating videos that are consistent in time and space.

[0046] Finally, during the decoding and output phase, the generated latent representation needs to be restored to the original data, namely the track scene video, through a decoding module (such as a 3D VAE decoder). The decoder has learned how to map the latent representation back to the original data space during model training. The decoding process involves operations such as feature mapping and data reconstruction. After the latent representation is input to the decoder, a series of convolution and upsampling operations gradually restore the resolution and detail of the feature map. Next, at the decoder's output layer, convolution operations map the feature map back to the dimensions and structure of the original data, generating an RGB video, a depth map, and a normal map. Finally, these different modalities are fused together to form a complete track scene video. During this process, alignment and fusion techniques may be required to ensure the generated video is visually consistent and coherent.

[0047] In an embodiment of the present invention, a lightweight diffusion model is constructed by an encoding module, a diffusion module, and a decoding module. The lightweight diffusion model is used to extract complex patterns and long-term dependencies to ensure accurate prediction at each time frame, thereby generating a video that is consistent in time and space, and improving the accuracy, speed, and efficiency of track scene prediction.

[0048] In another embodiment, after generating the scene trajectory prediction video, its spatiotemporal consistency can be verified according to spatiotemporal standards, and its performance can be evaluated by comparing it with real data. The model can be optimized based on the results to further improve the prediction quality.

[0049] For example, Figure 2 This is a flow chart of the lightweight diffusion model training process provided by the present invention, such as Figure 2 As shown, the lightweight diffusion model is trained based on the following steps: Step 201: Obtain a sample depth map, a sample normal map, a sample text prompt corresponding to each sample image in a real sample video corresponding to each key time step, and a sample generation task instruction corresponding to the real sample video.

[0050] It should be noted that different time steps add different degrees of noise to the real sample video. One time step includes multiple image frames. The initial lightweight diffusion model can predict the image of the next frame based on each image frame, and then compare the predicted image with the real image to obtain the difference value. According to the difference value, the model parameters of the initial lightweight diffusion model are continuously trained to obtain the lightweight diffusion model.

[0051] It should be noted that the sample text prompt can be understood as the real label in the RGB image. The sample text prompt generates descriptive text based on the image content. The RGB image of the track scene can be input into the Multimodal Large Language Models (MLLM) to obtain a text describing the objects, actions and environment in the image. This text is the sample text prompt, for example, the InternVL2.5-8B model.

[0052] Here, each sample image, the corresponding sample depth map, and the sample normal map may be pre-processed data, wherein the pre-processing includes but is not limited to normalization, standardization, and noise sampling.

[0053] Exemplarily, the method of obtaining a sample depth map, a sample normal map, and a sample text prompt corresponding to each sample image in a real sample video corresponding to each key time step includes: obtaining an initial RGB video related to a track scene; performing enhancement processing on each initial image frame in the initial RGB video to obtain an enhanced RGB video; determining an enhanced depth map and an enhanced normal map corresponding to each enhanced image frame in the enhanced RGB video; inputting each enhanced image frame into a multimodal large language model to obtain a text prompt corresponding to each enhanced image frame output by the multimodal large language model; and determining a sample depth map, a sample normal map, and a sample text prompt corresponding to each sample image in a first time interval based on the enhanced RGB video, the enhanced depth map, the enhanced normal map, and the text prompt.

[0054] Optionally, the initial RGB video can be a synthetic video generated by a simulator such as RLBench, or a video collected by a sensor or other device in a real track scene.

[0055] Here, the enhancement processing includes but is not limited to changing and fusing different background images with the original track scene image to simulate different environmental conditions; adjusting the light intensity and angle to simulate different lighting conditions; and performing random cropping, rotation, scaling and other operations on the image to increase the diversity of the data.

[0056] Here, the sample image and the corresponding sample depth map, sample normal map, and sample text prompt can be directly selected from the enhanced RGB video, enhanced depth map, enhanced normal map, and text prompt; or the sample image and the corresponding sample depth map, sample normal map, and sample text prompt can be obtained by performing noise sampling on the enhanced RGB video, enhanced depth map, and enhanced normal map.

[0057] In an embodiment of the present invention, multimodal data of track scenes are integrated and a high-quality dataset is constructed, and appropriate data enhancement is performed to provide a rich and effective data foundation for lightweight diffusion model training, thereby improving the model's prediction performance in track scenes.

[0058] Exemplarily, the method of determining the sample depth map, sample normal map, and sample text prompt corresponding to each sample image in the first time interval based on the enhanced RGB video, the enhanced depth map, the enhanced normal map, and the text prompt includes: standardizing each enhanced image, enhanced depth map, and enhanced normal map in the enhanced RGB video to obtain a standard RGB video, a standard depth map, and a standard normal map; inputting the standard RGB video, the standard depth map, and the standard normal map into a video diffusion model to obtain a video denoising trajectory; extracting multiple key time steps from the video denoising trajectory based on the quality of the RGB video; using each frame image of each key time step as a sample image, using the depth map and normal map corresponding to each frame image of the key time step as a sample depth map and sample normal map, respectively, and using the text prompt corresponding to each frame image of the key time step as a sample text prompt.

[0059] For example, the value range of the depth map and the normal map can be normalized to the interval [0, 1]. For RGB images, pixel values ​​can be normalized to map pixel values ​​from the interval [0, 255] to the interval [0, 1].

[0060] It should be noted that the video diffusion model converts data into a Gaussian distribution by gradually adding noise, and then generates data through a denoising process. Here, different noise is added at different time steps, and the frames corresponding to the time steps with good quality are extracted from the denoising trajectory generated by the video diffusion model, and the corresponding depth map, normal map and text prompt are extracted for each frame as a data sample, wherein the data sample includes a sample image, a sample depth map, a sample normal map and a sample text prompt.

[0061] In the embodiments of the present application, the data is further processed by normalization, standardization and noise sampling to ensure the quality and effectiveness of the final data set, providing solid data support for subsequent model training and optimization, and laying a foundation for efficient orbit scene prediction.

[0062] In step 202, the sample images, corresponding sample depth maps, sample normal maps and sample generation task instructions corresponding to the real sample video in each key time step are input into the initial lightweight diffusion model to obtain orbit scene sample prediction videos corresponding to each key time step output by the initial lightweight diffusion model.

[0063] For example, the initial lightweight diffusion model includes an initial encoding module, an initial diffusion module and an initial decoding module. The initial lightweight diffusion model is input into the initial lightweight diffusion model to obtain orbit scene sample prediction videos corresponding to each key time step output by the initial lightweight diffusion model, which includes: inputting the sample depth map, the sample normal map corresponding to each sample image in each key time step and the sample generation task instruction corresponding to the real sample video into the initial encoding module to obtain the sample latent representation corresponding to each key time step output by the initial encoding module; the dimension of the sample latent representation is less than the dimension of the initial feature corresponding to the key time step; input each sample latent representation into the initial diffusion module to obtain a predicted latent representation corresponding to each key time step; input each predicted latent representation into the initial decoding module to obtain an orbit scene sample prediction video corresponding to each key time step.

[0064] Here, the initial encoding module is used to encode the sample image, the corresponding sample depth map and the sample normal map to obtain a compressed sample latent representation, and the initial diffusion model is used to generate a predicted latent representation according to the compressed sample latent representation. The predicted latent representation is input into the initial decoding module to obtain an orbit sample prediction video.

[0065] It should be noted that the initial diffusion module can include a self-attention mechanism. After the potential representation is expanded into a sequence, it is processed through the self-attention mechanism. The self-attention mechanism allows the initial lightweight diffusion model to establish connections between different time steps and spatial positions, thereby capturing the spatiotemporal information in the video data.

[0066] In an embodiment of the present invention, the initial encoding module captures the key features of the data and reduces the computational complexity of the model. The initial diffusion module captures the long-term dependencies and complex patterns in the data. The initial decoder restores the image to achieve an efficient and accurate lightweight scene prediction model.

[0067] Step 203: Based on the target loss function, determine the loss information between the sample text prompt corresponding to the real sample video and the text prompt in each of the track scene sample prediction videos.

[0068] It should be noted that the loss information is used to represent the difference between the sample text prompts corresponding to the real sample video and the text prompts in the sample prediction video of each track scene. The larger the loss information, the greater the difference, and the smaller the loss information, the smaller the difference and the better the model.

[0069] The sample text prompt corresponding to the real sample video refers to the sample text prompt corresponding to each sample image in the real sample video.

[0070] Exemplarily, the target loss function includes a trajectory few-step guidance loss function, a consistency loss function and a regularization loss function, and the loss information between the sample text prompt corresponding to the real sample video and the text prompt in each of the track scene sample prediction videos is determined based on the target loss function, including: based on the trajectory few-step guidance loss function, determining a first loss value between each of the track scene sample prediction videos and the original prediction video; the original prediction video is a video predicted based on a pre-trained diffusion model; based on the consistency loss function and the regularization loss function, determining a second loss value corresponding to the degree of spatiotemporal consistency between adjacent frames in each of the track scene sample prediction videos; based on each of the first loss values ​​and each of the second loss values, determining the loss information between the sample text prompt corresponding to the real sample video and the text prompt in each of the track scene sample prediction videos.

[0071] It should be noted that the sample text prompts can be understood as the real labels in the RGB image. After the track scene sample prediction video is generated according to each key time step, it is necessary to compare the track scene sample prediction video with the text prompts of each sample image in the real sample video to obtain the difference information between the two. Based on the difference information, backpropagation is performed to optimize the initial lightweight diffusion model parameters to obtain the lightweight diffusion model.

[0072] Here, the trajectory few-step guidance loss function is used to train the initial diffusion model to learn the parameters of the diffusion model.

[0073] It should be noted that the original diffusion model is not a lightweight diffusion model, but the model needs to denoise for each time step to generate a video, which requires a large number of model parameters, making the prediction time long and slow, so the trajectory few-step guidance loss function is used to measure the difference between the key data points in the trajectory scene sample predicted video and the key data points generated by the original diffusion model, and the parameters of the initial lightweight diffusion model are optimized through back propagation, in this way, the initial lightweight diffusion model can generate a video with quality comparable to the teacher model in fewer steps. Wherein, the greater the first loss value, the greater the difference between the initial lightweight diffusion model and the original diffusion model, the smaller the first loss value, the smaller the difference between the initial lightweight diffusion model and the original diffusion model.

[0074] Here, the consistency between adjacent frames can be distinguished between background and dynamic regions by optical flow between adjacent frames, that is, the generated video needs to ensure spatiotemporal consistency, the greater the loss value, the worse the spatiotemporal consistency, the smaller the loss value, the better the spatiotemporal consistency.

[0075] The regularization loss function imposes consistency constraints on these regions. Specifically, the model uses optical flow calculation tools such as RAFT to calculate the optical flow field between adjacent frames, thereby determining the temporal correspondence of pixel points, based on this correspondence, the model defines consistency loss in dynamic and background regions, respectively, to ensure the consistency of depth maps over time.

[0076] The regularization loss function then constrains the generation process of the depth map by introducing prior knowledge. For example, it can be assumed that the surface in the scene is smooth, thereby imposing a smoothness constraint on the depth map. By minimizing the regularization loss function, the model can generate depth maps that better conform to the physical world, improving the accuracy of scene prediction.

[0077] In the embodiment of the present application, after selecting the data points of the key time steps, the trajectory few-step guidance mechanism is introduced to construct a shorter noise-to-video mapping path, thereby reducing the number of steps required for generation, and the consistency loss and regularization loss function are introduced to ensure that the generated video is consistent in time and space, improving the accuracy of scene prediction.

[0078] Step 204, based on each of the loss information, constantly update the model parameters of the initial lightweight diffusion model, to obtain the lightweight diffusion model.

[0079] The update process typically uses optimization algorithms such as stochastic gradient descent or Adam. By iteratively updating model parameters, the model can gradually converge to a stable state, resulting in a lightweight diffusion model that achieves consistent spatiotemporal predictions of orbital scenarios.

[0080] In an embodiment of the present invention, a lightweight diffusion model is constructed through an initial encoding module, an initial diffusion module, and an initial decoding module. The trajectory few-step guidance mechanism, the definition of consistency loss and regularization loss functions, and the training and optimization of the model are introduced to ensure that the model can efficiently process track scene data and generate spatiotemporally consistent prediction results, thereby realizing an efficient and accurate lightweight scene prediction model.

[0081] The orbit scene prediction device provided by the present invention is described below. The orbit scene prediction device described below and the orbit scene prediction method described above can be referenced to each other.

[0082] Figure 3 Schematic diagram of the structure of the orbit scene prediction method provided by the present invention. Figure 3 As shown, the orbit scene prediction device 300 includes: An acquisition module 310 is configured to acquire an RGB image corresponding to a current frame in a track scene, a depth map corresponding to the RGB image, a normal map corresponding to the RGB image, and generate a task instruction; The prediction module 320 is used to input the RGB image and the corresponding depth map, normal map and the generation task instruction into a lightweight diffusion model to obtain a track scene prediction video corresponding to the generation task instruction output by the lightweight diffusion model; the lightweight diffusion model is obtained by training an initial lightweight diffusion model based on the sample depth map, sample normal map and sample text prompt corresponding to each sample image at the key time step.

[0083] In another embodiment, the lightweight diffusion model includes an encoding module, a diffusion module, a decoding module, and a prediction module 320, which is specifically used to: input the RGB image and the corresponding depth map, normal map and the generation task instruction into the encoding module to obtain a first potential representation corresponding to the current frame output by the encoding module; input the first potential representation into the diffusion module to obtain a second potential representation corresponding to the next frame output by the diffusion module; continuously use the potential representation currently output by the diffusion module as the input of the diffusion module to obtain a new potential representation until the video length threshold is met, and then input all potential representations except the first potential representation into the decoding module to obtain a track scene prediction video corresponding to the generation task instruction output by the decoding module.

[0084] In another embodiment, the track scene prediction device 300 also includes a training module, which is specifically used to: obtain the sample depth map, sample normal map, sample text prompt corresponding to each sample image in the real sample video corresponding to each key time step, and the sample generation task instruction corresponding to the real sample video; input each sample image, the corresponding sample depth map, sample normal map and the sample generation task instruction corresponding to the real sample video in each key time step into the initial lightweight diffusion model to obtain the track scene sample prediction video corresponding to each key time step output by the initial lightweight diffusion model; based on the target loss function, determine the loss information between the sample text prompt corresponding to the real sample video and the text prompt in each track scene sample prediction video; based on each of the loss information, continuously update the model parameters of the initial lightweight diffusion model to obtain the lightweight diffusion model.

[0085] In another embodiment, the initial lightweight diffusion model includes an initial encoding module, an initial diffusion module and an initial decoding module, and a training module, and is further specifically used to: input the sample depth map, sample normal map and sample generation task instructions corresponding to each sample image in each key time step and the real sample video into the initial encoding module to obtain the sample potential representation corresponding to each key time step output by the initial encoding module; the dimension of the sample representation is smaller than the dimension of the initial feature corresponding to the key time step; input each of the sample potential representations into the initial diffusion module to obtain the predicted potential representation corresponding to each key time step; input each of the predicted potential representations into the initial decoding module to obtain the track scene sample predicted video corresponding to each key time step.

[0086] In another embodiment, the target loss function includes a trajectory few-step guidance loss function, a consistency loss function and a regularization loss function, and the training module is further specifically used to: determine a first loss value between each of the track scene sample prediction videos and the original prediction video based on the trajectory few-step guidance loss function; the original prediction video is a video predicted based on a pre-trained diffusion model; based on the consistency loss function and the regularization loss function, determine a second loss value corresponding to the degree of spatiotemporal consistency between adjacent frames in each of the track scene sample prediction videos; based on each of the first loss values ​​and each of the second loss values, determine the loss information between the sample text prompt corresponding to the real sample video and the text prompt in each of the track scene sample prediction videos.

[0087] In another embodiment, the training module is further specifically used to: obtain an initial RGB video related to the track scene; enhance the initial image of each frame in the initial RGB video to obtain an enhanced RGB video; determine the enhanced depth map and enhanced normal map corresponding to the enhanced image of each frame in the enhanced RGB video; input the enhanced image of each frame into a multimodal large language model to obtain a text prompt corresponding to the enhanced image of each frame output by the multimodal large language model; and determine the sample depth map, sample normal map, and sample text prompt corresponding to each sample image in the first time interval based on the enhanced RGB video, the enhanced depth map, the enhanced normal map, and the text prompt.

[0088] In another embodiment, the training module is further specifically used to: standardize each of the enhanced images, enhanced depth maps and enhanced normal maps in the enhanced RGB video to obtain a standard RGB video, a standard depth map and a standard normal map; input the standard RGB video, the standard depth map and the standard normal map into a video diffusion model to obtain a video denoising trajectory; based on the quality of the RGB video, extract multiple key time steps from the video denoising trajectory; use each frame image of each key time step as a sample image, use the depth map and normal map corresponding to each frame image of the key time step as a sample depth map and sample normal map respectively, and use the text prompt corresponding to each frame image of the key time step as a sample text prompt.

[0089] Figure 4 Schematic diagram of the structure of the electronic device provided by the present invention, such as Figure 4 As shown, the electronic device may include: a processor 410, a communications interface 420, a memory 430, and a communications bus 440, wherein the processor 410, the communications interface 420, and the memory 430 communicate with each other via the communications bus 440. The processor 410 may call logic instructions in the memory 430 to execute a track scene prediction method, which includes: obtaining an RGB image corresponding to a current frame of the track scene, a depth map corresponding to the RGB image, a normal map corresponding to the RGB image, and a generation task instruction; inputting the RGB image, the corresponding depth map, the normal map, and the generation task instruction into a lightweight diffusion model to obtain a track scene prediction video corresponding to the generation task instruction output by the lightweight diffusion model; the lightweight diffusion model is obtained by training an initial lightweight diffusion model based on sample depth maps, sample normal maps, and sample text prompts corresponding to sample images at key time steps.

[0090] Furthermore, the logic instructions in the aforementioned memory 430 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the portion that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product, stored in a storage medium, includes instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the methods described in various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, a mobile hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0091] On the other hand, the present invention also provides a computer program product, which includes a computer program, which can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the track scene prediction method provided by the above methods, which includes: obtaining the RGB image corresponding to the current frame in the track scene, the depth map corresponding to the RGB image, the normal map corresponding to the RGB image and the generation task instruction; inputting the RGB image and the corresponding depth map, normal map and the generation task instruction into a lightweight diffusion model to obtain a track scene prediction video corresponding to the generation task instruction output by the lightweight diffusion model; the lightweight diffusion model is obtained by training the initial lightweight diffusion model based on the sample depth map, sample normal map and sample text prompt corresponding to each sample image at the key time step.

[0092] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the track scene prediction method provided by the above-mentioned methods, the method comprising: obtaining an RGB image corresponding to the current frame in the track scene, a depth map corresponding to the RGB image, a normal map corresponding to the RGB image, and a generation task instruction; inputting the RGB image and the corresponding depth map, normal map, and the generation task instruction into a lightweight diffusion model to obtain a track scene prediction video corresponding to the generation task instruction output by the lightweight diffusion model; the lightweight diffusion model is obtained by training an initial lightweight diffusion model based on sample depth maps, sample normal maps, and sample text prompts corresponding to each sample image at a key time step.

[0093] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, i.e., they may be located in one location or distributed across multiple network units. Some or all of the modules may be selected based on actual needs to achieve the objectives of the present embodiment. Persons of ordinary skill in the art will be able to understand and implement the present invention without inventive effort.

[0094] Through the above description of the embodiments, those skilled in the art will clearly understand that each embodiment can be implemented using software plus a necessary general-purpose hardware platform, or of course, hardware. Based on this understanding, the essence of the above technical solution, or the portion that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a magnetic disk, or an optical disk, and includes a number of instructions for causing a computer device (such as a personal computer, server, or network device) to execute the methods described in each embodiment or certain portions of the embodiments.

[0095] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. A method for predicting an orbital scene, characterized in that: include: Obtain an RGB image corresponding to a current frame in a track scene, a depth map corresponding to the RGB image, a normal map corresponding to the RGB image, and generate a task instruction; The RGB image and the corresponding depth map, normal map and the generation task instruction are input into a lightweight diffusion model to obtain a track scene prediction video corresponding to the generation task instruction output by the lightweight diffusion model; the lightweight diffusion model is obtained by training an initial lightweight diffusion model based on the sample depth map, sample normal map and sample text prompt corresponding to each sample image at a key time step.

2. The orbit scene prediction method according to claim 1, characterized in that: The lightweight diffusion model includes an encoding module, a diffusion module, and a decoding module. The RGB image and the corresponding depth map, normal map, and the generation task instruction are input into the lightweight diffusion model to obtain a track scene prediction video corresponding to the generation task instruction output by the lightweight diffusion model, including: Inputting the RGB image and the corresponding depth map, normal map and the generation task instruction into the encoding module to obtain a first latent representation corresponding to the current frame output by the encoding module; Inputting the first latent representation into the diffusion module to obtain a second latent representation corresponding to the next frame output by the diffusion module; The potential representation currently output by the diffusion module is continuously used as the input of the diffusion module to obtain a new potential representation until a video length threshold is met. Then, all potential representations except the first potential representation are input into the decoding module to obtain the track scene prediction video corresponding to the generation task instruction output by the decoding module.

3. The orbit scene prediction method according to claim 1, characterized in that: The lightweight diffusion model is trained based on the following steps: Obtaining a sample depth map, a sample normal map, a sample text prompt corresponding to each sample image in a real sample video corresponding to each key time step, and a sample generation task instruction corresponding to the real sample video; Inputting each sample image, the corresponding sample depth map, the sample normal map and the sample generation task instruction corresponding to the real sample video in each key time step into the initial lightweight diffusion model, and obtaining the track scene sample prediction video corresponding to each key time step output by the initial lightweight diffusion model; Determining loss information between the sample text prompt corresponding to the real sample video and the text prompt in each of the track scene sample prediction videos based on a target loss function; Based on each of the loss information, the model parameters of the initial lightweight diffusion model are continuously updated to obtain the lightweight diffusion model.

4. The orbit scene prediction method according to claim 3, characterized in that: The initial lightweight diffusion model includes an initial encoding module, an initial diffusion module, and an initial decoding module. The sample generation task instructions corresponding to each sample image, the corresponding sample depth map, the sample normal map, and the real sample video in each key time step are input into the initial lightweight diffusion model to obtain the track scene sample prediction video corresponding to each key time step output by the initial lightweight diffusion model, including: Inputting the sample depth map, sample normal map corresponding to each sample image at each key time step, and the sample generation task instruction corresponding to the real sample video into the initial encoding module, obtaining the sample potential representation corresponding to each key time step output by the initial encoding module; the dimension of the sample potential representation is smaller than the dimension of the initial feature corresponding to the key time step; Inputting each of the sample potential representations into the initial diffusion module to obtain a predicted potential representation corresponding to each key time step; Each of the predicted potential representations is input into the initial decoding module to obtain a track scene sample prediction video corresponding to each key time step.

5. The orbit scene prediction method according to claim 3, characterized in that: The target loss function includes a trajectory few-step guidance loss function, a consistency loss function, and a regularization loss function. The determining, based on the target loss function, of the loss information between the sample text prompt corresponding to the real sample video and the text prompt in each of the track scene sample prediction videos includes: Determining a first loss value between a text prompt of a predicted video of each track scene sample and a text prompt of an original predicted video based on the track few-step guidance loss function; wherein the original predicted video is a video predicted based on a pre-trained diffusion model; Determining, based on the consistency loss function and the regularization loss function, a second loss value corresponding to a degree of spatiotemporal consistency between adjacent frames in each of the track scene sample prediction videos; Based on each of the first loss values ​​and each of the second loss values, loss information between the sample text prompt corresponding to the real sample video and the text prompt in each of the track scene sample prediction videos is determined.

6. The orbit scene prediction method according to claim 3, characterized in that: The step of obtaining a sample depth map, a sample normal map, and a sample text prompt corresponding to each sample image in a real sample video corresponding to each key time step includes: Get the initial RGB video related to the track scene; Performing enhancement processing on each frame of the initial image in the initial RGB video to obtain an enhanced RGB video; Determine an enhanced depth map and an enhanced normal map corresponding to each enhanced image frame in the enhanced RGB video; Inputting each enhanced image frame into a multimodal large language model to obtain a text prompt corresponding to each enhanced image frame output by the multimodal large language model; Based on the enhanced RGB video, the enhanced depth map, the enhanced normal map and the text prompt, a sample depth map, a sample normal map and a sample text prompt corresponding to each sample image in a first time interval are determined.

7. The orbit scene prediction method according to claim 6, characterized in that: The determining, based on the enhanced RGB video, the enhanced depth map, the enhanced normal map, and the text prompt, a sample depth map, a sample normal map, and a sample text prompt corresponding to each sample image in the first time interval includes: Standardizing the enhanced images, enhanced depth maps, and enhanced normal maps in the enhanced RGB video to obtain standard RGB videos, standard depth maps, and standard normal maps; Inputting the standard RGB video, the standard depth map, and the standard normal map into a video diffusion model to obtain a video denoising trajectory; extracting a plurality of key time steps from the video denoising trajectory based on the quality of the RGB video; Each frame image of each key time step is used as a sample image, the depth map and normal map corresponding to each frame image of the key time step are used as a sample depth map and a sample normal map respectively, and the text prompt corresponding to each frame image of the key time step is used as a sample text prompt.

8. A track scene prediction device, characterized in that: include: An acquisition module is used to acquire an RGB image corresponding to a current frame in a track scene, a depth map corresponding to the RGB image, a normal map corresponding to the RGB image, and generate a task instruction; The prediction module is used to input the RGB image and the corresponding depth map, normal map and the generation task instruction into a lightweight diffusion model to obtain a track scene prediction video corresponding to the generation task instruction output by the lightweight diffusion model; the lightweight diffusion model is obtained by training an initial lightweight diffusion model based on the sample depth map, sample normal map and sample text prompt corresponding to each sample image at a key time step.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that: When the processor executes the computer program, the orbit scene prediction method according to any one of claims 1 to 7 is implemented.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the orbit scene prediction method according to any one of claims 1 to 7 is implemented.

Images