Scene data generation method and device, electronic equipment and storage medium

Abstract

The invention provides a scene data generation method and device, electronic equipment and a storage medium, and a scene data generation model is formed by respectively embedding a cross attention module, a position network module and a time convolution attention module in a universal text-to-image generation model. During application, firstly, a global description text is responded through a cross attention module to control a visual style, a layout instruction is responded through a position network module to control a space structure, and generation of a scene style and layout is jointly controlled; and then, independently controlling evolution of a time dimension by using a time convolution attention module, generating scene video data, and accurately decomposing a complex dynamic scene generation task into three mutually independent and cooperative sub-tasks of visual style rendering, spatial layout planning and time dynamic evolution, therefore, the generation of complex scenes such as mines and the like with high visual fidelity, precise and controllable space and real physical movement is realized.

Description

Technical Field

[0001] This invention relates to the field of computer vision generation technology, and in particular to a method, apparatus, electronic device, and storage medium for generating scene data. Background Technology

[0002] The safe deployment of autonomous driving in unstructured industrial environments such as mines depends on the system's robustness in massive and diverse testing scenarios. However, building high-fidelity virtual testing environments presents significant challenges, especially in efficiently generating and accurately controlling "long-tail scenarios" such as equipment collisions, sudden rockfalls, and extreme weather, where data collection is scarce but extremely risky. Existing technologies exhibit clear bottlenecks. Limited by the difficulty and safety costs of real-world data collection, synthesizing such data using generative methods has become a consensus. Current generative methods mainly include implementations based on physics simulation engines / professional simulation software, large-scale models based on holistic fine-tuning, and end-to-end video generation / simulation based on world models.

[0003] However, for scenarios like mines that are complex, dynamic, have a prominent long tail distribution, and feature unique dynamics of heavy equipment, the generative methods mentioned above often struggle to simultaneously satisfy the three elements of high visual fidelity, precise spatial control of key objects, and temporal dynamic physical realism. Summary of the Invention

[0004] This invention provides a method, apparatus, electronic device, and storage medium for generating scene data, which addresses the shortcomings of traditional end-to-end generative models in the prior art, which struggle to achieve scene generation that takes into account the three elements of visual style, spatial layout, and temporal dynamic evolution.

[0005] This invention provides a method for generating scene data, comprising: The system acquires global description text, layout instructions, motion description, and scene data generation model. The scene data generation model is constructed by embedding a cross-attention module, a location network module, and a temporal convolutional attention module into a general text-to-image generation model. The global description text and the layout instructions are input into the scene data generation model. The global description text is applied based on the cross attention module, and the layout instructions are applied based on the location network module to jointly control the generation of scene style and layout. The motion description is applied based on the temporal convolutional attention module to generate scene video data.

[0006] According to a method for generating scene data provided by the present invention, the training steps of the cross-attention module and the location network module include: Acquire the global description text of the sample, sample layout instructions, noise data, as well as the initial cross-attention module and the initial position network module; The global description text of the sample, the sample layout instructions, and the noise data are input into a general text-to-image generation model that embeds the initial cross-attention module and the initial position network module. Based on the initial cross-attention module, the global description text of the sample is applied, and based on the initial position network module, the sample layout instructions are applied to gradually reduce the noise data to obtain the first sample prediction output at each time step. The static prediction loss is calculated based on the first sample prediction output and the first sample label at any time step. Based on the static prediction loss, the network parameters of the initial cross-attention module and the initial location network module are iterated to obtain the cross-attention module and the location network module.

[0007] According to a method for generating scene data provided by the present invention, the step of iterating the network parameters of the initial cross-attention module and the initial location network module simultaneously based on the static prediction loss to obtain the cross-attention module and the location network module includes: The general network parameters of the general text-to-image generation model are frozen; Based on the static prediction loss, the update amount of the first parameter is calculated; the update amount of the first parameter is represented based on a low-rank decomposition matrix. Based on the parameter update amount, the network parameters of the initial cross-attention module and the initial location network module are iterated simultaneously to obtain the cross-attention module and the location network module.

[0008] According to a method for generating scene data provided by the present invention, the training steps of the temporal convolutional attention module include: Acquire sample scene video data, sample generation conditions, and an initial temporal convolutional attention module; the sample generation conditions include global sample description text, sample layout instructions, and sample motion description. The initial temporal convolutional attention module, the cross-attention module, and the location network module are embedded into the general text-to-image generation model to obtain the initial scene data generation model; The noisy scene video data corresponding to the sample scene video data and the sample generation conditions are input into the initial scene data generation model to obtain the second sample prediction output at each time step. The dynamic prediction loss is calculated based on the second sample prediction output and the second sample label at any time step. Based on the dynamic prediction loss, the network parameters of the initial temporal convolutional attention module are iterated to obtain the temporal convolutional attention module.

[0009] According to a method for generating scene data provided by the present invention, the step of iterating the network parameters of the initial temporal convolutional attention module based on the dynamic prediction loss to obtain the temporal convolutional attention module includes: The model parameters of the general text-to-image generation model embedded with the cross-attention module and the location network module are frozen; Based on the dynamic prediction loss, the second parameter update amount is calculated, and the second parameter update amount is represented based on a low-rank decomposition matrix. Based on the second parameter update amount, the network parameters of the initial temporal convolutional attention module are iterated to obtain the temporal convolutional attention module.

[0010] According to a method for generating scene data provided by the present invention, the steps for obtaining the global description text of the sample and the sample layout instructions include: Obtain the initial scene image; The initial scene image is input into a large language model to obtain sample global description text of the initial scene image; The initial scene image is input into the target detection model to obtain the bounding box coordinates of the target detection results in the initial scene image, and the bounding box coordinates are used to generate the sample layout instruction.

[0011] According to a method for generating scene data provided by the present invention, the step of obtaining an initial scene image includes: Obtain scene description text and real scene images; Based on the scene description text, a scene generation prompt text is constructed. The scene description text is then input into the general text-to-image generation model to obtain the supplementary scene image corresponding to the scene description text. The real scene image and the supplementary scene image are used as the initial scene image.

[0012] The present invention also provides a scene data generation apparatus, comprising: The acquisition unit acquires global description text, layout instructions, motion description, and scene data generation model; the scene data generation model is constructed by embedding a cross-attention module, a location network module, and a temporal convolutional attention module into a general text-to-image generation model. The static scene generation unit inputs the global description text and the layout instructions into the scene data generation model, applies the global description text based on the cross attention module, and applies the layout instructions based on the location network module to jointly control the generation of scene style and layout; The dynamic scene generation unit generates scene video data by applying the motion description based on the temporal convolutional attention module.

[0013] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the method for generating scene data as described above.

[0014] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method for generating scene data as described above.

[0015] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the method for generating scene data as described above.

[0016] The method, apparatus, electronic device, and storage medium for generating scene data provided by this invention form a scene data generation model by embedding a cross-attention module, a location network module, and a temporal convolutional attention module into a general text-to-image generation model. In application, the cross-attention module first responds to global descriptive text to control the visual style, and the location network module responds to layout instructions to control the spatial structure, jointly controlling the generation of scene style and layout. Subsequently, the temporal convolutional attention module independently controls the evolution of the temporal dimension, ultimately generating scene video data. This successfully decomposes the complex dynamic scene generation task into three independent yet collaborative sub-tasks: visual style rendering, spatial layout planning, and temporal dynamic evolution. This achieves the generation of complex scenes such as mines that combine high visual fidelity, precise spatial control, and realistic physical motion. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0018] Figure 1 This is a flowchart illustrating the method for generating scene data provided by the present invention; Figure 2This is a flowchart illustrating the concurrent training method for the cross-attention module and the location network module provided by the present invention. Figure 3 This is a flowchart illustrating the training method for the temporal convolutional attention module provided by the present invention; Figure 4 This is a schematic diagram of the training process of the scene data generation model provided by the present invention; Figure 5 This is a schematic diagram of the scene data generation device provided by the present invention; Figure 6 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0019] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0020] It should be understood that the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Therefore, features defined with "first" and "second" may explicitly or implicitly include one or more of the stated features. In the description of embodiments of the present invention, "a plurality of" means two or more, unless otherwise explicitly specified.

[0021] It should be noted that generative methods based on physics simulation engines / professional simulation software offer strong controllability, but they differ significantly from real mining areas in details such as texture, lighting, dust, and materials, resulting in high manpower and time costs for constructing high-precision scenes. Generative methods based on large-scale models with holistic fine-tuning, while improving domain style similarity through end-to-end fine-tuning of a general generative model, are prone to catastrophic forgetting, poor controllability of spatial elements, and kinematic inconsistencies with the physical laws of heavy mining vehicles after extension to video. Furthermore, generative methods based on world models for end-to-end video generation / simulation are structurally complex, require large amounts of training data and computational resources, and face difficulties in accurately intervening in specific scene elements. Therefore, existing scene generation methods struggle to simultaneously satisfy high-fidelity visual reproduction at the sensory level, accurate object space layout at the semantic level, and consistent dynamic temporal sequence at the physical level.

[0022] To address the aforementioned problems, this invention provides a method for generating scene data, enabling precise control over scene style, layout, and dynamics. It should be noted that the scene data generation method provided in this application is applicable to the generation of scene data for complex scenarios such as autonomous driving scenarios in mines. This embodiment of the invention uses a method for generating autonomous driving scenarios in mines as an example for method description. Figure 1 This is a flowchart illustrating the method for generating scene data provided by the present invention, as shown below. Figure 1 As shown, the method includes: Step 110: Obtain the global description text, layout instructions, motion description, and scene data generation model.

[0023] The scene data generation model is constructed by embedding a cross-attention module, a location network module, and a temporal convolutional attention module into a general text-to-image generation model.

[0024] Here, global description text refers to natural language text used to describe the macro-environment, weather conditions, lighting conditions, and overall atmosphere of a mining operation scene, such as a dusty, dimly lit open-pit mine at dusk. Layout instructions refer to control information used to precisely define the spatial location, size, and category of key objects in the scene, such as mining trucks, excavators, and falling rocks; these can typically be represented as a sequence of object-level bounding box coordinates. Motion description refers to the description of the movement state and environmental changes of key objects in the scene, such as vehicle movement and dust generated by braking.

[0025] Furthermore, the scene data generation model here refers to the overall deep learning network architecture constructed to generate the final dynamic video data. The scene data generation model is built by embedding a cross-attention module, a location network module, and a temporal convolutional attention module into a general text-to-image generation model. The general text-to-image generation model refers to a pre-trained large-scale diffusion model, such as Stable Diffusion or GLIGEN-SDXL, which possesses basic image generation capabilities. The cross-attention module refers to the attention mechanism layer in the network used to fuse textual and image features, controlling the visual style of the generated image. The location network module refers to a sub-network structure in the network specifically used to encode and process spatial location information. The temporal convolutional attention module refers to a module embedded in the network layers to process temporal information, typically including temporal convolutional layers and temporal attention layers, used to control the inter-frame coherence and dynamic evolution of the video sequence.

[0026] Specifically, the process begins by acquiring global descriptive text and layout instructions for the scene to be generated. This input data can originate from a real-world mining scene database or be automatically generated using a large language model combined with a mining domain knowledge base. Simultaneously, a pre-built scene data generation model can be loaded. It should be noted that the construction process of the scene data generation model primarily embodies modular and decoupled design principles. The process includes: using a general text-to-image generation model, such as the UNet architecture, as a foundation, firstly, embedding cross-attention modules and location network modules into its network structure to enable image generation that simultaneously responds to text and spatial instructions. Subsequently, a temporal convolutional attention module is further embedded, thereby constructing a scene data generation model with spatiotemporal processing capabilities. For example, a temporal layer module from a pre-trained video generation model, such as Hotshot-XL, can be introduced. This temporal layer module typically consists of temporal convolution and temporal attention layers. Then, these temporal layers are interleaved between the various layers of the solidified spatial UNet network, thus constructing a complete spatiotemporal UNet network architecture capable of simultaneously processing spatial and temporal information, resulting in the scene data generation model.

[0027] In a preferred embodiment, the scene data generation model can be a base model with strong text-to-image generation capabilities and the ability to additionally parse spatial layout instructions, such as GLIGEN-SDXL. Here, the general text-to-image generation model includes a backbone UNet network, which is responsible for progressive denoising in the latent space and is the core of image generation. Based on this general text-to-image generation model, cross-attention layers are embedded in the UNet to inject semantic information from text prompts into the generation process, guiding the generation style. In addition, the GLIGEN model's unique position network module, PositionNet, is used to parse spatial instructions such as bounding boxes, guiding the layout of objects in the image. Finally, a temporal layer module, namely the temporal convolutional attention module, from a pre-trained video generation model such as Hotshot-XL is introduced to construct the scene data generation model.

[0028] It should be noted that by constructing a scene data generation model containing specific functional modules, the decoupling preparation of the generation task is achieved. The model is divided into a spatial part that processes static visual information and a temporal part that processes dynamic evolution. The static part is further subdivided into a cross-attention module that processes style and a position network module that processes layout. This lays the foundation for the subsequent precise control of the three core elements of scene style, layout and motion, and overcomes the shortcomings of traditional end-to-end models that cannot simultaneously take into account the control accuracy of each element.

[0029] Step 120: Input the global description text and the layout instructions into the scene data generation model. Apply the global description text based on the cross-attention module and apply the layout instructions based on the location network module to jointly control the generation of scene style and layout.

[0030] Specifically, the acquired global descriptive text and layout instructions are input into the scene data generation model. During the forward propagation of the scene data generation model, the cross-attention module receives the embedding vector of the global descriptive text and injects semantic information about the mine environment, weather, and lighting into the generated image features, thereby dominating the scene style rendering of the generated image. Simultaneously, the location network module receives the layout instructions, encodes the bounding box coordinates of objects such as mining trucks and excavators into location features, and guides the model to generate the corresponding object entities in the specified pixel regions. These two modules work concurrently during the generation process, jointly controlling the generation of scene style and layout.

[0031] It should be noted that by simultaneously utilizing the cross-attention module and the location network module, concurrent decoupled control of visual style and spatial layout is achieved. This mechanism forces the model to incorporate both style and layout constraints during the generation process, ensuring that the final generated scene video data visually presents realistic mining environment characteristics, such as specific rock textures and industrial dust. Simultaneously, it achieves precise spatial positioning of key hazards or operating equipment, solving the technical challenge of balancing image style realism and spatial location controllability in existing technologies.

[0032] Step 130: Apply the motion description based on the temporal convolutional attention module to generate scene video data.

[0033] Here, scene video data refers to a sequence of images with temporal continuity that extends from scene images. This sequence vividly demonstrates the motion state of an object and environmental changes over a period of time, such as vehicle movement, braking, and dust.

[0034] Specifically, activating the temporal convolutional attention module in the scene data generation model involves inputting the latent spatial features corresponding to the scene style and layout as conditions, using motion descriptions as conditions. The temporal convolutional attention module then handles the temporal correlations in sequence generation. More specifically, it captures local temporal dependencies through temporal convolution operations and processes long-distance inter-frame correlations through a temporal attention mechanism. Based on preset motion logic or learned physical laws, it drives pixel displacement and changes in static scenes. For example, based on static images of mining trucks, it generates continuous footage of the trucks starting, turning, or braking under heavy load conditions, ultimately outputting a scene video data that conforms to physical laws. For instance, it outputs a 5-10 second short video sequence, vividly reproducing a high-risk, long-tail scenario of a mining truck urgently avoiding a suddenly appearing excavator in dust at dusk. This video data can be directly used for simulation testing and training of autonomous driving perception and control algorithms.

[0035] It should be noted that by independently applying the temporal convolutional attention module on top of the combined effect of the cross-attention module and the location network module, dynamic information can be accurately injected without destroying the original visual style and spatial layout features, so that the movement of objects such as mining trucks and excavators in the generated video sequence strictly conforms to the real dynamic characteristics under heavy-load conditions.

[0036] It should also be noted that by orthogonally decoupling the three core attributes of the scene—visual style, spatial layout, and temporal dynamics—and constructing independent, parameter-efficient, expert-level low-rank adaptive network modules for each, this approach fundamentally differs from existing technologies that rely on holistic fine-tuning or single black-box models.

[0037] The method provided in this invention forms a scene data generation model by embedding a cross-attention module, a location network module, and a temporal convolutional attention module into a general text-to-image generation model. In application, the cross-attention module first responds to global descriptive text to control visual style, and the location network module responds to layout instructions to control spatial structure, jointly controlling the generation of scene style and layout. Subsequently, the temporal convolutional attention module independently controls the evolution of the temporal dimension, ultimately generating scene video data. This successfully decomposes the complex dynamic scene generation task into three independent yet collaborative sub-tasks: visual style rendering, spatial layout planning, and temporal dynamic evolution. This achieves the generation of complex scenes such as mines that combine high visual fidelity, precise spatial control, and realistic physical motion.

[0038] Based on any of the above embodiments Figure 2 This is a flowchart illustrating the concurrent training method for the cross-attention module and the location network module provided by the present invention, as shown below. Figure 2As shown, the method includes: Step 210: Obtain the global description text of the sample, the sample layout instructions and noise data, as well as the initial cross-attention module and the initial position network module.

[0039] Here, the global description text of the samples refers to the natural language annotations in the training dataset used to describe the characteristics of the real-world scene environment, such as "open-pit mine, dusty, strong sunlight." The sample layout instructions refer to the list of bounding boxes in the training dataset corresponding to the images, annotating the categories and location coordinates of key objects in the scene, such as mining trucks and excavators. The noisy data refers to the image data obtained by adding Gaussian noise at a specific time step to the real sample images during the training of the diffusion model, or it can be pure Gaussian noise data, depending on whether the prediction target is noise or the original image.

[0040] In addition, the initial cross-attention module and the initial position network module here refer to the sub-networks to be trained that are inserted into the general text to image generation model and whose parameters are in the initial state.

[0041] Specifically, a structured, multimodal mixed reality training dataset is first constructed, containing triplets of [image, global text description, and object-level bounding box list]. The global description text and sample layout instructions are extracted from this dataset. Simultaneously, based on the training mechanism of the diffusion model, real sample images are encoded into the latent space, and noise is added according to random sampling time steps to generate noisy data. Here, the generation of noisy data can be expressed based on the following formula, as shown in the following equation: ; In the formula, Representing latent variables of the original image conduct Latent spatial representation of data after adding noise at time steps; , These represent noise scheduling coefficients, indicating how much information of the original image is retained and how much noise intensity is added, respectively. This represents Gaussian noise.

[0042] In addition, regarding model construction, a pre-trained general text-to-image generation model is selected as the base, keeping its original parameters frozen. Here, the general text-to-image generation model can be GLIGEN-SDXL. Then, initial cross-attention modules and initial position network modules are embedded at key positions of the general text-to-image generation model.

[0043] Step 220: Input the global description text of the sample, the sample layout instructions, and the noise data into a general text-to-image generation model that embeds the initial cross-attention module and the initial position network module. Apply the global description text of the sample based on the initial cross-attention module, and apply the sample layout instructions based on the initial position network module to gradually reduce the noise data to obtain the first sample prediction output at each time step.

[0044] Here, the first sample prediction output refers to the prediction result of the noise value added to the noisy data during the forward propagation process of training, based on the current input conditions and the current network parameter state.

[0045] Specifically, the forward propagation process of the model is executed. Noisy data is input into the backbone network of the general text-to-image generation model. Simultaneously, the initial cross-attention module receives the feature vector of the global descriptive text of the samples, injecting semantic information into the network to guide style generation. Additionally, the initial location network module receives sample layout instructions, such as bounding boxes, encoding them into a location embedding injection network to guide object placement. Thus, the text-to-image generation model, embedding the initial cross-attention module and the initial location network module, processes the input noisy data at the current time step, comprehensively considering style and layout constraints, and calculates and outputs the first sample prediction output for each time step. For example, the model attempts to predict the noise components contained in the current image against a mining background, given the presence of a mining card at specified coordinates.

[0046] It should be noted that by simultaneously activating the cross-attention module responsible for style and the location network module responsible for layout in a single forward propagation, the model is forced to establish a strong correlation between style and layout when understanding the scene. This mechanism simulates the real generation process, that is, what the mining card should look like and where it should be located in the style of a mine, thus providing accurate prediction values ​​for subsequent concurrent parameter optimization.

[0047] Step 230: Calculate the static prediction loss based on the first sample prediction output and the first sample label at any time step.

[0048] Here, the first sample label refers to the true value used as the standard answer during training, which is usually the noise value actually added to the image, or a real noise-free sample image. The static prediction loss is a value calculated using a loss function that measures the difference between the first sample prediction output and the first sample label; this value reflects the error in the model's current generation capability.

[0049] Specifically, the difference between the predicted output of the first sample and the label of the first sample is calculated, such as by calculating the L2 norm distance, to obtain the static prediction loss. It should be noted that if a general text-to-image generation model uses... In the -prediction form, the static prediction loss can be calculated based on the following formula, as shown below: ; in, Indicates adoption -Static prediction loss in the form of prediction; This represents the random noise actually added to the real noise, which can be understood as the first sample label; Indicates the input conditions In this case, a general text-to-image generation model is constructed by embedding an initial cross-attention module and an initial position network module. Model parameters In the Noise-sensitive data output at time steps The first sample prediction output obtained from the prediction is the predicted noise. This represents the global description text of the sample; This indicates a sample layout instruction.

[0050] Furthermore, if the general text-to-image generation model uses v-prediction, the static prediction loss can be calculated based on the following formula, as shown below: ; In the formula, This represents the static prediction loss in the form of v-prediction; This represents a more stable prediction target, which is noise. and the original image The linear combination of these factors provides more numerically stable predictions when the signal-to-noise ratio changes drastically. ; Indicates the input conditions In this case, a general text-to-image generation model is constructed by embedding an initial cross-attention module and an initial position network module. In parameters ,exist Time step for noisy data The speed of noise prediction.

[0051] Step 240: Based on the static prediction loss, the network parameters of the initial cross-attention module and the initial location network module are iterated simultaneously to obtain the cross-attention module and the location network module.

[0052] Specifically, a concurrent training mechanism is employed during the backpropagation phase. That is, the gradient generated by the same static prediction loss is simultaneously fed back to both the initial cross-attention module and the initial position network module. It should be noted that here, the original parameters of the general model are kept completely frozen; only the parameters of these two modules are updated, without introducing additional weight coefficients or regularization terms. This process is repeated on a large amount of sample data until the loss converges, ultimately resulting in the trained cross-attention module (style expert) and position network module (layout expert).

[0053] It is important to emphasize that by constraining both modules simultaneously using the same loss function, the visual style and spatial layout learned by the cross-attention module and the location network module are highly coordinated. For example, training the layout first and then the style might lead to the layout features being overridden by the style features. Furthermore, by updating only a small number of embedding module parameters, such as LoRA parameters, catastrophic forgetting of general knowledge from the basic model is effectively prevented while maintaining generation quality, ensuring that the generated mine scene is both realistic and controllable.

[0054] Furthermore, after training, the optimized parameters of the cross-attention module and the location network module are extracted. These two well-trained module parameters are then formally loaded or merged into the original general text-to-image generation model. At this point, the model is transformed into an image generation model with expertise in the mining field, capable of controlling the generation of scene style and layout based on new inference instructions.

[0055] Based on any of the above embodiments, step 240 includes: The general network parameters of the general text-to-image generation model are frozen; Based on the static prediction loss, the update amount of the first parameter is calculated; the update amount of the first parameter is represented based on a low-rank decomposition matrix. Based on the parameter update amount, the network parameters of the initial cross-attention module and the initial location network module are iterated simultaneously to obtain the cross-attention module and the location network module.

[0056] Here, the first parameter update amount refers to the amount of change that the network weights need to be adjusted during training to minimize the loss function, denoted as . Here, the low-rank decomposition matrix refers to the product of two low-rank matrices used to approximate the parameter update amount, denoted as BA, where the ranks of matrices B and A are much smaller than the dimension of the original weight matrix.

[0057] Specifically, firstly, at the start of training, all the original weight parameters of the general text-to-image generation model can be marked as untrainable through the interface of the deep learning framework. This means that during backpropagation, these parameters will not receive gradient updates and will always retain their initial values ​​from pre-training. For example, for a model with billions of parameters, this step locks the vast majority of parameters, keeping only the parameters of newly inserted modules active. It should be noted that by freezing the general network parameters, the memory usage and computational resource consumption during training are greatly reduced.

[0058] Next, the principle of low-rank adaptive LoRA can be applied. Instead of directly calculating the update amount of the full weights, the parameter update amount to be learned is decomposed into the product of two low-rank matrices. In forward propagation, the input data passes through the frozen original weights and the trainable low-rank matrix branches respectively; in backpropagation, based on the static prediction loss, the chain rule can be used to calculate only the gradients of these two low-rank matrices A and B. For example, if the original weights are... Then we introduce rank ( Two matrices , Therefore, the training objective is only to optimize the two small matrices A and B. The forward propagation process of the model can be represented by the following formula, as shown below: ; In the formula, The output refers to the scene style and layout output by the general text-to-image generation model that embeds the initial cross-attention module and the initial position network module. This represents the model parameters of a general text-to-image generation model; This indicates the amount of change that needs to be adjusted, namely the network parameters of the initial cross-attention module and the initial position network module; This indicates input.

[0059] In detail, for the initial cross-attention module, its trainable weight matrix ( This is strategically injected into the cross-attention module of the backbone UNet network. Its training task is to learn the complex mapping relationship between the global descriptive text of the samples and the visual features unique to the mining environment, such as the texture of exposed rock, the diffuse feeling of industrial dust, and the color atmosphere under specific lighting conditions. Additionally, for the initial position network module, its trainable weight matrix ( This is specifically injected into the location network module, whose training task is to learn the mapping relationship between sample layout instructions and the precise generation of corresponding objects such as specific models of mining trucks and excavators at specified locations in the image.

[0060] It should be noted that by employing a parameter-efficient low-rank adaptive technique, only a very small number of parameters need to be trained, significantly reducing the demand for computing resources. Compared to training large-scale world models or constructing complex physical simulation environments, the fine-tuning training method provided in this embodiment of the invention offers significant advantages in terms of economy and timeliness.

[0061] Finally, a concurrent optimization strategy can be executed. The same static prediction loss, calculated based on the same batch of data, is simultaneously backpropagated to the initial cross-attention module (Style LoRA) responsible for visual style and the initial position network module (Layout LoRA) responsible for spatial control. The optimizer synchronously updates the A and B matrix parameters in these two modules based on their respective calculated low-rank gradients. After a preset number of iterations, the parameters of these two modules gradually converge. Ultimately, a cross-attention module and a position network module capable of accurately responding to mine-style text and complex layout instructions are obtained.

[0062] The method provided in this invention introduces a low-rank decomposition matrix to represent the parameter update amount, compressing the number of parameters to be optimized to a minimum ratio of the original model. This not only significantly improves the training speed and convergence efficiency, but also allows the parameter update amounts trained for different tasks to be stored and distributed as lightweight plugins, achieving efficient modularization of model functions.

[0063] It should be noted that extending the aforementioned high-fidelity static scene into short video clips that are temporally coherent and realistically motionous is fundamentally about solving the kinematic simulation problem for special objects such as heavy machinery. Based on any of the above embodiments, Figure 3 This is a flowchart illustrating the training method for the temporal convolutional attention module provided by the present invention, as shown below. Figure 3 As shown, the method includes: Step 310: Obtain sample scene video data, sample generation conditions, and initial temporal convolutional attention module.

[0064] The sample generation conditions include global sample description text, sample layout instructions, and sample motion description.

[0065] Here, sample scene video data refers to continuous video clips collected from the real world that reflect the actual movement patterns of objects in mining operations, such as heavy mining trucks driving on unstructured roads, excavators digging, or videos of falling rocks. The sample generation conditions include conditional text describing visual style, spatial layout, and temporal dynamic evolution, specifically global sample description text, sample layout instructions, and sample motion description. The sample motion description refers to the description of the motion state of key objects and the environmental state in the sample scene video data.

[0066] Additionally, here, the initial temporal convolutional attention module refers to the neural network component to be trained for handling temporal dependencies. It typically consists of convolutional layers and attention layers in the temporal dimension, and its parameters are in an initialized state.

[0067] Specifically, real-world video data of mining scenes is extracted from a pre-built mixed reality dataset. This video data is preprocessed to a uniform frame rate and resolution, serving as sample data for the model to learn the laws of physical motion. Simultaneously, a set of temporal convolutional attention modules can be initialized. These modules can be designed to be adaptably inserted into the network layers of a general text-to-image generation model that embeds the initial cross-attention module and the initial position network module. In a preferred embodiment, these modules can also be defined in a low-rank adaptive form, i.e., as a dynamic expert LoRA, to reduce the number of training parameters.

[0068] Step 320: Embed the initial temporal convolutional attention module, the cross-attention module, and the location network module into the general text-to-image generation model to obtain the initial scene data generation model.

[0069] Here, the cross-attention module and the location network module refer to the already trained network modules that possess mine-style and layout control capabilities. At this point, their internal general parameters and style / layout expert parameters have typically been merged or frozen. The initial scene data generation model refers to a novel network architecture that adds temporal processing capabilities to the spatial-dimensional network structure.

[0070] Specifically, initial temporal convolutional attention modules can be interleaved and inserted between the spatial layers of the fixed text-to-image generation model. For example, a temporal Transformer module can be added after each spatial Transformer module in UNet. In this case, the model transforms from a 2D network capable of processing only a single image into a 3D network capable of processing image sequences, forming an initial scene data generation model with initial spatiotemporal processing capabilities. Here, the fixed text-to-image generation model is essentially a general text-to-image generation model with the trained cross-attention and location network modules inserted.

[0071] It should be noted that the model dimension was upgraded by structural embedding, which ensures that the model can directly reuse the pre-trained high-quality static features when processing the spatial features of each frame of image. The newly added temporal module is specifically responsible for handling the correlation between frames, thus achieving spatial and temporal decoupling at the architecture level.

[0072] Step 330: Input the noisy scene video data corresponding to the sample scene video data and the sample generation conditions into the initial scene data generation model to obtain the second sample prediction output for each time step.

[0073] Here, noisy scene video data refers to the data where each frame of the sample scene video data is encoded in the latent space, and then a noise sequence with a specific time step is added. The noisy scene video data corresponding to the sample scene video data can be calculated using the following formula, as shown below: ; In the formula, Indicates the first Frame video frames in Latent spatial representation of noisy scene video frames at time steps; and express The noise scheduling coefficients at each time step represent the original image information retained and the intensity of the added noise, respectively. Indicates the first Latent spatial representation of sample scene video images of a video frame; Indicates the first Gaussian noise in each video frame. The latent variables of the sample scene video data can be denoted as... .

[0074] Additionally, the second sample prediction output here refers to the model's prediction of the original noise sequence or the denoised video sequence for the input noisy video sequence.

[0075] Specifically, the forward propagation process of the video diffusion model is executed. The noisy scene video data and sample generation conditions are input into the initial scene data generation model. During computation, the cross-attention module and the location network module are responsible for extracting visual style features and spatial features within each frame, while the newly embedded temporal convolutional attention module is responsible for transferring information between frame sequences, capturing the continuity of motion and causal relationships, and finally outputting the second sample prediction output for the entire video sequence.

[0076] Step 340: Calculate the dynamic prediction loss based on the second sample prediction output and the second sample label at any time step.

[0077] Here, the second sample label refers to the real noise sequence or the original noise-free video data used as the training ground value. The dynamic prediction loss is a function that measures the difference between the generated video sequence and the real physical motion video sequence, typically using the sequence-level mean squared error.

[0078] Specifically, the second sample prediction output from the initial scene data generation model is compared with the corresponding second sample label. The difference between the two over the entire video sequence is calculated to obtain the dynamic prediction loss. It can be understood that the dynamic prediction loss function here mainly reflects the accuracy of the initial scene data generation model in simulating object motion trajectories, velocity changes, and dynamic environmental responses.

[0079] In one embodiment, if the base model adopts In this form, the dynamic prediction loss can be calculated using the following formula: ; In the formula, This represents the dynamic prediction loss corresponding to the sample scene video data. This represents the total number of frames in the sample scene video data. Indicates the index of the video frame; In the sample scene video data, the first... Noise in the frame image; Indicates the conditions for sample generation In this case, a model is generated from the initial scene data. Model parameters In the The time-step output is for the entire noisy video sequence. The The second sample prediction output obtained by predicting each video frame is the predicted noise. This represents the global description text of the sample; Indicates sample layout instructions; This describes the motion of the sample.

[0080] In another embodiment, if the base weights adopt the prediction form of v-prediction, the dynamic prediction loss can be calculated using the following formula: ; In the formula, This represents the dynamic prediction loss using the v-prediction prediction method. This represents the total number of frames in the sample scene video data. Indicates the index of the video frame; In the sample scene video data, the first... The velocity target of the frame image ; Indicates input conditions In this case, a model is generated from the initial scene data. parameters ,exist Time steps for the entire noisy video sequence The The speed of noise prediction per video frame.

[0081] Step 350: Based on the dynamic prediction loss, iterate the network parameters of the initial temporal convolutional attention module to obtain the temporal convolutional attention module.

[0082] Specifically, during the backpropagation phase, the network parameters of the initial temporal convolutional attention module can be iteratively evaluated by dynamically predicting the loss to obtain the temporal convolutional attention module. Here, the network parameters of the initial temporal convolutional attention module can be represented as a trainable weight matrix. Therefore, the weight matrix Update the network. Here, while iterating the network parameters, freeze the network parameters of the general model, the cross-attention module, and the location network module. This process is repeated on a large amount of sample data until the loss converges, finally obtaining the trained temporal convolutional attention module, i.e., the dynamic expert.

[0083] The method provided in this invention iterates over the network parameters of the initial temporal convolutional attention module using dynamic prediction loss, with the optimization target limited to the parameters of the dynamic expert LoRA module injected into the temporal layer. This targeted fine-tuning strategy precisely modifies the model's temporal processing pathway without affecting the already well-established spatial processing pathway. Therefore, by optimizing only a very small number of parameters, the model can learn dynamic patterns specific to a particular domain, such as the unique kinematic characteristics of a mining truck starting and turning under heavy load. Furthermore, it fundamentally avoids damaging or contaminating the hard-earned high-fidelity visual style and precise spatial control capabilities when learning dynamic knowledge, ensuring that the final generated video maintains extremely high visual quality in every frame.

[0084] Based on any of the above embodiments, the network parameters of the initial temporal convolutional attention module are iterated based on the dynamic prediction loss to obtain the temporal convolutional attention module, including: The model parameters of the general text-to-image generation model embedded with the cross-attention module and the location network module are frozen; Based on the dynamic prediction loss, the second parameter update amount is calculated, and the second parameter update amount is represented based on a low-rank decomposition matrix. Based on the second parameter update amount, the network parameters of the initial temporal convolutional attention module are iterated to obtain the temporal convolutional attention module.

[0085] Specifically, firstly, at the start of training, all the original weight parameters of the general text-to-image generation model with embedded cross-attention modules and location network modules can be marked as untrainable through the interface of the deep learning framework. This means that during backpropagation, these parameters will not receive gradient updates and will always retain their initial values ​​from pre-training.

[0086] It should be noted that after solidifying the spatial layer of style and layout knowledge, targeted fine-tuning can be performed through a temporal convolutional attention module that only operates on the model's temporal layer. This allows for the precise injection of domain-specific kinematic laws and avoids the catastrophic forgetting of high-fidelity visual features from a mechanistic perspective.

[0087] Next, the principle of Low-Rank Adaptive LoRA can be applied. Instead of directly calculating the update amount of the full weights, the parameter update amount to be learned is decomposed into the product of two low-rank matrices. In forward propagation, the input data passes through the frozen original weights and the trainable low-rank matrix branches respectively; in backpropagation, based on the dynamic prediction loss, the chain rule can be used to calculate only the gradients of these two low-rank matrices A and B. For example, if the original weights are... Then we introduce rank ( Two matrices , Therefore, the training objective is simply to optimize these two small matrices, A and B.

[0088] Finally, an optimization strategy is executed, backpropagating the same dynamic prediction loss calculated based on the same batch of data to the initial temporal convolutional attention module responsible for temporal dynamics. Here, the optimizer synchronously updates the parameters of matrices A and B in these two modules based on their respective calculated low-rank gradients. After a preset number of iterations, the parameters of the initial temporal convolutional attention module gradually converge. Ultimately, the trained temporal convolutional attention module, cross-attention module, and location network module are embedded into a general text-to-image generation model, resulting in a scene data generation model that can accurately respond to mine-style text, complex layout instructions, and physically consistent scenes.

[0089] The method provided in this invention introduces a low-rank decomposition matrix to represent the parameter update amount, compressing the number of parameters to be optimized to a minimum ratio of the original model. This not only significantly improves the training speed and convergence efficiency, but also allows the parameter update amounts trained for different tasks to be stored and distributed as lightweight plugins, achieving efficient modularization of model functions.

[0090] Understandably, if further refinement of the scene generation dimensions is required, the parameters of the trained model can be frozen, and corresponding functional modules can be added to the trained model to learn low-rank adaptive network modules, making the scene data generation model highly flexible and scalable.

[0091] In other words, the functionality of the scene data generation model can be defined by pluggable expert network modules, facilitating independent iteration, troubleshooting, and upgrades. When adapting to new mining areas or new operational processes, only minor adjustments or additions of corresponding expert network modules are needed, greatly improving the system's maintainability and cross-scene migration capabilities. For example, the system's generation capabilities can be easily expanded by adding new weather expert network modules or sensor noise expert network modules without the need for costly retraining of the entire model.

[0092] Based on any of the above embodiments, the steps for obtaining the global description text of the sample and the sample layout instructions include: Obtain the initial scene image; The initial scene image is input into a large language model to obtain sample global description text of the initial scene image; The initial scene image is input into the target detection model to obtain the bounding box coordinates of the target detection results in the initial scene image, and the bounding box coordinates are used to generate the sample layout instruction.

[0093] Here, the initial scene image carries various visual information about the scene, including visual style, spatial location, and temporal dynamics. Subsequent processing is performed on this image to obtain global description text and sample layout instructions.

[0094] Specifically, firstly, scene images matching the characteristics of mining areas can be selected from a dedicated image library for autonomous driving in mines. The acquired scene images can then undergo mine scene adaptation preprocessing, including adjusting the image resolution to the model adaptation standard and enhancing the contrast of typical mine features. Then, the processed images are used as the initial scene images.

[0095] Here, the image library sources include real-time data collected by mining vehicle-mounted cameras, extracted video frames from mining area monitoring, and publicly available mining autonomous driving datasets. Image selection criteria cover different weather conditions, time periods, and different mining operation areas to ensure that the initial scene images cover typical mining scene variations. The image library can be sourced from publicly available datasets such as AutoMine.

[0096] Then, the initial scene image can be input into a large-scale language model to obtain a sample global description text for the initial scene image. This description text is then generated through multiple rounds of semantic reasoning using a large-scale language model such as Qwen2.5-VL, serving as the sample global description text corresponding to the initial scene image. This process goes beyond simple object recognition, focusing more on a deeper understanding of scene atmosphere, environmental conditions, and events, outputting high-level semantic labels such as "In the evening, under dusty conditions, an excavator is working on earthmoving."

[0097] Additionally, the initial scene image can be input into an object detection model such as Grounding DINO. This model automatically locates and labels predefined objects in the initial scene image, such as mining trucks, excavators, personnel, and warning signs, using bounding boxes. It's worth noting that the object detection model can precisely associate text prompts with specific pixel regions in the image, generating bounding box coordinates for each object. Therefore, the target detection results and their bounding box coordinates can be used as sample layout instructions.

[0098] Finally, the semantic and spatial annotations corresponding to the initial scene images can be associated, that is, the global description text and sample layout instructions of the same initial scene image can be associated, and manual review and correction can be carried out to ensure the high quality and consistency of the data.

[0099] The method provided in this invention outputs global description text of the initial scene image through a large language model and outputs sample layout instructions of the initial scene image through an object detection model, thereby achieving high-quality and efficient batch semantic annotation and spatial location annotation, and improving the quality of training data and training efficiency.

[0100] It should be noted that, in order to further improve the data quality of the training data, based on any of the above embodiments, obtaining the initial scene image includes: Obtain scene description text and real scene images; Based on the scene description text, a scene generation prompt text is constructed. The scene description text is then input into the general text-to-image generation model to obtain the supplementary scene image corresponding to the scene description text. The real scene image and the supplementary scene image are used as the initial scene image.

[0101] Here, scene description text refers to scenes not covered by real-world scene images. Examples include scenes with abnormal weather conditions such as dense fog and blizzards; scenes with abnormal lighting conditions such as strong backlighting and dust storms; scenes with abnormal events such as equipment intrusion and slope collapses; and scenes with abnormal road surfaces such as large puddles and icy surfaces. Real-world scene images refer to multi-source data from actual mining environments, including but not limited to continuous video streams and discrete image frames captured by vehicle-mounted forward-facing cameras, such as those from publicly available datasets like AutoMine.

[0102] Specifically, firstly, a text corpus describing long-tail scenarios in mines can be constructed by combining domain expert knowledge to obtain scene description text. Secondly, continuous video streams and discrete image frames captured by a vehicle-mounted forward-looking camera can be used as real-world scene images. Then, the scene description text can be filled into a pre-constructed prompt text template to generate scene-generated prompt text.

[0103] Furthermore, the scene-generated prompt text can be batch-input into a large-scale, general-purpose text-to-image generation model to obtain supplementary scene images corresponding to the scene description text. This generates a comprehensive long-tail scene initial image corpus, containing key scenes that are scarce or missing in real-world datasets, such as equipment collisions, sudden rockfalls, and extreme weather. Finally, the real-world scene images and supplementary scene images are used as initial scene images to train the model. It is understood that the generated supplementary scene images will serve as an important supplement to the real-world dataset, jointly forming a hybrid data source used in subsequent annotation and training stages, achieving efficient construction of a comprehensive and balanced training dataset.

[0104] It should be noted that complex long-tail scenes in the mine are generated based on a general text-to-image generation model. For example, a mining truck encounters an excavator that suddenly appears on the road in a dusty environment at dusk. Generally, long-tail scenes incorporate special lighting conditions, harsh environments, and hazardous working conditions; this type of data is difficult to collect on a large scale and safely in the real world. It should be noted that the definition of a long-tail scene in the mine can include: weather / environment including blizzards, heavy rain, sandstorms, dense fog, extreme high / low temperatures; lighting / visual conditions including nighttime, strong backlighting, strong reflections from black, white, and gray rocks, and glare from lights; road / terrain conditions including icing, water accumulation / mud, potholes, uphill curves, and cliff edges; objects / equipment including mining trucks, excavators, loaders, pickup trucks, pedestrians, and wild animals; and time / hazard sources including rockfalls, landslides, equipment malfunctions, intrusions by personnel, and oncoming vehicles.

[0105] The method provided in this invention constructs scene generation prompt text through scene description text, inputs the scene description text into the general text-to-image generation model, and obtains supplementary scene images corresponding to the scene description text. It can generate test data covering various conventional and extreme weather, lighting, sudden events and complex interactions on demand, at large scale and at low cost. It fundamentally solves the bottleneck of traditional methods in obtaining long-tail data, provides a complete and reliable virtual testing environment for autonomous driving systems, and significantly enhances the robustness and safety of the algorithm.

[0106] It should be noted that for complex, dynamic, and long-tailed scenarios like mines, where heavy equipment dynamics are unique, the above-mentioned approaches often fail to simultaneously satisfy: high-fidelity domain vision; precise spatial control of objects / hazards; temporal dynamic physical realism; and both training efficiency and modular scalability. Therefore, a decoupled, parameter-efficient generation framework, distinct from holistic fine-tuning, is needed: orthogonally separating the scene's style, layout, and motion, learning them separately, and coordinating their control. This ensures that the generated data maintains both visual realism and semantic consistency within the mine area, while also placing hazardous elements such as falling rocks, water accumulation, and personnel intrusion in designated locations, and ensuring that the kinematics of the mining trucks / excavators in the video sequence conform to the real dynamic characteristics under heavy-load conditions.

[0107] In one embodiment, Figure 4 This is a schematic diagram of the training process of the scene data generation model provided by the present invention, as shown below. Figure 4 As shown, the training process consists of three stages: the first stage is data preparation, and the second and third stages are fine-tuning training.

[0108] In the first stage, the data preparation stage, scene description text is first obtained from the scene text knowledge base and input into the base model (a general text-to-image generation model). The base model outputs a synthetic image dataset (supplementing scene images). Then, the real multi-source dataset (real scene images) and the synthetic image dataset are used as the image dataset (initial scene images). By inputting the image dataset into the image annotation model (a large language model and object detection model), the global description text and bounding box localization annotations (sample layout instructions) of the image dataset are obtained. Then, the image dataset and its global description text can be used as the fine-tuning dataset for the style expert LORA, and the image dataset and its sample layout instructions can be used as the fine-tuning dataset for the layout expert LORA.

[0109] In the second phase, the concurrent fine-tuning training of the layout expert and style expert, the style expert LoRA is injected into the cross-attention layer of UNet. Specifically, training can be performed using the entire triplet dataset produced in the first phase, focusing on learning the mapping between global textual descriptions and mine-specific visual styles, such as the texture of exposed rock, the color atmosphere under evening light, and the pervasive feeling of industrial dust. Additionally, the layout expert LoRA is injected into PositionNet. It also uses triplet data, but its training task is to learn the mapping between a list of object-level bounding boxes and the generation of specific objects at specified locations in the image, such as a specific model of mining truck or excavator.

[0110] It should be noted that during training, the general network parameters of the general text-to-image generation model are frozen. In each iteration, based on the input triplet data, the gradients of the style expert and layout expert are simultaneously calculated and backpropagated, and their parameters are updated synchronously. Therefore, after training, to generate a static image of the target scene, only a text prompt "Dusty mine at dusk" is needed, along with a layout instruction that precisely defines the position and size of the mining truck and excavator, for example, [{"label": "Mining Truck", "box":}, {"label": "Excavator", "box":}]. The model will output a high-fidelity static image. That is, the image as a whole presents the style of dusk and dust, while a mining truck and an excavator are precisely drawn in the positions specified by the instruction.

[0111] In the third stage, the fine-tuning training stage for motion experts, the goal is to expand the static images generated in the previous step into physically realistic short videos. First, the weights of the style expert and layout expert LoRA trained in the second stage are merged (through matrix product BA) with the original weights of the base UNet model to form a new, solidified expert model for generating static scenes in the mine.

[0112] Then, based on this expert model, a temporal layer module of a pre-trained video generation model (such as Hotshot-XL) is introduced. This module typically includes temporal convolutional and temporal attention layers, which are interleaved and inserted into the various layers of the solidified spatial UNet network to construct a complete spatiotemporal UNet network architecture. Here, the training data for the third stage can use short video clips of real mining vehicle motion prepared in the first stage, such as videos of mining trucks braking or turning. During training, the parameters of all layers responsible for processing spatial information—that is, the solidified static scene expert model part of the mine—are completely frozen and do not participate in gradient updates. Furthermore, a new dynamic expert, LoRA, is defined and injected only into the newly introduced temporal layer module. The optimization objective is limited to the parameters of this dynamic expert.

[0113] After the third stage of training, the static image of the mining truck encountering the excavator, generated in the second stage, is used as the initial frame, along with a motion description, such as the mining truck moving towards the excavator and braking suddenly. A spatiotemporal model equipped with the dynamics expert LoRA is invoked. Because the spatial layer is fixed, each frame of the video maintains extremely high visual realism and accurate spatial layout. Simultaneously, due to targeted fine-tuning of the temporal layer, the movement of the mining truck, especially its braking posture and dust effects under heavy load, will conform to realistic physical dynamics. Thus, through decoupling and concurrent optimization, the generated scene achieves industry-leading levels in visual realism, layout accuracy, and dynamic rationality, directly solving the problems of poor realism in physical simulation methods and the uncontrollability and unrealistic nature of general-purpose generation models.

[0114] The method provided in this embodiment first involves data preparation and structured definition of long-tail scenes. Then, concurrent decoupling fine-tuning of static scene generation style and layout is performed. Further, the injection of physical realism into dynamic scene generation is performed. Finally, the trained scene data generation model achieves large-scale continuous frame generation of long-tail scenes, successfully decomposing a complex dynamic scene generation task into three controllable sub-tasks: style, layout, and motion. Utilizing a parameter-efficient, expert-level LoRA module, high-quality, high-fidelity, and physically realistic mine autonomous driving test data, which is difficult to obtain using traditional methods, is generated at a lower cost.

[0115] Based on any of the above embodiments Figure 5 This is a schematic diagram of the scene data generation device provided by the present invention, as shown below. Figure 5 As shown, the device includes: The acquisition unit 510 acquires global description text, layout instructions, motion description, and scene data generation model; the scene data generation model is constructed by embedding a cross-attention module, a location network module, and a temporal convolutional attention module into a general text-to-image generation model. The static scene generation unit 520 inputs the global description text and the layout instructions into the scene data generation model, applies the global description text based on the cross attention module, and applies the layout instructions based on the location network module to jointly control the generation of scene style and layout; The dynamic scene generation unit 530 generates scene video data based on the motion description applied by the temporal convolutional attention module.

[0116] The apparatus provided in this invention forms a scene data generation model by embedding a cross-attention module, a location network module, and a temporal convolutional attention module into a general text-to-image generation model. In application, the cross-attention module first responds to global descriptive text to control the visual style, and the location network module responds to layout instructions to control the spatial structure, jointly controlling the generation of scene style and layout. Subsequently, the temporal convolutional attention module independently controls the evolution of the temporal dimension, ultimately generating scene video data. This successfully decomposes the complex dynamic scene generation task into three independent yet collaborative sub-tasks: visual style rendering, spatial layout planning, and temporal dynamic evolution. This achieves the generation of complex scenes such as mines, which combine high visual fidelity, precise spatial control, and realistic physical motion.

[0117] Based on any of the above embodiments, the device further includes a training unit, which is specifically used for: Acquire the global description text of the sample, sample layout instructions, noise data, as well as the initial cross-attention module and the initial position network module; The global description text of the sample, the sample layout instructions, and the noise data are input into a general text-to-image generation model that embeds the initial cross-attention module and the initial position network module. Based on the initial cross-attention module, the global description text of the sample is applied, and based on the initial position network module, the sample layout instructions are applied to gradually reduce the noise data to obtain the first sample prediction output at each time step. The static prediction loss is calculated based on the first sample prediction output and the first sample label at any time step. Based on the static prediction loss, the network parameters of the initial cross-attention module and the initial location network module are iterated to obtain the cross-attention module and the location network module.

[0118] Based on any of the above embodiments, the training unit is further specifically used for: The general network parameters of the general text-to-image generation model are frozen; Based on the static prediction loss, the update amount of the first parameter is calculated; the update amount of the first parameter is represented based on a low-rank decomposition matrix. Based on the parameter update amount, the network parameters of the initial cross-attention module and the initial location network module are iterated simultaneously to obtain the cross-attention module and the location network module.

[0119] Based on any of the above embodiments, the training unit is further specifically used for: Acquire sample scene video data, sample generation conditions, and an initial temporal convolutional attention module; the sample generation conditions include global sample description text, sample layout instructions, and sample motion description. The initial temporal convolutional attention module, the cross-attention module, and the location network module are embedded into the general text-to-image generation model to obtain the initial scene data generation model; The noisy scene video data corresponding to the sample scene video data and the sample generation conditions are input into the initial scene data generation model to obtain the second sample prediction output at each time step. The dynamic prediction loss is calculated based on the second sample prediction output and the second sample label at any time step. Based on the dynamic prediction loss, the network parameters of the initial temporal convolutional attention module are iterated to obtain the temporal convolutional attention module.

[0120] Based on any of the above embodiments, the training unit is further specifically used for: The model parameters of the general text-to-image generation model embedded with the cross-attention module and the location network module are frozen; Based on the dynamic prediction loss, the second parameter update amount is calculated, and the second parameter update amount is represented based on a low-rank decomposition matrix. Based on the second parameter update amount, the network parameters of the initial temporal convolutional attention module are iterated to obtain the temporal convolutional attention module.

[0121] Based on any of the above embodiments, the device further includes a dataset construction unit, which is specifically used for: Obtain the initial scene image; The initial scene image is input into a large language model to obtain sample global description text of the initial scene image; The initial scene image is input into the target detection model to obtain the bounding box coordinates of the target detection results in the initial scene image, and the bounding box coordinates are used to generate the sample layout instruction.

[0122] Based on any of the above embodiments, the dataset construction unit is further specifically used for: Obtain scene description text and real scene images; Based on the scene description text, a scene generation prompt text is constructed. The scene description text is then input into the general text-to-image generation model to obtain the supplementary scene image corresponding to the scene description text. The real scene image and the supplementary scene image are used as the initial scene image.

[0123] Figure 6 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 6 As shown, the electronic device may include a processor 610, a communication interface 620, a memory 630, and a communication bus 640, wherein the processor 610, the communication interface 620, and the memory 630 communicate with each other via the communication bus 640. The processor 610 can call logical instructions in the memory 630 to execute a scene data generation method. This method includes: acquiring global description text, layout instructions, motion description, and a scene data generation model; the scene data generation model is constructed based on embedding a cross-attention module, a location network module, and a temporal convolutional attention module into a general text-to-image generation model; inputting the global description text and the layout instructions into the scene data generation model; applying the global description text based on the cross-attention module and applying the layout instructions based on the location network module to jointly control the generation of scene style and layout; and applying the motion description based on the temporal convolutional attention module to generate scene video data.

[0124] Furthermore, the logical instructions in the aforementioned memory 630 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0125] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the scene data generation method provided by the above methods. The method includes: acquiring global description text, layout instructions, motion description, and a scene data generation model; the scene data generation model is constructed based on embedding a cross-attention module, a location network module, and a temporal convolutional attention module into a general text-to-image generation model; inputting the global description text and the layout instructions into the scene data generation model; applying the global description text based on the cross-attention module and applying the layout instructions based on the location network module to jointly control the generation of scene style and layout; and applying the motion description based on the temporal convolutional attention module to generate scene video data.

[0126] In another aspect, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon. When executed by a processor, the computer program implements a method for generating scene data provided by the methods described above. The method includes: acquiring global descriptive text, layout instructions, motion description, and a scene data generation model; the scene data generation model is constructed based on embedding a cross-attention module, a location network module, and a temporal convolutional attention module into a general text-to-image generation model; inputting the global descriptive text and the layout instructions into the scene data generation model; applying the global descriptive text based on the cross-attention module and applying the layout instructions based on the location network module to jointly control the generation of scene style and layout; and applying the motion description based on the temporal convolutional attention module to generate scene video data.

[0127] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0128] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0129] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for generating scene data, characterized in that, include: The system acquires global description text, layout instructions, motion description, and scene data generation model. The scene data generation model is constructed by embedding a cross-attention module, a location network module, and a temporal convolutional attention module into a general text-to-image generation model. The global description text and the layout instructions are input into the scene data generation model. The global description text is applied based on the cross attention module, and the layout instructions are applied based on the location network module to jointly control the generation of scene style and layout. The motion description is applied based on the temporal convolutional attention module to generate scene video data.

2. The method for generating scene data according to claim 1, characterized in that, The training steps for the cross-attention module and the location network module include: Acquire the global description text of the sample, sample layout instructions, noise data, as well as the initial cross-attention module and the initial position network module; The global description text of the sample, the sample layout instructions, and the noise data are input into a general text-to-image generation model that embeds the initial cross-attention module and the initial position network module. Based on the initial cross-attention module, the global description text of the sample is applied, and based on the initial position network module, the sample layout instructions are applied to gradually reduce the noise data to obtain the first sample prediction output at each time step. The static prediction loss is calculated based on the first sample prediction output and the first sample label at any time step. Based on the static prediction loss, the network parameters of the initial cross-attention module and the initial location network module are iterated to obtain the cross-attention module and the location network module.

3. The method for generating scene data according to claim 2, characterized in that, The step of iterating on the network parameters of the initial cross-attention module and the initial location network module based on the static prediction loss to obtain the cross-attention module and the location network module includes: The general network parameters of the general text-to-image generation model are frozen; Based on the static prediction loss, the update amount of the first parameter is calculated; the update amount of the first parameter is represented based on a low-rank decomposition matrix. Based on the parameter update amount, the network parameters of the initial cross-attention module and the initial location network module are iterated simultaneously to obtain the cross-attention module and the location network module.

4. The method for generating scene data according to any one of claims 1 to 3, characterized in that, The training steps for the temporal convolutional attention module include: Acquire sample scene video data, sample generation conditions, and an initial temporal convolutional attention module; the sample generation conditions include global sample description text, sample layout instructions, and sample motion description. The initial temporal convolutional attention module, the cross-attention module, and the location network module are embedded into the general text-to-image generation model to obtain the initial scene data generation model; The noisy scene video data corresponding to the sample scene video data and the sample generation conditions are input into the initial scene data generation model to obtain the second sample prediction output at each time step. The dynamic prediction loss is calculated based on the second sample prediction output and the second sample label at any time step. Based on the dynamic prediction loss, the network parameters of the initial temporal convolutional attention module are iterated to obtain the temporal convolutional attention module.

5. The method for generating scene data according to claim 4, characterized in that, The process of iterating the network parameters of the initial temporal convolutional attention module based on the dynamic prediction loss to obtain the temporal convolutional attention module includes: The model parameters of the general text-to-image generation model embedded with the cross-attention module and the location network module are frozen; Based on the dynamic prediction loss, the second parameter update amount is calculated, and the second parameter update amount is represented based on a low-rank decomposition matrix. Based on the second parameter update amount, the network parameters of the initial temporal convolutional attention module are iterated to obtain the temporal convolutional attention module.

6. The method for generating scene data according to any one of claims 2 to 3, characterized in that, The steps for obtaining the global description text of the sample and the sample layout instructions include: Obtain the initial scene image; The initial scene image is input into a large language model to obtain sample global description text of the initial scene image; The initial scene image is input into the target detection model to obtain the bounding box coordinates of the target detection results in the initial scene image, and the bounding box coordinates are used to generate the sample layout instruction.

7. The method for generating scene data according to claim 6, characterized in that, The process of obtaining the initial scene image includes: Obtain scene description text and real scene images; Based on the scene description text, a scene generation prompt text is constructed. The scene description text is then input into the general text-to-image generation model to obtain the supplementary scene image corresponding to the scene description text. The real scene image and the supplementary scene image are used as the initial scene image.

8. A device for generating scene data, characterized in that, include: The acquisition unit acquires global description text, layout instructions, motion description, and scene data generation model; the scene data generation model is constructed by embedding a cross-attention module, a location network module, and a temporal convolutional attention module into a general text-to-image generation model. The static scene generation unit inputs the global description text and the layout instructions into the scene data generation model, applies the global description text based on the cross attention module, and applies the layout instructions based on the location network module to jointly control the generation of scene style and layout; The dynamic scene generation unit generates scene video data by applying the motion description based on the temporal convolutional attention module.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the method for generating scene data as described in any one of claims 1 to 7.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method for generating scene data as described in any one of claims 1 to 7.

Images