Scene generation, model training method and device, electronic equipment and storage medium

Abstract

Embodiments of the present application provide a scene generation method, a model training method, a device, an electronic device and a storage medium. The scene generation method comprises: obtaining first condition information corresponding to a to-be-processed scene and a first noise sequence, the first condition information comprising at least one of first road sketch information, first scene description information and first object prior information, and the first noise sequence being obtained by randomly sampling noise of a Gaussian distribution; and using a space-time diffusion model to perform feature extraction in a spatial dimension and a time dimension according to the first condition information and the first noise sequence, to obtain a first point cloud scene image sequence corresponding to the to-be-processed scene. In the embodiments of the present application, the noise sequence does not need to be obtained through a complex manner, so the complexity of this manner is low, and on the basis of the first noise sequence, the first condition information is further added, so that the description of the to-be-processed scene is more detailed, and the accuracy of the point cloud scene can be further improved.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a method, apparatus, electronic device, and storage medium for scene generation and model training. Background Technology

[0002] Realistic LiDAR (Light Detection and Ranging) point cloud scene generation is a core enabling technology for autonomous driving development. The generated LiDAR point cloud scene can realistically simulate diverse road scenarios in the real world. By providing efficient, safe, and comprehensive data support, it significantly reduces R&D costs and risks, while improving the reliability and safety of the system.

[0003] Current methods for generating LiDAR point cloud scenes typically employ generative adversarial networks (GANs). A low-quality point cloud scene is input into the GAN, which then processes it and outputs a high-fidelity point cloud scene. However, this approach requires collecting a large amount of low-quality point cloud scenes, resulting in high complexity, and analyzing only low-quality point cloud scenes leads to low accuracy. Summary of the Invention

[0004] In view of the above problems, this application proposes a scene generation and model training method, apparatus, electronic device and storage medium to solve the problems of high complexity and low accuracy of current point cloud scene generation methods.

[0005] According to one aspect of an embodiment of this application, a scene generation method is provided, the method comprising:

[0006] Obtain first condition information and first noise sequence corresponding to the scene to be processed. The first condition information includes at least one of first road sketch information, first scene description information and first object prior information. The first noise sequence is obtained by randomly sampling Gaussian distributed noise.

[0007] Using a spatiotemporal diffusion model, features are extracted in both spatial and temporal dimensions based on the first conditional information and the first noise sequence to obtain the first point cloud scene image sequence corresponding to the scene to be processed.

[0008] Optionally, the step of using a spatiotemporal diffusion model to extract features in the spatial and temporal dimensions based on the first conditional information and the first noise sequence to obtain the first point cloud scene image sequence corresponding to the scene to be processed includes:

[0009] Convolutional feature extraction is performed in both spatial and temporal dimensions based on the first conditional information and the first noise sequence to obtain the first feature image sequence.

[0010] Attention features are extracted from the first feature image sequence in both spatial and temporal dimensions to obtain the second feature image sequence.

[0011] Convolutional feature extraction is performed on the second feature image sequence in both spatial and temporal dimensions to obtain the first point cloud scene image sequence.

[0012] Optionally, the spatiotemporal diffusion model includes an equidistant columnar spatiotemporal convolution module, which comprises a spatial convolution module, a temporal convolution module, and a fusion module; the step of extracting convolutional features in the spatial and temporal dimensions based on the first conditional information and the first noise sequence to obtain a first feature image sequence includes:

[0013] Using the spatial convolution module, spatial features are extracted in the spatial dimension based on the first conditional information and the first noise sequence to obtain a first spatial feature image sequence;

[0014] Using the temporal convolution module, temporal features are extracted in the temporal dimension based on the first spatial feature image sequence to obtain the first temporal feature image sequence;

[0015] The fusion module is used to fuse the first spatial feature image sequence and the first temporal feature image sequence to obtain the first feature image sequence.

[0016] Optionally, the equidistant columnar spatiotemporal convolution module further includes a feature addition module;

[0017] Before using the spatial convolution module to extract spatial features in the spatial dimension based on the first conditional information and the first noise sequence to obtain the first spatial feature image sequence, the method further includes: using the feature addition module to add Fourier features to the first noise sequence to obtain the target noise sequence.

[0018] The step of using the spatial convolution module to extract spatial features in the spatial dimension based on the first conditional information and the first noise sequence to obtain a first spatial feature image sequence includes: using the spatial convolution module to extract spatial features in the spatial dimension based on the first conditional information and the target noise sequence to obtain a first spatial feature image sequence.

[0019] Optionally, the spatiotemporal diffusion model includes an equidistant cylindrical spatiotemporal transformer, which includes a spatial attention module and a temporal attention module; the step of extracting attention features based on the first feature image sequence in both spatial and temporal dimensions to obtain a second feature image sequence includes:

[0020] Using the spatial attention module, feature extraction is performed based on the first feature image sequence to obtain the first spatial attention feature image sequence;

[0021] Using the temporal attention module, feature extraction is performed based on the first spatial attention feature image sequence to obtain the second feature image sequence.

[0022] Optionally, the prior information of the first object is obtained in the following way:

[0023] Obtain the first object feature information corresponding to the scene to be processed;

[0024] Using a point cloud generation model, prior information of the first object is generated based on the feature information of the first object.

[0025] According to another aspect of the embodiments of this application, a model training method is provided, the method comprising:

[0026] Acquire the second condition information, the second point cloud scene image sequence, and the actual noise sequence corresponding to the sample scene. The second condition information includes at least one of the second road sketch information, the second scene description information, and the second object prior information.

[0027] Using the spatiotemporal diffusion model to be trained, features are extracted in the spatial and temporal dimensions based on the second conditional information and the second point cloud scene image sequence to obtain the prediction noise sequence corresponding to the sample scene;

[0028] The loss function is calculated based on the predicted noise sequence and the actual noise sequence. After the loss function satisfies the target condition, the spatiotemporal diffusion model as described above is obtained after training.

[0029] According to another aspect of the embodiments of this application, a scene generation apparatus is provided, the apparatus comprising:

[0030] The first acquisition module is used to acquire first condition information and a first noise sequence corresponding to the scene to be processed. The first condition information includes at least one of first road sketch information, first scene description information and first object prior information. The first noise sequence is obtained by randomly sampling Gaussian distributed noise.

[0031] The generation module is used to extract features in the spatial and temporal dimensions based on the first condition information and the first noise sequence using a spatiotemporal diffusion model, so as to obtain the first point cloud scene image sequence corresponding to the scene to be processed.

[0032] Optionally, the generation module includes:

[0033] The first generation unit is used to perform convolutional feature extraction in the spatial and temporal dimensions based on the first conditional information and the first noise sequence to obtain a first feature image sequence.

[0034] The second generation unit is used to perform attention feature extraction based on the first feature image sequence in the spatial and temporal dimensions to obtain the second feature image sequence.

[0035] The third generation unit is used to perform convolutional feature extraction based on the second feature image sequence in both spatial and temporal dimensions to obtain the first point cloud scene image sequence.

[0036] Optionally, the spatiotemporal diffusion model includes an equidistant columnar spatiotemporal convolution module, which comprises a spatial convolution module, a temporal convolution module, and a fusion module; the first generation unit is specifically used for:

[0037] The first extraction subunit is used to extract spatial features in the spatial dimension based on the first condition information and the first noise sequence using the spatial convolution module, so as to obtain a first spatial feature image sequence.

[0038] The second extraction subunit is used to extract time features in the time dimension based on the first spatial feature image sequence using the time convolution module to obtain the first time feature image sequence.

[0039] The fusion subunit is used to fuse the first spatial feature image sequence and the first temporal feature image sequence using the fusion module to obtain the first feature image sequence.

[0040] Optionally, the equidistant columnar spatiotemporal convolution module further includes a feature addition module; the first generation unit further includes: an addition subunit, used to add Fourier features to the first noise sequence using the feature addition module to obtain a target noise sequence; the first extraction subunit is specifically used to use the spatial convolution module to extract spatial features in the spatial dimension based on the first conditional information and the target noise sequence to obtain a first spatial feature image sequence.

[0041] Optionally, the spatiotemporal diffusion model includes an equidistant cylindrical spatiotemporal transformer, which includes a spatial attention module and a temporal attention module; the second generation unit is specifically used for:

[0042] Using the spatial attention module, feature extraction is performed based on the first feature image sequence to obtain the first spatial attention feature image sequence;

[0043] Using the temporal attention module, feature extraction is performed based on the first spatial attention feature image sequence to obtain the second feature image sequence.

[0044] Optionally, the first object prior information is obtained by: acquiring the first object feature information corresponding to the scene to be processed; and generating the first object prior information based on the first object feature information using a point cloud generation model.

[0045] According to another aspect of the embodiments of this application, a model training apparatus is provided, the apparatus comprising:

[0046] The second acquisition module is used to acquire second condition information, second point cloud scene image sequence and actual noise sequence corresponding to the sample scene. The second condition information includes at least one of second road sketch information, second scene description information and second object prior information.

[0047] The training module is used to extract features in the spatial and temporal dimensions based on the second conditional information and the second point cloud scene image sequence using the spatiotemporal diffusion model to be trained, to obtain the predicted noise sequence corresponding to the sample scene; calculate the loss function based on the predicted noise sequence and the actual noise sequence, and obtain the trained spatiotemporal diffusion model as described above after the loss function satisfies the target condition.

[0048] According to another aspect of the embodiments of this application, an electronic device is provided, the electronic device including a processor and a computer-readable storage medium storing a computer program; when the computer program is executed by the processor, the processor causes the processor to perform the scene generation method as described in any of the preceding claims, or to perform the model training method as described in any of the preceding claims.

[0049] According to another aspect of the embodiments of this application, a computer-readable storage medium is provided, on which a computer program is stored, which, when executed by a processor, causes the processor to perform the scene generation method as described in any of the preceding claims, or to perform the model training method as described in any of the preceding claims.

[0050] In this embodiment, first condition information and a first noise sequence corresponding to the scene to be processed are obtained. The first condition information includes at least one of first road sketch information, first scene description information, and first object prior information. Using a spatiotemporal diffusion model, feature extraction is performed in the spatial and temporal dimensions based on the first condition information and the first noise sequence to obtain a first point cloud scene image sequence corresponding to the scene to be processed. Therefore, in this embodiment, on the one hand, the spatiotemporal diffusion model is used to analyze the first noise sequence. Compared to collecting low-quality point cloud scenes, the first noise sequence is obtained by randomly sampling Gaussian-distributed noise, eliminating the need for complex acquisition methods, thus reducing complexity. On the other hand, first condition information is further added to the first noise sequence. The first condition information can provide conditional descriptions of the scene to be processed, making the description of the scene more detailed and further improving the accuracy of the point cloud scene.

[0051] The above description is only an overview of the technical solution of this application. In order to better understand the technical means of this application and to implement it in accordance with the contents of the specification, and to make the above and other objects, features and advantages of this application more obvious and understandable, the following are specific embodiments of this application. Attached Figure Description

[0052] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the description of the embodiments of this application will be briefly introduced below. Obviously, the drawings described below are only some drawings of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0053] Figure 1 This is a flowchart of a scene generation method according to an embodiment of this application;

[0054] Figure 2 This is a flowchart of a model training method according to an embodiment of this application;

[0055] Figure 3 This is a schematic diagram of the structure of a spatiotemporal diffusion model according to an embodiment of this application;

[0056] Figure 4 This is a schematic diagram of another spatiotemporal diffusion model according to an embodiment of this application;

[0057] Figure 5 This is a structural block diagram of a scene generation device according to an embodiment of this application;

[0058] Figure 6 This is a structural block diagram of a model training device according to an embodiment of this application;

[0059] Figure 7 This is a structural block diagram of an electronic device according to an embodiment of this application;

[0060] Figure 8 This is a structural block diagram of a computer-readable storage medium according to an embodiment of this application. Detailed Implementation

[0061] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0062] In this application embodiment, a spatiotemporal diffusion model (which may be called LiDARDrive4D) is proposed, which can realize forward diffusion and backward diffusion processes. In the forward diffusion process, the spatiotemporal diffusion model can predict the noise sequence based on the point cloud scene image sequence and conditional information. In the backward diffusion process, the spatiotemporal diffusion model can reconstruct the point cloud scene image sequence based on the noise sequence and conditional information.

[0063] In this embodiment, during model training, the forward diffusion process of the spatiotemporal diffusion model is trained to obtain a spatiotemporal diffusion model capable of accurately predicting noise sequences based on point cloud scene image sequences and conditional information. This enables the spatiotemporal diffusion model to convert between point cloud scene image sequences, conditional information, and noise sequences. During scene generation, the backward diffusion process of the trained spatiotemporal diffusion model is used to reconstruct the point cloud scene image sequence based on the noise sequence and conditional information.

[0064] Reference Figure 1 The diagram shows a flowchart of a scene generation method according to an embodiment of this application.

[0065] like Figure 1 As shown, the scene generation method may include the following steps:

[0066] Step 101: Obtain first condition information and first noise sequence corresponding to the scene to be processed. The first condition information includes at least one of first road sketch information, first scene description information and first object prior information. The first noise sequence is obtained by randomly sampling Gaussian distributed noise.

[0067] A scene to be processed refers to a scene for which a sequence of point cloud images needs to be generated, based on actual requirements. For example, a scene to be processed can be any one or more of the following: an accident scene, a red light violation scene, or an obstacle scene, etc.

[0068] The conditional information corresponding to the scene to be processed is called the first conditional information. The first conditional information can specify the conditions for the scene to be processed. The first conditional information may include, but is not limited to, at least one of the following: first road sketch information, first scene description information, and first object prior information.

[0069] In this embodiment, the first road sketch information provides the first road layout information and first object feature information corresponding to the scene to be processed. The first road layout information is accurately depicted through road boundary lines and lane dividers, which enables pixel-level control of the road spatial structure. The first object feature information is encoded using 3D bounding boxes. Each bounding box contains the object's size (e.g., length, width, height), position (e.g., 3D coordinates), orientation information (e.g., heading angle), category, etc. This representation method has dual advantages: 1) It supports precise object-level manipulation, allowing fine-grained control over each object element in the scene; 2) By using multiple 3D bounding boxes superimposed, it can implicitly express the occlusion relationship between objects, effectively conveying depth information and spatial correlation without explicit encoding. The objects can be vehicles, pedestrians, obstacles, etc., in the scene to be processed. To integrate the first road layout information and the first object feature information into a unified representation, both the first road layout information and the first object feature information are projected onto the equidistant cylindrical image plane of the LiDAR sensor. The resulting first road sketch information has the same size as the input first noise sequence. Using first road sketch information helps to precisely control the position and size of objects, and can effectively control foreground elements such as vehicles and pedestrians.

[0070] In this embodiment, the first road sketch information can be set manually according to actual needs. For example, the first road layout information and the first object feature information corresponding to the scene to be processed can be provided manually, and then the first road layout information and the first object feature information can be projected onto the equidistant cylindrical image plane of the lidar sensor using a projection algorithm to obtain the first road sketch information.

[0071] In this embodiment, the first scene description information provides background element information such as buildings and trees corresponding to the scene to be processed, enabling fine-grained control over background elements and thus improving the realism of the generated point cloud scene image. The first scene description information can be set manually according to actual needs. For example, a scene description text corresponding to the scene to be processed can be provided manually, such as "a building exists at a certain location," and this scene description text can be used as the first scene description information.

[0072] In this embodiment, the prior information of the first object provides a point cloud image of the first object corresponding to the scene to be processed. LiDAR measurements of different elements often exhibit an imbalance in the number of point clouds; for example, a vehicle may contain only a few hundred points, while the ground may cover thousands. If the overall scene approximation is taken as the goal, without specifically addressing this imbalance, the quality of the generated object will be poor. Therefore, this application proposes synthesizing the object point cloud image as a priori condition, which can provide more accurate guidance for the spatiotemporal diffusion process, thereby guiding the spatiotemporal diffusion model to complete the overall scene generation and further improving the realism at the object level.

[0073] For example, the first object prior information is obtained by: acquiring the first object feature information corresponding to the scene to be processed; and generating the first object prior information based on the first object feature information using a point cloud generation model.

[0074] For example, the feature information of the first object corresponding to the scene to be processed is used as the input of the point cloud generation model. After the feature information of the first object is analyzed and processed by the point cloud generation model, the point cloud generation model can output the point cloud data of the first object corresponding to the scene to be processed. Then, the point cloud data of the first object is projected onto the equidistant columnar image plane of the lidar sensor, thereby generating the point cloud image of the first object, which is the prior information of the first object.

[0075] The first object feature information corresponding to the scene to be processed can be set according to actual needs. The first object feature information can be encoded using 3D bounding boxes. Each bounding box contains the object's size (such as length, width, and height), position (such as three-dimensional coordinates), orientation information (such as heading angle), category, etc.

[0076] Object categories: For example, you can set categories such as cars, trucks, trailers, buses, construction vehicles, bicycles, motorcycles, pedestrians, traffic cones, and guardrails.

[0077] Object position: The Cartesian coordinates (x, y, z) of the object are converted into polar coordinates (ρ, θ, φ) relative to the radar sensor, where ρ represents the distance, θ represents the azimuth angle, and φ represents the elevation angle, in order to better represent the angular relationship between the radar sensor and the object. The coordinates are normalized to the range [-1, 1].

[0078] The dimensions of an object: length l, width w, height h.

[0079] Object orientation: The heading angle Φ is normalized to the range [-1, 1] to represent the alignment direction of the object.

[0080] The point cloud generation model can be a pre-trained model. For example, sample object feature information and corresponding actual object point cloud data can be obtained. The sample object feature information is used as input to the point cloud generation model to be trained, resulting in predicted object point cloud data output by the model. A loss function for the model is calculated based on the actual and predicted object point cloud data. If the loss function does not meet a preset condition, the parameters of the model are adjusted and training continues. Once the loss function meets the preset condition, training is considered complete, resulting in a trained point cloud generation model.

[0081] For example, the point cloud generation model can use any suitable model structure such as DiT-3D (Diffusion Transformer for 3D). DiT-3D can employ a continuous-time diffusion strategy, thereby generating high-quality object point cloud data with fewer steps, significantly reducing computational overhead. The loss function of the point cloud generation model can be any suitable loss function such as cross-entropy loss function or logarithmic loss function.

[0082] In this embodiment, considering that the continuity of LiDAR data is crucial for capturing the dynamic behavior of road users, which greatly helps autonomous driving systems understand their surroundings, incorporating the time dimension is essential. Therefore, a first noise sequence corresponding to the scene to be processed is obtained, resulting in a series of temporally continuous noises. This enables the spatiotemporal diffusion model to generate a series of point cloud scene images that maintain temporal consistency, i.e., a point cloud scene image sequence, thus ensuring temporal continuity.

[0083] For example, the first noise sequence can be a noise sequence randomly sampled from Gaussian distributed noise, which is easier to obtain.

[0084] Step 102: Using a spatiotemporal diffusion model, feature extraction is performed in the spatial and temporal dimensions based on the first conditional information and the first noise sequence to obtain the first point cloud scene image sequence corresponding to the scene to be processed.

[0085] The first condition information and the first noise sequence are used as inputs to the spatiotemporal diffusion model. The spatiotemporal diffusion model is used to extract features in the spatial and temporal dimensions based on the first condition information and the first noise sequence, thereby obtaining the first point cloud scene image sequence corresponding to the scene to be processed, thus ensuring the spatiotemporal consistency of the first point cloud scene image sequence.

[0086] For example, the process of using a spatiotemporal diffusion model to extract features in the spatial and temporal dimensions based on the first conditional information and the first noise sequence to obtain a first point cloud scene image sequence may include: performing convolutional feature extraction in the spatial and temporal dimensions based on the first conditional information and the first noise sequence to obtain a first feature image sequence; performing attention feature extraction in the spatial and temporal dimensions based on the first feature image sequence to obtain a second feature image sequence; and performing convolutional feature extraction in the spatial and temporal dimensions based on the second feature image sequence to obtain the first point cloud scene image sequence.

[0087] For example, the spatiotemporal diffusion model includes an equirectangular spatial-temporal convolution module (EST-Conv) and an equirectangular spatial-temporal transformer (EST-Trans).

[0088] The point cloud scene image in this embodiment is an equidistant columnar image. The equidistant columnar image differs from RGB (Red, Green) and Blue (red, green, blue) images in the following ways: (1) The pixel coordinates (θ, φ) of the equidistant columnar image are strongly correlated with the data pattern of the lidar sensor (e.g., the adjacent and lower-positioned area usually represents the ground); (2) The left and right boundaries of the equidistant columnar image represent continuous areas in the measurement space of the lidar sensor (especially lidar sensors with a 360-degree horizontal field of view); (3) The pixel value (distance, reflectivity, etc.) distribution of the equidistant columnar image differs significantly from that of the RGB image. Therefore, this embodiment proposes an equidistant columnar spatiotemporal convolution module for learning the spatiotemporal features of the equidistant columnar image.

[0089] In this embodiment, the equidistant columnar spatiotemporal convolution module can efficiently process features with progressively increasing receptive fields, but it has limitations in capturing long-range correlations. To further enhance spatiotemporal consistency, an equidistant columnar spatiotemporal transformer is introduced to capture relationships within a single frame of equidistant columnar images and promote feature interactions between serialized equidistant columnar images.

[0090] For example, the equidistant columnar spatiotemporal convolution module includes a spatial convolution module, a temporal convolution module, and a fusion module. The process of extracting convolutional features in the spatial and temporal dimensions based on the first conditional information and the first noise sequence to obtain a first feature image sequence may include: using the spatial convolution module to extract spatial features in the spatial dimension based on the first conditional information and the first noise sequence to obtain a first spatial feature image sequence; using the temporal convolution module to extract temporal features in the temporal dimension based on the first spatial feature image sequence to obtain a first temporal feature image sequence; and using the fusion module to fuse the first spatial feature image sequence and the first temporal feature image sequence to obtain the first feature image sequence.

[0091] For example, the fusion module can be an Alpha mixer, and the processing procedure of the Alpha mixer can be represented as follows:

[0092] y = α·x S +(1-α)·x T

[0093] Where α represents the learnable parameter in the Alpha mixer, x s Let x represent the first spatial feature image sequence. T This represents the first time-series feature image sequence.

[0094] For example, the equidistant columnar spatiotemporal convolution module further includes a feature addition module. Before using the spatial convolution module to extract spatial features in the spatial dimension based on the first conditional information and the first noise sequence to obtain a first spatial feature image sequence, the module further includes: using the feature addition module to add Fourier features to the first noise sequence to obtain a target noise sequence. The step of using the spatial convolution module to extract spatial features in the spatial dimension based on the first conditional information and the first noise sequence to obtain a first spatial feature image sequence includes: using the spatial convolution module to extract spatial features in the spatial dimension based on the first conditional information and the target noise sequence to obtain a first spatial feature image sequence. In this embodiment, to enhance spatial consistency, Fourier features are extracted and concatenated with the input, utilizing the correlation between pixel coordinates and LiDAR sensor measurements. The integration of Fourier features helps the model better capture the underlying geometric patterns in the data.

[0095] For example, the equidistant columnar spatiotemporal transformer includes a spatial attention module and a temporal attention module. The process of extracting attention features based on the first feature image sequence in both spatial and temporal dimensions to obtain a second feature image sequence may include: using the spatial attention module to extract features from the first feature image sequence to obtain a first spatial attention feature image sequence; and using the temporal attention module to extract features from the first spatial attention feature image sequence to obtain the second feature image sequence. In this embodiment, by introducing an attention mechanism to effectively model long-range dependencies, in the equidistant columnar spatiotemporal transformer, after the input sequence is expanded along the batch dimension, spatial attention is applied first, followed by temporal attention. Spatial attention captures the relationships within a single frame of equidistant columnar images, while temporal attention promotes feature interactions between serialized equidistant columnar images. This architecture improves the spatiotemporal consistency of features while reducing computational costs. Specifically, to reduce computational costs, an equidistant columnar spatiotemporal transformer can be introduced at the bottleneck layer.

[0096] For example, the process of obtaining the first point cloud scene image sequence by performing convolutional feature extraction based on the second feature image sequence in both spatial and temporal dimensions may include: using the spatial convolution module to perform spatial feature extraction based on the second feature image sequence in the spatial dimension to obtain a second spatial feature image sequence; using the temporal convolution module to perform temporal feature extraction based on the second spatial feature image sequence in the temporal dimension to obtain a second temporal feature image sequence; and using the fusion module to fuse the second spatial feature image sequence and the second temporal feature image sequence to obtain the first point cloud scene image sequence.

[0097] In this embodiment, on the one hand, the first noise sequence is analyzed using a spatiotemporal diffusion model. Compared with the method of collecting low-quality point cloud scenes, the first noise sequence is obtained by randomly sampling Gaussian distributed noise, which does not require a complicated method to obtain, so this method has lower complexity. On the other hand, based on the first noise sequence, first condition information is further added. The first condition information can provide conditional descriptions of the scene to be processed, thereby making the description of the scene to be processed more detailed and further improving the accuracy of the point cloud scene.

[0098] Reference Figure 2 The diagram shows a flowchart of a model training method according to an embodiment of this application.

[0099] like Figure 2 As shown, the model training method may include the following steps:

[0100] Step 201: Obtain the second condition information, the second point cloud scene image sequence, and the actual noise sequence corresponding to the sample scene. The second condition information includes at least one of the second road sketch information, the second scene description information, and the second object prior information.

[0101] The sample scenario can be a real-world autonomous driving scenario. In this embodiment, the second condition information corresponding to the sample scenario and the second point cloud scene image sequence corresponding to the sample scenario can be obtained from the real-world autonomous driving scenario dataset.

[0102] The actual noise sequence can be a noise sequence sampled from Gaussian distributed noise.

[0103] The conditional information corresponding to the sample scene is called the second conditional information. The second conditional information can provide conditional constraints on the sample scene. The second conditional information may include, but is not limited to, at least one of the following: second road sketch information, second scene description information, and second object prior information.

[0104] In this embodiment, the second road sketch information provides second road layout information and second object feature information corresponding to the sample scene. The second road layout information and second object feature information are basically the same as the first road layout information and first object feature information mentioned above; please refer to the relevant descriptions above for details. The second road sketch information can be obtained by analyzing the second point cloud scene image sequence corresponding to the sample scene and the images captured by the camera corresponding to the sample scene. It belongs to pre-obtained label information, and the specific processing procedure will not be discussed in detail here.

[0105] In this embodiment, the second scene description information provides background element information such as buildings and trees corresponding to the sample scene. The second scene description information can be obtained from images captured by the camera corresponding to the sample scene.

[0106] For example, a panoramic image captured by a camera is obtained, and the panoramic image and prompt information are input into a large language model to obtain the second scene description information output by the large language model.

[0107] The large language model can adopt any applicable model structure, and this embodiment does not impose any restrictions on it.

[0108] The panoramic image can include images captured by multiple cameras, such as front left, front, front right, rear left, rear, and rear right. The front left, front, and front right images can be stitched together to form a front view stitched image, and the rear left, rear, and rear right images can be stitched together to form a rear view stitched image. The front view stitched image and the rear view stitched image are then input into the large language model.

[0109] The prompts are used to guide the large language model to infer scene description information based on the panoramic image and to provide inference rules.

[0110] For example, a possible prompt could be: "Two images are provided, captured simultaneously by different cameras on the same vehicle. The left area of ​​the first image corresponds to the camera at the front left of the vehicle, the middle area to the camera directly in front of the vehicle, and the right area to the camera at the front right of the vehicle. The left area of ​​the second image corresponds to the camera at the rear left of the vehicle, the middle area to the camera at the rear of the vehicle, and the right area to the camera at the rear right of the vehicle. Images from different cameras may overlap. Please describe the entire driving scenario from a holistic perspective in a short, detailed text, enabling anyone to recreate the original, complete scene based on your description. The description should include the weather (rainy or sunny), time (day or night), driving scenario (city or highway), and other driving precautions. Do not provide analysis or reasons. Avoid using expressions like 'because' or 'therefore.' Simply describe the scenario. Avoid offering advice to the driver. Do not use expressions like 'the driver should…'. Avoid using uncertain expressions like 'maybe' or 'probably.' Avoid using 'the first image' or 'the second image'; the description should be fluent and natural, forming a coherent overall picture of the driving scenario."

[0111] In this embodiment of the application, the prior information of the second object provides a point cloud image of the second object corresponding to the sample scene. For example, the prior information of the second object is obtained in the following manner: acquiring the feature information of the second object corresponding to the sample scene; and generating the prior information of the second object based on the feature information of the second object using a point cloud generation model.

[0112] In this embodiment, the second point cloud scene image in the second point cloud scene image sequence corresponding to the sample scene can be an equidistant columnar image. The raw measurement data provided by the lidar sensor is in the form of point cloud data, which has the characteristics of being disordered and non-uniformly distributed within the scene. To promote effective model learning, equidistant columnar images can be used because they are efficient in describing large scenes.

[0113] For example, for a lidar sensor with an H×W spatial resolution, the point cloud data it collects includes distance and reflectivity. By capturing distance and reflectivity at W azimuth angles and H elevation angles, the azimuth and elevation angles are used as pixel coordinates of an isometric histogram, and the distance and reflectivity are used as pixel values ​​of the isometric histogram to generate an isometric histogram. Considering the large difference in distance, the distance variable can be scaled and normalized to the range [-1, 1]. For example, this can be achieved through d... ′ =log(d+1) / log(d) max +1) Normalize the distance, where d represents the distance before normalization. ′ d represents the normalized distance. maxThis indicates the maximum distance set.

[0114] Step 202: Using the spatiotemporal diffusion model to be trained, feature extraction is performed in the spatial and temporal dimensions based on the second conditional information and the second point cloud scene image sequence to obtain the prediction noise sequence corresponding to the sample scene.

[0115] For example, using the spatiotemporal diffusion model to be trained, feature extraction is performed in the spatial and temporal dimensions based on the second conditional information and the second point cloud scene image sequence to obtain the prediction noise sequence corresponding to the sample scene. This includes: performing convolutional feature extraction in the spatial and temporal dimensions based on the second conditional information and the second point cloud scene image sequence to obtain a third feature image sequence; performing attention feature extraction in the spatial and temporal dimensions based on the third feature image sequence to obtain a fourth feature image sequence; and performing convolutional feature extraction in the spatial and temporal dimensions based on the fourth feature image sequence to obtain the prediction noise sequence corresponding to the sample scene.

[0116] For example, performing convolutional feature extraction in both spatial and temporal dimensions based on the second conditional information and the second point cloud scene image sequence to obtain a third feature image sequence includes: using a spatial convolution module to extract spatial features in the spatial dimension based on the second conditional information and the second point cloud scene image sequence to obtain a third spatial feature image sequence; using a temporal convolution module to extract temporal features in the temporal dimension based on the second spatial feature image sequence to obtain a third temporal feature image sequence; and using a fusion module to fuse the third spatial feature image sequence and the third temporal feature image sequence to obtain the prediction noise sequence.

[0117] For example, before using the spatial convolution module to extract spatial features in the spatial dimension based on the second conditional information and the second point cloud scene image sequence to obtain the third spatial feature image sequence, the method further includes: using a feature addition module to add Fourier features to the second point cloud scene image sequence to obtain the target point cloud scene image sequence. Extracting spatial features in the spatial dimension based on the second conditional information and the second point cloud scene image sequence to obtain the third spatial feature image sequence using the spatial convolution module includes: using the spatial convolution module to extract spatial features in the spatial dimension based on the second conditional information and the point cloud scene image sequence to obtain the third spatial feature image sequence.

[0118] Exemplarily, performing attention feature extraction on the third feature image sequence in the spatial dimension and the temporal dimension to obtain a fourth feature image sequence includes: performing feature extraction according to the third feature image sequence by using the spatial attention module to obtain a second spatial attention feature image sequence; and performing feature extraction according to the second spatial attention feature image sequence by using the temporal attention module to obtain the fourth feature image sequence.

[0119] In step 203, a loss function is calculated according to the predicted noise sequence and the actual noise sequence, and after the loss function satisfies a target condition, a trained spatio-temporal diffusion model is obtained.

[0120] In the spatio-temporal diffusion model, the forward diffusion process gradually corrupts the original sample (such as the second point cloud scene image) by adding Gaussian noise at each time step t, and the specific formula is as follows:

[0121] x t =α t x+σ t ∈

[0122] wherein, x t is the noisy sample after the time step t (such as the second point cloud scene image with noise added), x represents the original sample (such as the second point cloud scene image), ∈ is the noise sampled from the standard Gaussian distribution (that is, the actual noise), I represents the covariance matrix of the noise, and α t and σ t are hyperparameters that control the noise schedule. Exemplarily, an α cosine scheduling strategy may be adopted, where α t =cos(πt / 2), σ t =sin(πt / 2).

[0123] The noisy sample will eventually converge to a Gaussian distribution, whose expression is as follows:

[0124]

[0125] wherein, q(x t |x) represents the distribution of the noisy sample, represents the Gaussian distribution.

[0126] Therefore, the intermediate transition process from time step s to time step t (0<s<t<1) can be expressed as:

[0127]

[0128] wherein, q(x t |x s ) represents the intermediate transition process of the noisy sample from time step s to time step t, xs The noise sample at time step s, α t|s =α t / α s , σ t|s =σ t -α t|s σ s .

[0129] In the spatiotemporal diffusion model, the backdiffusion process is used to denoise samples in the latent space, and it can be expressed as:

[0130]

[0131] μ t (x,x t )=(α t|s σ s 2 / σ t 2 )·x t +(α s σ t|s 2 / σ t 2 )·x

[0132] Σ t 2 =σ t|s 2 σ s 2 / σ t 2

[0133] Where p(x) s |x t ) represents the sample obtained after denoising (e.g., the first point cloud scene image).

[0134] During training, for a noisy sample (such as a second point cloud scene image), the model learns to estimate the unknown x. s noise reduction x t Specifically, the model predicts noise ∈ by estimating x, that is, reparameterizing x as x. t Functions of ∈.

[0135] The loss function during training can be expressed as:

[0136]

[0137] Where L represents the loss function, and ∈ represents the actual noise. 'c' represents prediction noise, and 'c' represents conditional information. This represents the expectation of the joint distribution of noise samples and time steps, and its actual calculation is achieved through random sampling and Monte Carlo estimation.

[0138] After calculating the loss function based on the predicted noise sequence and the actual noise sequence, if the loss function does not meet the target condition, the parameters of the spatiotemporal diffusion model are adjusted and training continues. Once the loss function meets the target condition, the trained spatiotemporal diffusion model is obtained. The target condition can be that the loss function is less than a set value, or that the number of training iterations reaches a set number, etc.

[0139] In the implementation, the Adam optimizer can be used for training with a learning rate of 10. -4 A cosine scheduler is employed. Furthermore, an EMA (Exponential Moving Average) strategy is used with a decay rate of 0.995, updating every 10 steps. The model is trained for a total of 40,000 steps, using the Accelerate library for distributed training and automatic precision mixing. During denoising, the number of iterations can be set to 256.

[0140] The specific structure of the spatiotemporal diffusion model will be introduced below.

[0141] Reference Figure 3 The diagram shows a structural schematic of a spatiotemporal diffusion model according to an embodiment of this application.

[0142] like Figure 3 As shown, during the forward diffusion process, the point cloud scene image sequence, road sketch information, scene description information, and object prior information are used as inputs to the spatiotemporal diffusion model. In the spatiotemporal diffusion model, the input sequence is processed by stacked equidistant columnar spatiotemporal convolutional modules. First, Fourier features are added to the input sequence through a feature addition module. Then, it is processed sequentially through a spatial convolutional module and a temporal convolutional module. Finally, a fusion module (such as an alpha mixer) fuses the processing results of the spatial and temporal convolutional modules to obtain the output sequence. The input sequence is processed by an equidistant columnar spatiotemporal convolution module, and then by an equidistant columnar spatiotemporal transformer. The input sequence is then processed sequentially by a spatial attention module and a temporal attention module to obtain the output sequence. After processing by the equidistant columnar spatiotemporal transformer, the input sequence is processed again by stacked equidistant columnar spatiotemporal convolution modules. The specific processing is similar to that of the equidistant columnar spatiotemporal convolution modules described above, and will not be discussed in detail here. After processing by the equidistant columnar spatiotemporal convolution modules again, the spatiotemporal diffusion model outputs a noise sequence. A feature addition module can be set in the first equidistant columnar spatiotemporal convolution module.

[0143] like Figure 3As shown, in the back diffusion process, the noise sequence, road sketch information, scene description information, and object prior information are used as inputs to the spatiotemporal diffusion model. The processing in the spatiotemporal diffusion model is similar to that in the forward diffusion process, and can be referred to the relevant description above. This embodiment will not be discussed in detail here.

[0144] It should be noted that the road sketch information, scene description information, and object prior information mentioned above are in sequence form corresponding to the point cloud scene image sequence and noise sequence.

[0145] Reference Figure 4 The diagram shows a structural schematic of another spatiotemporal diffusion model according to an embodiment of this application.

[0146] like Figure 4 As shown, scene captions and timestep information are processed and integrated through a conditional fusion module. Text embeddings and timestep embeddings are fused and then injected into the equidistant columnar spatiotemporal convolution module (EST-Conv). Object priors are processed by input convolution (InConv) and then injected into the equidistant columnar spatiotemporal convolution module (EST-Conv). Road sketches and range images are concatenated along the channel dimension to provide structured guidance, and then processed by input convolution (InConv) before being injected into the equidistant columnar spatiotemporal convolution module (EST-Conv). After processing by the equidistant columnar spatiotemporal convolution module (EST-Conv) and the equidistant columnar spatiotemporal transformer (EST-Trans), a noisy sequence is output.

[0147] like Figure 4 As shown, the Condition Fusion module includes modules such as Learned SinusoidalPosEmb, Multilayer Perceptron (MLP), Encoder (T5Encoder), PerceiverAttention, and MeanPooling. Time-step information is input to the Learned SinusoidalPosEmb, scene description information is input to the Encoder, the output of the MLP and the output of MeanPooling are concatenated to obtain the text embedding, and the output of the MLP and the output of MeanPooling are added to obtain the time-step embedding.

[0148] like Figure 4As shown, the equidistant columnar spatiotemporal convolution module (EST-Conv) includes modules such as GroupNorm, SiLU activation function, 2D convolution (Conv2D), CrossAttention, Adaptive Normalization (AdaNorm), SiLU activation function + Dropout, 3D convolution (Conv3D), and AlphaBlender. For the specific structure, please refer to [reference needed]. Figure 4 That's it. The learnable α value in the Alpha mixer can be set to α = 0.5, etc.

[0149] like Figure 4 As shown, the equidistant cylindrical spatiotemporal transformer (EST-Trans) may include modules such as layer normalization, modulation, spatial attention, multilayer perceptron (MLP), and temporal attention. For the specific structure, please refer to [reference needed]. Figure 4 That's all.

[0150] In this embodiment, a novel scene generation framework is proposed, integrating multimodal conditional information and an isometric columnar spatiotemporal diffusion model (4D LiDAR). The multimodal conditional information enables precise scene manipulation of foreground objects, background elements, and object-level fidelity, while the spatiotemporal diffusion model ensures the spatiotemporal consistency of the generated serialized point cloud scene.

[0151] Reference Figure 5 The diagram shows a structural block diagram of a scene generation apparatus according to an embodiment of this application.

[0152] like Figure 5 As shown, the scene generation device may include the following modules:

[0153] The first acquisition module 501 is used to acquire first condition information and a first noise sequence corresponding to the scene to be processed. The first condition information includes at least one of first road sketch information, first scene description information and first object prior information. The first noise sequence is obtained by randomly sampling Gaussian distributed noise.

[0154] The generation module 502 is used to extract features in the spatial and temporal dimensions based on the first condition information and the first noise sequence using a spatiotemporal diffusion model, so as to obtain the first point cloud scene image sequence corresponding to the scene to be processed.

[0155] Optionally, the generation module 502 includes:

[0156] The first generation unit is used to perform convolutional feature extraction in the spatial and temporal dimensions based on the first conditional information and the first noise sequence to obtain a first feature image sequence.

[0157] The second generation unit is used to perform attention feature extraction based on the first feature image sequence in the spatial and temporal dimensions to obtain the second feature image sequence.

[0158] The third generation unit is used to perform convolutional feature extraction based on the second feature image sequence in both spatial and temporal dimensions to obtain the first point cloud scene image sequence.

[0159] Optionally, the spatiotemporal diffusion model includes an equidistant columnar spatiotemporal convolution module, which comprises a spatial convolution module, a temporal convolution module, and a fusion module; the first generation unit is specifically used for:

[0160] The first extraction subunit is used to extract spatial features in the spatial dimension based on the first condition information and the first noise sequence using the spatial convolution module, so as to obtain a first spatial feature image sequence.

[0161] The second extraction subunit is used to extract time features in the time dimension based on the first spatial feature image sequence using the time convolution module to obtain the first time feature image sequence.

[0162] The fusion subunit is used to fuse the first spatial feature image sequence and the first temporal feature image sequence using the fusion module to obtain the first feature image sequence.

[0163] Optionally, the equidistant columnar spatiotemporal convolution module further includes a feature addition module; the first generation unit further includes: an addition subunit, used to add Fourier features to the first noise sequence using the feature addition module to obtain a target noise sequence; the first extraction subunit is specifically used to use the spatial convolution module to extract spatial features in the spatial dimension based on the first conditional information and the target noise sequence to obtain a first spatial feature image sequence.

[0164] Optionally, the spatiotemporal diffusion model includes an equidistant cylindrical spatiotemporal transformer, which includes a spatial attention module and a temporal attention module; the second generation unit is specifically used for:

[0165] Using the spatial attention module, feature extraction is performed based on the first feature image sequence to obtain the first spatial attention feature image sequence;

[0166] Using the temporal attention module, feature extraction is performed based on the first spatial attention feature image sequence to obtain the second feature image sequence.

[0167] Optionally, the first object prior information is obtained by: acquiring the first object feature information corresponding to the scene to be processed; and generating the first object prior information based on the first object feature information using a point cloud generation model.

[0168] Reference Figure 6 The diagram shows a structural block diagram of a model training device according to an embodiment of this application.

[0169] like Figure 6 As shown, the model training device may include the following modules:

[0170] The second acquisition module 601 is used to acquire second condition information, second point cloud scene image sequence and actual noise sequence corresponding to the sample scene. The second condition information includes at least one of second road sketch information, second scene description information and second object prior information.

[0171] The training module 602 is used to extract features in the spatial and temporal dimensions based on the second conditional information and the second point cloud scene image sequence using the spatiotemporal diffusion model to be trained, to obtain the predicted noise sequence corresponding to the sample scene; calculate the loss function based on the predicted noise sequence and the actual noise sequence, and obtain the spatiotemporal diffusion model as described in any of the above embodiments after the loss function satisfies the target condition.

[0172] In this embodiment, on the one hand, the first noise sequence is analyzed using a spatiotemporal diffusion model. Compared with the method of collecting low-quality point cloud scenes, the first noise sequence is obtained by randomly sampling Gaussian distributed noise, which does not require a complicated method to obtain, so this method has lower complexity. On the other hand, based on the first noise sequence, first condition information is further added. The first condition information can provide conditional descriptions of the scene to be processed, thereby making the description of the scene to be processed more detailed and further improving the accuracy of the point cloud scene.

[0173] As the device embodiment is basically similar to the method embodiment, the description is relatively simple, and relevant parts can be found in the description of the method embodiment.

[0174] In embodiments of this application, an electronic device is also provided. This electronic device may include a processor and a computer-readable storage medium storing a computer program; when the computer program is executed by the processor, the processor causes the processor to perform the scene generation method as described in any of the preceding embodiments, or to perform the model training method as described in any of the preceding embodiments.

[0175] Reference Figure 7 This diagram illustrates a structural block diagram of an electronic device according to an embodiment of this application. Figure 7 As shown, the electronic device 11 includes a processor 111 and a computer-readable storage medium 112, on which a computer program 1121 is stored.

[0176] The processor 111 is used to execute the computer program 1121 stored on the computer-readable storage medium 112. When executing the computer program 1121, the processor 111 executes the scene generation method as described in any of the above embodiments, or executes the model training method as described in any of the above embodiments, and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0177] The processor 111 mentioned above may include, but is not limited to: a central processing unit (CPU), a network processor (NP), a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc.

[0178] The aforementioned computer-readable storage medium 112 may include, but is not limited to: read-only memory (ROM), random access memory (RAM), compact disc read-only memory (CD-ROM), electronically erasable programmable read-only memory (EEPROM), hard disk, floppy disk, flash memory, etc.

[0179] In embodiments of this application, a computer-readable storage medium is also provided, on which a computer program is stored. The computer program can be executed by a processor of an electronic device. When the computer program is executed by the processor, the processor performs the scene generation method as described in any of the above embodiments, or performs the model training method as described in any of the above embodiments.

[0180] Reference Figure 8 This diagram illustrates a structural block diagram of a computer-readable storage medium according to an embodiment of this application. Figure 8 As shown, a computer program 211 is stored on a computer-readable storage medium 21. When the computer program 211 is executed by a processor, the processor performs the scene generation method as described in any of the above embodiments, or the model training method as described in any of the above embodiments, and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0181] The various embodiments in this specification are related to each other and are described in a progressive manner. Each embodiment focuses on the differences from other embodiments, and the same or similar parts between the embodiments can be referred to each other.

[0182] It should be noted that all actions involving the acquisition of signals, information, or data in this application are carried out in compliance with the relevant data protection laws and regulations of the locality and with authorization from the owner of the relevant device.

[0183] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal device that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or terminal device. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or terminal device that includes said element.

[0184] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application.

[0185] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of this application.

[0186] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed in this application can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0187] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0188] In the embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative. For instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0189] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0190] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0191] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. In summary, the content of this specification should not be construed as a limitation of this application.

Claims

1. A scene generation method, characterized in that, The method includes: Obtain first condition information and first noise sequence corresponding to the scene to be processed. The first condition information includes at least one of first road sketch information, first scene description information and first object prior information. The first noise sequence is obtained by randomly sampling Gaussian distributed noise. Using a spatiotemporal diffusion model, features are extracted in both spatial and temporal dimensions based on the first conditional information and the first noise sequence to obtain the first point cloud scene image sequence corresponding to the scene to be processed.

2. The method according to claim 1, characterized in that, The step of using a spatiotemporal diffusion model to extract features in both spatial and temporal dimensions based on the first conditional information and the first noise sequence to obtain a first point cloud scene image sequence corresponding to the scene to be processed includes: Convolutional feature extraction is performed in both spatial and temporal dimensions based on the first conditional information and the first noise sequence to obtain the first feature image sequence. Attention features are extracted from the first feature image sequence in both spatial and temporal dimensions to obtain the second feature image sequence. Convolutional feature extraction is performed on the second feature image sequence in both spatial and temporal dimensions to obtain the first point cloud scene image sequence.

3. The method according to claim 2, characterized in that, The spatiotemporal diffusion model includes an equidistant columnar spatiotemporal convolution module, which comprises a spatial convolution module, a temporal convolution module, and a fusion module; the step of extracting convolutional features in the spatial and temporal dimensions based on the first conditional information and the first noise sequence to obtain a first feature image sequence includes: Using the spatial convolution module, spatial features are extracted in the spatial dimension based on the first conditional information and the first noise sequence to obtain a first spatial feature image sequence; Using the temporal convolution module, temporal features are extracted in the temporal dimension based on the first spatial feature image sequence to obtain the first temporal feature image sequence; The fusion module is used to fuse the first spatial feature image sequence and the first temporal feature image sequence to obtain the first feature image sequence.

4. The method according to claim 3, characterized in that, The equidistant columnar spatiotemporal convolution module also includes a feature addition module; Before using the spatial convolution module to extract spatial features in the spatial dimension based on the first conditional information and the first noise sequence to obtain the first spatial feature image sequence, the method further includes: using the feature addition module to add Fourier features to the first noise sequence to obtain the target noise sequence. The step of using the spatial convolution module to extract spatial features in the spatial dimension based on the first conditional information and the first noise sequence to obtain a first spatial feature image sequence includes: using the spatial convolution module to extract spatial features in the spatial dimension based on the first conditional information and the target noise sequence to obtain a first spatial feature image sequence.

5. The method according to any one of claims 2 to 4, characterized in that, The spatiotemporal diffusion model includes an equidistant cylindrical spatiotemporal transformer, which includes a spatial attention module and a temporal attention module. The step of extracting attention features from the first feature image sequence in both spatial and temporal dimensions to obtain the second feature image sequence includes: Using the spatial attention module, feature extraction is performed based on the first feature image sequence to obtain the first spatial attention feature image sequence; Using the temporal attention module, feature extraction is performed based on the first spatial attention feature image sequence to obtain the second feature image sequence.

6. The method according to any one of claims 1 to 5, characterized in that, The prior information of the first object is obtained in the following way: Obtain the first object feature information corresponding to the scene to be processed; Using a point cloud generation model, prior information of the first object is generated based on the feature information of the first object.

7. A model training method, characterized in that, The method includes: Acquire the second condition information, the second point cloud scene image sequence, and the actual noise sequence corresponding to the sample scene. The second condition information includes at least one of the second road sketch information, the second scene description information, and the second object prior information. Using the spatiotemporal diffusion model to be trained, features are extracted in the spatial and temporal dimensions based on the second conditional information and the second point cloud scene image sequence to obtain the prediction noise sequence corresponding to the sample scene; The loss function is calculated based on the predicted noise sequence and the actual noise sequence. After the loss function satisfies the target condition, the spatiotemporal diffusion model of any one of claims 1-6 is obtained after training.

8. A scene generation device, characterized in that, The device includes: The first acquisition module is used to acquire first condition information and a first noise sequence corresponding to the scene to be processed. The first condition information includes at least one of first road sketch information, first scene description information and first object prior information. The first noise sequence is obtained by randomly sampling Gaussian distributed noise. The generation module is used to extract features in the spatial and temporal dimensions based on the first condition information and the first noise sequence using a spatiotemporal diffusion model, so as to obtain the first point cloud scene image sequence corresponding to the scene to be processed.

9. A model training device, characterized in that, The device includes: The second acquisition module is used to acquire second condition information, second point cloud scene image sequence and actual noise sequence corresponding to the sample scene. The second condition information includes at least one of second road sketch information, second scene description information and second object prior information. The training module is used to extract features in the spatial and temporal dimensions based on the second conditional information and the second point cloud scene image sequence using the spatiotemporal diffusion model to be trained, to obtain the predicted noise sequence corresponding to the sample scene; calculate the loss function based on the predicted noise sequence and the actual noise sequence, and obtain the spatiotemporal diffusion model according to any one of claims 1-6 after the loss function satisfies the target condition.

10. An electronic device, characterized in that, The electronic device includes a processor and a computer-readable storage medium on which a computer program is stored; When the computer program is executed by the processor, the processor performs the scene generation method as described in any one of claims 1 to 6, or the model training method as described in claim 7.

11. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, causes the processor to perform the scene generation method as described in any one of claims 1 to 6, or the model training method as described in claim 7.

Images