Virtual scene data generation method and system with controllable place features

Abstract

The invention relates to a place feature controllable virtual scene data generation method and system, belongs to the technical field of virtual scenes, and solves the problem of poor background consistency of multi-view image generation in the prior art. The method comprises the following steps: extracting multi-view scene images of different scenes and corresponding multi-modal scene information, and constructing a training sample set; a scene generation model is constructed, the scene generation model comprises a diffusion model and a Place-ID encoder, and the Place-ID encoder is used for generating a site ID code of the scene; taking the place ID code and the multi-modal scene information as generation conditions of a diffusion model; training the scene generation model based on a training sample set to obtain a trained scene generation model; inputting a real scene image into a Place-ID encoder to obtain a site ID code, and forming a to-be-generated condition by the site ID code and to-be-generated multi-modal scene information; and inputting the to-be-generated condition and the random tensor into the diffusion model to generate a virtual scene image. Therefore, multi-view virtual scene generation with consistent backgrounds is realized.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of virtual scene generation, and in particular to a virtual scene data generation method and system with controllable location features. BACKGROUND

[0002] In the field of computer vision and autonomous driving, in order to improve the environmental perception and positioning ability, a large number of diverse street scene image data are often needed for model training. Since there are many long-tail and extreme cases in real road scenes, it is difficult to cover all situations by relying on field collection, so it is an important demand to synthesize realistic training images.

[0003] In recent years, the generation model (especially the diffusion model) has made significant progress in image generation quality and stability. For example, a series of multi-view image generation methods have been able to generate street scene images with certain 3D geometric consistency according to the bird's eye view layout (BEV map) and other conditions. For example, the BEVGen model can synthesize street scene views from BEV layouts. In addition, methods such as MagicDrive apply diffusion models to street scene generation, supporting the control of camera poses, road topologies, 3D target boxes, and text descriptions, and ensuring consistency between multi-camera views to some extent through cross-view attention modules.

[0004] However, these existing technologies still have deficiencies in multi-view background consistency and location perception control. Specifically, current methods mainly rely on text, BEV maps, and target boxes for generation, lack explicit control of specific location features, and are difficult to ensure that the background content of the generated images at different angles remains consistent. For example, when changing the camera angle or generating different weather conditions, the building layout in the background often does not match, lacking the coherence of "scene identity". This deficiency makes it difficult for the synthesized images to truly reflect the environment of a fixed location, which is not conducive to using the generated images for training enhancement in tasks such as visual positioning (location recognition). SUMMARY

[0005] In view of the above analysis, the embodiments of the present application aim to provide a virtual scene data generation method and system with controllable location features to solve the problem of poor background consistency of images generated at different angles of the same scene in existing virtual scene generation.

[0006] In one aspect, the embodiments of the present application provide a virtual scene data generation method with controllable location features, comprising the following steps:

[0007] Extracting multi-view scene images and corresponding multi-modal scene information of different scenes to construct a training sample set;

[0008] construct a scene generation model, the scene generation model comprising a diffusion model and a Place-ID encoder for generating a place ID code of a scene; and encode the place ID code and multi-modal scene information as a generation condition of the diffusion model;

[0009] train the scene generation model based on the training sample set to obtain a trained scene generation model;

[0010] input a real scene image into the Place-ID encoder to obtain a corresponding place ID code, and input the place ID code and to-be-generated multi-modal scene information as a to-be-generated condition; and input the to-be-generated condition and a random tensor into the diffusion model to generate a corresponding virtual scene image.

[0011] In further improvement of the above method, the Place-ID encoder comprises:

[0012] a preprocessing module for preprocessing a scene image to obtain a preprocessed image;

[0013] a visual feature extraction module for inputting the preprocessed image into a trained CLIP image encoder to obtain a place visual feature;

[0014] an initial place embedding extraction module for inputting the preprocessed image into a trained visual place recognition network to obtain an initial place embedding;

[0015] a linear mapping module for mapping the initial place embedding to a fixed dimension;

[0016] a perception module for fusing the place visual feature into the initial place embedding by using a multi-layer cross-modal attention mechanism to obtain a place ID code of the scene image.

[0017] In further improvement of the above method, the diffusion model is a latent space diffusion model, and the latent space diffusion model comprises a pre-trained encoder and a Unet diffusion module.

[0018] training the scene generation model based on the training sample set comprises:

[0019] converting a scene image of each view of each sample in the training sample set into a latent representation based on the pre-trained encoder;

[0020] encoding multi-modal scene information corresponding to the scene image of each view to obtain a multi-modal condition representation;

[0021] In each training batch, for each scene image of each view of each sample in the batch, generate random noise, superimpose the random noise on the latent representation corresponding to the scene image to obtain a noisy latent representation; the multi-modal conditional representation and the place ID code corresponding to the scene image constitute the generation condition representation corresponding to the scene image; input the noisy latent representation and the generation condition representation corresponding to each scene image into the Unet diffusion module for noise prediction; update the parameters of the Place-ID encoder and the Unet diffusion model based on the prediction result.

[0022] Based on the further improvement of the above method, the training loss is calculated by the following formula:

[0023]

[0024] Wherein, represents the prediction loss of the kth sample, represents the contrastive loss of the kth sample, λ represents the weight, and K represents the number of samples in the current training batch.

[0025] Based on the further improvement of the above method, the contrastive loss of the kth sample is calculated by the following formula:

[0026]

[0027] Wherein, N represents the number of scene images of the kth sample, cl i represents the place ID code of the i-th scene image, cl j represents the place ID code of the j-th scene image, cl m represents the place ID code of the m-th scene image, p i represents the place visual feature of the i-th scene image.

[0028] Based on the further improvement of the above method, the perception module adopts a multi-layer Transformer structure;

[0029] The first layer of the Transformer structure takes the initial place embedding as the query, and takes the place visual feature as the key and value to perform cross-modal attention calculation; the output feature of the last layer of the Transformer structure is the place ID code of the scene.

[0030] Based on the further improvement of the above method, the multi-modal scene information includes the bird's eye view of the scene, the three-dimensional object information in the scene, the environment description information and the view parameter.

[0031] In another aspect, the embodiment of the present application provides a virtual scene data generation system with controllable location features, comprising:

[0032] a sample set construction module configured to extract multi-view scene images and corresponding multi-modal scene information of different scenes, and construct a training sample set;

[0033] a model construction module configured to construct a scene generation model, wherein the scene generation model comprises a diffusion model and a Place-ID encoder configured to generate a location ID code of a scene; and the location ID code and the multi-modal scene information are used as generation conditions of the diffusion model;

[0034] a model training module configured to train the scene generation model based on the training sample set, and obtain a trained scene generation model;

[0035] a scene generation module configured to input a real scene image into the Place-ID encoder to obtain a corresponding location ID code, input the location ID code and to-be-generated multi-modal scene information into a to-be-generated condition, and input the to-be-generated condition and a random tensor into the diffusion model to generate a corresponding virtual scene image.

[0036] In a further improvement based on the above system, the Place-ID encoder comprises:

[0037] a preprocessing module configured to pre-process a scene image to obtain a pre-processed image;

[0038] a visual feature extraction module configured to input the pre-processed image into a trained CLIP image encoder to obtain a location visual feature;

[0039] an initial location embedding extraction module configured to input the pre-processed image into a trained visual location recognition network to obtain an initial location embedding;

[0040] a linear mapping module configured to map the initial location embedding to a fixed dimension;

[0041] a perception module configured to adopt a multi-layer cross-modal attention mechanism to integrate the location visual feature into the initial location embedding to obtain a location ID code of the scene image.

[0042] In a further improvement based on the above system, the diffusion model is a latent space diffusion model, and the latent space diffusion model comprises a pre-trained encoder and a Unet diffusion module.

[0043] The training of the scene generation model based on the training sample set comprises:

[0044] conversion of the scene image of each view of each sample in the training sample set into a latent representation based on the pre-trained encoder.

[0045] Encode the multi-modal scene information corresponding to the scene image of each view to obtain a multi-modal conditional representation;

[0046] In each training batch, for each view of the scene image of each sample in the batch, generate random noise, superimpose the random noise on the latent representation corresponding to the scene image to obtain a noisy latent representation; the multi-modal conditional representation and the place ID encoding corresponding to the scene image constitute the generation condition representation corresponding to the scene image; input the noisy latent representation and the generation condition representation corresponding to each scene image into the Unet diffusion module for noise prediction; update the parameters of the Place-ID encoder and the Unet diffusion model based on the prediction result.

[0047] Compared with the prior art, the present application constructs a training sample set by extracting multi-view scene images and corresponding multi-modal scene information of different scenes, generates the place ID encoding of the scene as the condition of scene generation by adding a Place-ID encoder in the scene generation model, so that the background of the same scene is uniformly controlled and the background consistency of different views is maintained, trains the scene generation model based on the training sample set, inputs the real scene image into the Place-ID encoder to obtain the corresponding place ID encoding, and constitutes the to-be-generated condition by the place ID encoding and the to-be-generated multi-modal scene information, inputs the to-be-generated condition and the random tensor into the diffusion model, and thus the multi-view and background consistent virtual scene image can be generated.

[0048] The above technical solutions in the present application can also be combined with each other to realize more preferred combination solutions. Other features and advantages of the present application will be described in the subsequent specification, and some advantages will become apparent from the specification or be understood by implementing the present application. The purposes and other advantages of the present application can be realized and obtained from the contents specifically pointed out in the specification and the drawings. BRIEF DESCRIPTION OF DRAWINGS

[0049] The accompanying drawings are included to provide a further understanding of the application and are incorporated in and constitute a part of this specification, illustrate embodiments of the application and together with the description serve to explain the principles of the application. In the drawings:

[0050] Figure 1 Flow chart of the place feature controllable virtual scene data generation method of the embodiment of the present application;

[0051] Figure 2 Block diagram of the place feature controllable virtual scene data generation system of the embodiment of the present application. DETAILED DESCRIPTION

[0052] Preferred embodiments of the present application will be described in detail below with reference to the drawings, wherein the drawings constitute a part of this application, and are used to explain the principles of the embodiments of the present application, but are not used to limit the scope of the present application.

[0053] One specific embodiment of the present application discloses a virtual scene data generation method with controllable location features, as shown in the figure, comprising the following steps: Figure 1

[0054] S1, extract multi-view scene images and corresponding multi-modal scene information of different scenes to construct a training sample set;

[0055] S2, construct a scene generation model, the scene generation model includes a diffusion model and a Place-ID encoder, the Place-ID encoder is used to generate a location ID code of the scene; the location ID code and the multi-modal scene information are used as the generation conditions of the diffusion model;

[0056] S3, train the scene generation model based on the training sample set to obtain a trained scene generation model;

[0057] S4, input a real scene image into the Place-ID encoder to obtain a corresponding location ID code, and construct a to-be-generated condition with the location ID code and to-be-generated multi-modal scene information; input the to-be-generated condition and a random tensor into the diffusion model to generate a corresponding virtual scene image.

[0058] Compared with the prior art, the virtual scene data generation method with controllable location features provided in the embodiment extracts multi-view scene images and corresponding multi-modal scene information of different scenes to construct a training sample set, adds a Place-ID encoder in the scene generation model to generate a location ID code of the scene as the condition of scene generation, so that the background of the same scene is uniformly controlled and the background consistency of different perspectives is maintained, trains the scene generation model based on the training sample set, inputs a real scene image into the Place-ID encoder to obtain a corresponding location ID code, constructs a to-be-generated condition with the location ID code and to-be-generated multi-modal scene information; inputs the to-be-generated condition and a random tensor into the diffusion model, so as to generate a multi-view and background consistent virtual scene image, thereby producing a virtual scene image with controllable location features.

[0059] In implementation, the multi-modal scene information includes a bird's eye view of the scene, three-dimensional object information in the scene, environment description information, and perspective parameters.

[0060] The three-dimensional object information in the scene includes the type of the object and the three-dimensional bounding box parameters.

[0061] The environment description information includes weather, time, and environment description information, which is in the form of text.​

[0062] The view angle parameter is a parameter of a shooting camera, for example, a front view, a rear view or a side view angle, and a camera intrinsic parameter, etc.

[0063] Under the same timestamp, the multi-view scene image of a scene and the multi-modal scene information corresponding to the image of each view constitute a sample, thereby constructing a training sample set. For example, if each sample has six views, it contains the scene image and multi-modal scene information corresponding to the six views.

[0064] In order to generate more virtual scenes and maintain place consistency in the generated virtual scenes, a scene generation model is constructed, the scene generation model comprising a diffusion model and a Place-ID encoder for generating a place ID code of a scene; and the place ID code and the multi-modal scene information are used as generation conditions of the diffusion model.

[0065] In implementation, the diffusion model adopts an existing latent space diffusion model, for example, a Stable Diffusion model, comprising a pre-trained encoder and decoder and a Unet diffusion module.

[0066] Specifically, the Place-ID encoder comprises:

[0067] a preprocessing module configured to preprocess the scene image to obtain a preprocessed image;

[0068] a visual feature extraction module configured to input the preprocessed image into a trained CLIP image encoder to obtain a place visual feature;

[0069] an initial place embedding extraction module configured to input the preprocessed image into a trained visual place recognition network to obtain an initial place embedding;

[0070] a linear mapping module configured to map the initial place embedding to a fixed dimension;

[0071] a perception module configured to use a multi-layer cross-modal attention mechanism to integrate the place visual feature into the initial place embedding to obtain a place ID code of the scene image.

[0072] In implementation, in order to extract a unified place code of a scene, the scene image is first preprocessed. In implementation, for the scene image of each view, the sky region of the scene image can be shielded using an existing method, thereby guiding the network intersection to the street view and minimizing the interference of the sky and weather conditions.

[0073] The preprocessed image is input into a trained CLIP image encoder to obtain a place visual feature, denoted as E clip .

[0074] The pre-processed image is input into the trained visual place recognition network to obtain an initial place embedding, denoted as E MixVPR In implementation, the visual place recognition network adopts MixVPR, a background semantic feature embedding vector extracted from a real scene image, which is essentially a fixed-length (such as 4096-dimensional) high-dimensional representation for distinguishing the location characteristics of different places (rather than the perspective, weather, or foreground objects). These embeddings remain unchanged during training and represent “where and what this location is”, rather than “how the camera takes it from which angle”. Therefore, different perspective images of the same place will be mapped to close embedding spaces in the MixVPR network.

[0075] In implementation, in order to adapt to the diffusion model and align with other condition vectors, a linear mapping module is used to map the initial embedding to a fixed dimension, such as 1024 dimensions.

[0076] Then, based on the perception module, a multi-layer cross-modal attention mechanism is used to integrate the place visual features into the initial place embedding to obtain the place ID encoding of the scene, i.e., E place .

[0077] In implementation, the perception module adopts a multi-layer Transformer structure.

[0078] The first layer of the Transformer structure takes the initial place embedding as the query, and the place visual features as the key and value, to perform cross-modal attention calculation; the other layers of the Transformer structure take the output features of the previous layer of the Transformer structure as the query, and the place visual features as the key and value, to perform cross-modal attention calculation; and the output features of the last layer of the Transformer structure are the place ID encoding E place .

[0079] This interaction process can be understood as: the place ID encoding “focuses” on the visual content of the actual image, and extracts key background features related to the place from it. The final place ID encoding not only retains the unique scene ID information contained in the original place identifier, but also aligns to the visual semantic space, providing an easily accessible background condition for the diffusion model.

[0080] In implementation, the Place-ID encoder processes each perspective scene image in the training sample set.

[0081] Specifically, the scene generation model is trained based on the training sample set, including:

[0082] The pre-trained encoder based on the latent space diffusion model converts each perspective scene image of each sample in the training sample set into a latent representation;

[0083] The multi-modal scene information corresponding to the scene image of each view is encoded to obtain a multi-modal conditional representation;

[0084] In each training batch, for each view of the scene image of each sample in the batch, a random noise is generated, and the random noise is superimposed on the latent representation corresponding to the scene image to obtain a noisy latent representation; the multi-modal conditional representation and the place ID encoding corresponding to the scene image form a generation condition representation corresponding to the scene image; the noisy latent representation and the generation condition representation corresponding to the scene image are input into the Unet diffusion model for noise prediction; the loss is calculated based on the prediction result, and the parameters of the Place-ID encoder and the Unet diffusion model are updated.

[0085] In implementation, a pre-trained encoder can be used to convert each scene image of each sample in the sample set into a low-dimensional and compact representation to obtain a latent representation corresponding to each image, which is a clean and noise-free latent representation z.

[0086] In implementation, the scene information of different modalities can be encoded by corresponding encoders. For example, the environment description text can be encoded by using a pre-trained text embedding model (such as a Transformer encoder) or a CLIP text encoder to convert the environment description into a semantic vector representation E text .

[0087] The BEV map is processed by using a pre-trained aggregation neural network to extract a feature representation E map .

[0088] The encoding vector corresponding to the three-dimensional object information is represented as E box , and the encoding vector of the view parameter is represented as E cam .

[0089] The E text , E map , E box , E cam and E place corresponding to the scene image of each view constitute the generation condition of the scene image of the view.

[0090] For a sample in the current training batch, a random Gaussian noise is sampled for each view of the scene image of the sample, and the random Gaussian noise is superimposed on the latent representation corresponding to the scene image of the view to obtain a noisy latent representation. The specific process is as follows: an integer t is randomly sampled, and the added noisy latent representation is determined based on the integer wherein represents the number of noises determined according to t, and ε represents a random Gaussian noise.

[0091] The multi-modal condition representation and the place ID code of each view of the sample form a generation condition corresponding to the view. The corresponding noisy latent representation of the view and the generation condition are input into the diffusion model to obtain the predicted noise of the model. The loss is calculated based on the prediction result, and the parameters of the Place-ID encoder and the diffusion model are updated by back propagation, that is, the Place-ID encoder and the diffusion model are jointly trained.

[0092] Specifically, the training loss is calculated by the following formula:

[0093]

[0094] wherein, represents the prediction loss of the kth sample, represents the contrast loss of the kth sample, λ represents the weight, and K represents the number of samples in the current training batch.

[0095] wherein, the prediction loss is the prediction loss of the diffusion model, that is, the loss between the predicted noise and the actual noise. In implementation, the sum of the losses between the predicted noise and the actual noise corresponding to each scene image of the kth sample is calculated to obtain the prediction loss of the kth sample. The mean square error can be used to calculate the loss between the predicted noise and the actual noise.

[0096] In order to ensure that the place features output by the perception module accurately correspond to the background semantics of the target place, a SoftCLIP contrast loss is introduced in the training process.

[0097] Specifically, the contrast loss of the kth sample is calculated by the following formula:

[0098]

[0099] wherein, N represents the number of scene images of the kth sample, cl i represents the place ID code of the i th scene image, cl j represents the place ID code of the j th scene image, cl m represents the place ID code of the m th scene image, p i represents the place visual feature of the i th scene image. The number of scene images of the sample is the number of views.

[0100] In the contrast learning, which is the teacher network, is the similarity softmax distribution between CLIP image features (place visual features), that is, the i th scene image "thinks" the similarity between it and all other scene images of views.

[0101] In the contrast learning, The student network is equivalent to the place ID encoder, which represents the matching similarity of the location ID code to all CLIP image features (location visual features).

[0102] The SoftCLIP loss takes the teacher model as the "reference distribution" and measures the difference between the student model and the teacher model using the KL divergence:

[0103]

[0104] The SoftCLIP loss measures whether the semantic distribution output by the student model can successfully "imitate" the image semantic distribution of the teacher model (CLIP). The smaller the loss, the closer the place ID code representation to the image semantic space of CLIP.

[0105] The introduction of the SoftCLIP loss function can effectively enhance the consistency between the output features of the Place-ID encoder and the scene location semantics, and improve the location semantic accuracy of the generated image.

[0106] The SoftCLIP loss provides a "soft constraint" during training, which pushes the features output by the Place-ID encoder to fall into a semantic space compatible with real images. Through the above design, the Place-ID encoder can convert the location ID code into a semantic vector in the fixed CLIP space. This vector is used as a conditional input for the diffusion generation network to fuse, so that the generation network is always guided by the specific location semantics during image synthesis, ensuring the consistency and authenticity of the generated image background under different perspectives. It is worth emphasizing that this location feature mainly affects the background environmental elements (such as architectural style and street layout), but does not constrain the arrangement of foreground objects. It allows the foreground vehicles, pedestrians, etc. to change flexibly according to the input conditions while keeping the background unchanged. Therefore, the invention realizes the effect of "constant background and variable foreground" for street scene image generation.

[0107] When the scene generation model converges, stop training and obtain the trained scene generation network model.

[0108] For a real scene, input the scene image of a certain angle into the Place-ID encoder to obtain the corresponding location ID code. Input the location ID code and the to-be-generated multi-modal scene information (E text , E map , E box , E cam) to be generated. A random tensor is generated by a noise generation component based on a latent space diffusion model, and the to-be-generated condition and the random tensor are input into a Unet diffusion module of the diffusion model to obtain a noise-free latent representation, and then input into a pre-trained decoder of the latent space diffusion model to obtain a corresponding image. Since the Place-ID encodings of different viewpoints of the same scene are similar, a set of virtual multi-view image data with time synchronization and unified background can be obtained.

[0109] One specific embodiment of the present application discloses a virtual scene data generation system with controllable place features, as shown in the accompanying drawings, comprising: Figure 2

[0110] A sample set construction module is configured to extract multi-view scene images and corresponding multi-modal scene information of different scenes, and construct a training sample set.

[0111] A model construction module is configured to construct a scene generation model, wherein the scene generation model comprises a diffusion model and a Place-ID encoder configured to generate a place ID encoding of a scene; and the place ID encoding and the multi-modal scene information are used as generation conditions of the diffusion model.

[0112] A model training module is configured to train the scene generation model based on the training sample set to obtain a trained scene generation model.

[0113] A scene generation module is configured to input a real scene image into the Place-ID encoder to obtain a corresponding place ID encoding, and input the place ID encoding and to-be-generated multi-modal scene information into a to-be-generated condition; and input the to-be-generated condition and a random tensor into the diffusion model to generate a corresponding virtual scene image.

[0114] Preferably, the Place-ID encoder comprises:

[0115] A preprocessing module is configured to preprocess a scene image to obtain a preprocessed image.

[0116] A visual feature extraction module is configured to input the preprocessed image into a trained CLIP image encoder to obtain a place visual feature.

[0117] An initial place embedding extraction module is configured to input the preprocessed image into a trained visual place recognition network to obtain an initial place embedding.

[0118] A linear mapping module is configured to map the initial place embedding to a fixed dimension.

[0119] A perception module is configured to use a multi-layer cross-modal attention mechanism to integrate the place visual feature into the initial place embedding to obtain a place ID encoding of the scene image.

[0120] ​Preferably, the diffusion model is a latent space diffusion model, and the latent space diffusion model comprises a pre-trained encoder and a Unet diffusion module.

[0121] The scene generation model is trained based on the training sample set, including:

[0122] The scene image of each view of each sample in the training sample set is converted into a latent representation based on the pre-trained encoder.

[0123] The multi-modal scene information corresponding to the scene image of each view is encoded to obtain a multi-modal conditional representation.

[0124] In each training batch, for the scene image of each view of each sample in the batch, a random noise is generated, and the random noise is superimposed on the latent representation corresponding to the scene image to obtain a noisy latent representation; the multi-modal conditional representation corresponding to the scene image and the place ID code constitute the generation condition representation corresponding to the scene image; the noisy latent representation corresponding to each scene image and the generation condition representation are input into the Unet diffusion module for noise prediction; the loss is calculated based on the prediction result, and the parameters of the Place-ID encoder and the Unet diffusion model are updated. The method embodiment and the system embodiment are based on the same principle, and the related parts can be mutually borrowed and the same technical effect can be achieved. For specific implementation process, refer to the foregoing embodiments, which will not be repeated here.

[0125] Electronic device embodiment:

[0126] One specific implementation of the present application discloses an electronic device, comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein the processor executes the program to implement the steps of the virtual scene data generation method with controllable location features in the method embodiment.

[0127] Readable storage medium embodiment:

[0128] One specific implementation of the present application discloses a non-transitory computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement the steps of the virtual scene data generation method with controllable location features in the method embodiment.

[0129] Those skilled in the art can understand that all or part of the processes of the above-mentioned embodiments can be completed by a computer program instructing related hardware, and the program can be stored in a computer readable storage medium. The computer readable storage medium includes a disk, an optical disk, a read-only memory, a random access memory, etc.

[0130] The above merely describes preferred specific embodiments of the present application, but the protection scope of the present application is not limited thereto, and any person skilled in the art can easily think of changes or replacements within the technical scope disclosed by the present application, which should be covered within the protection scope of the present application.

Claims

1. A method for generating virtual scene data with controllable location features, characterized in that, Includes the following steps: Extract multi-view scene images and corresponding multimodal scene information from different scenarios to construct a training sample set; A scene generation model is constructed, which includes a diffusion model and a Place-ID encoder. The Place-ID encoder is used to generate the location ID encoding of the scene. The location ID encoding and multimodal scene information are used as the generation conditions of the diffusion model. The scene generation model is trained based on the training sample set to obtain a trained scene generation model; The real scene image is input into the Place-ID encoder to obtain the corresponding location ID code. The location ID code and the multimodal scene information to be generated constitute the conditions to be generated. The conditions to be generated and the random tensor are input into the diffusion model to generate the corresponding virtual scene image.

2. The method for generating virtual scene data with controllable location features according to claim 1, characterized in that, The Place-ID encoder includes: The preprocessing module is used to preprocess the scene image to obtain a preprocessed image; The visual feature extraction module is used to input the preprocessed image into the trained CLIP image encoder to obtain location visual features; The initial location embedding extraction module is used to input the preprocessed image into the trained visual location recognition network to obtain the initial location embedding; The linear mapping module is used to embed the initial location into a fixed-dimensional mapping. The perception module is used to incorporate location visual features into the initial location embedding to obtain the location ID encoding of the scene image using a multi-layer cross-modal attention mechanism.

3. The method for generating virtual scene data with controllable location features according to claim 1, characterized in that, The diffusion model is a latent spatial diffusion model, which includes a pre-trained encoder and a Unet diffusion module; Training the scene generation model based on the training sample set includes: The pre-trained encoder converts scene images from each viewpoint of each sample in the training sample set into a latent representation; The multimodal scene information corresponding to the scene image from each viewpoint is encoded to obtain a multimodal conditional representation; In each training batch, for each scene image of each sample in that batch from each viewpoint, random noise is generated and superimposed on the latent representation corresponding to the scene image to obtain a noisy latent representation. The multimodal conditional representation and location ID encoding corresponding to the scene image constitute the generated conditional representation corresponding to the scene image. The noisy latent representation and generated conditional representation corresponding to each scene image are input into the Unet diffusion module for noise prediction. The loss is calculated based on the prediction results, and the parameters of the Place-ID encoder and the Unet diffusion model are updated.

4. The method for generating virtual scene data with controllable location features according to claim 3, characterized in that, The training loss is calculated using the following formula: in, This represents the prediction loss for the k-th sample. Let λ represent the contrast loss for the k-th sample, λ represent the weights, and K represent the number of samples in the current training batch.

5. The method for generating virtual scene data with controllable location features according to claim 4, characterized in that, The contrast loss for the k-th sample is calculated using the following formula: Where N represents the number of scene images for the k-th sample, cl i cl represents the location ID encoding of the i-th scene image. j cl represents the location ID encoding of the j-th scene image. m p represents the location ID encoding of the m-th scene image. i This represents the location visual features of the i-th scene image.

6. The method for generating virtual scene data with controllable location features according to claim 2, characterized in that, The sensing module adopts a multi-layer Transformer structure; The first Transformer layer uses the initial location embedding as the query and the location visual features as the key and value to perform cross-modal attention computation; the other Transformer layers use the output features of the previous Transformer layer as the query and the location visual features as the key and value to perform cross-modal attention computation; the output feature of the last Transformer layer is the location ID encoding of the scene.

7. The method for generating virtual scene data with controllable location features according to claim 1, characterized in that, The multimodal scene information includes a bird's-eye view of the scene, information on three-dimensional objects in the scene, environmental description information, and viewpoint parameters.

8. A virtual scene data generation system with controllable location features, characterized in that, include: The sample set construction module is used to extract multi-view scene images and corresponding multimodal scene information from different scenes to construct a training sample set; The model building module is used to build a scene generation model, which includes a diffusion model and a Place-ID encoder. The Place-ID encoder is used to generate the location ID encoding of the scene. The location ID encoding and multimodal scene information are used as the generation conditions of the diffusion model. The model training module trains the scene generation model based on the training sample set to obtain a trained scene generation model. The scene generation module is used to input real scene images into the Place-ID encoder to obtain the corresponding location ID encoding, and to use the location ID encoding and the multimodal scene information to be generated to form the conditions to be generated; and to input the conditions to be generated and the random tensor into the diffusion model to generate the corresponding virtual scene image.

9. The location-feature-controllable virtual scene data generation system according to claim 8, characterized in that, The Place-ID encoder includes: The preprocessing module is used to preprocess the scene image to obtain a preprocessed image; The visual feature extraction module is used to input the preprocessed image into the trained CLIP image encoder to obtain location visual features; The initial location embedding extraction module is used to input the preprocessed image into the trained visual location recognition network to obtain the initial location embedding; The linear mapping module is used to embed the initial location into a fixed-dimensional mapping. The perception module is used to incorporate location visual features into the initial location embedding to obtain the location ID encoding of the scene image using a multi-layer cross-modal attention mechanism.

10. The location-feature-controllable virtual scene data generation system according to claim 8, characterized in that, The diffusion model is a latent spatial diffusion model, which includes a pre-trained encoder and a Unet diffusion module; Training the scene generation model based on the training sample set includes: The pre-trained encoder converts scene images from each viewpoint of each sample in the training sample set into a latent representation; The multimodal scene information corresponding to the scene image from each viewpoint is encoded to obtain a multimodal conditional representation; In each training batch, for each scene image of each sample in that batch from each viewpoint, random noise is generated and superimposed on the latent representation corresponding to the scene image to obtain a noisy latent representation. The multimodal conditional representation and location ID encoding corresponding to the scene image constitute the generated conditional representation corresponding to the scene image. The noisy latent representation and generated conditional representation corresponding to each scene image are input into the Unet diffusion module for noise prediction. The loss is calculated based on the prediction results, and the parameters of the Place-ID encoder and the Unet diffusion model are updated.

Images