Image generation method, device, equipment and medium based on Brownian bridge diffusion model

Abstract

The present application relates to an image generation method, apparatus, device, and medium based on a Brownian bridge diffusion model. The method includes: receiving a set of image combinations, the image combination including a satellite image and a ground panoramic image corresponding to the satellite image; extracting shared features of the satellite image and the ground panoramic image using a cross-view image joint encoder; performing a polar coordinate transformation on the satellite image, encoding the polar-transformed satellite image and the ground panoramic image into a latent space to obtain an initial satellite image latent vector and a ground panoramic image latent vector; gradually adding noise to the ground panoramic image latent vector in the latent space to obtain a satellite image latent vector; gradually removing noise from the satellite image latent vector based on the satellite image latent vector and the shared features to generate a target latent vector; and decoding the target latent vector to generate a target ground panoramic image. This application improves the efficiency and quality of converting ground panoramic images from satellite images.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to an image generation method, device, equipment and medium based on a Brownian bridge diffusion model. Background Art

[0002] With the rapid development of remote sensing technology, satellite imagery has become a crucial tool for understanding and monitoring the Earth. However, directly acquired satellite imagery is often limited by factors such as resolution, viewing angle, and lighting conditions, making it difficult to directly use it as ground-level panoramic images. Therefore, methods for converting existing satellite imagery into ground-level panoramic images have become an important research area.

[0003] Existing methods for converting satellite imagery into ground-based panoramic images often rely on additional ground-based image information (such as semantic segmentation maps and depth maps) for training or inference, which is not suitable for real-world applications. Furthermore, existing methods suffer from training instability, requiring careful parameter adjustment and complex network structure design to maintain a balanced training process. Consequently, existing methods for converting ground-based panoramic images from satellite imagery have low efficiency and quality. Summary of the Invention

[0004] The purpose of the embodiments of the present application is to propose an image generation method, device, equipment and medium based on the Brownian bridge diffusion model to achieve efficiency and quality in converting satellite images into ground panoramic images.

[0005] In order to solve the above technical problems, the embodiment of the present application provides an image generation method based on the Brownian bridge diffusion model, including:

[0006] receiving a set of image combinations, wherein the image combinations include satellite images and ground panoramic images corresponding to the satellite images;

[0007] extracting shared features of the satellite image and the ground panoramic image through a cross-view image joint encoder;

[0008] Performing polar coordinate transformation on the satellite image, and encoding the polar coordinate transformed satellite image and the ground panoramic image into a latent space to obtain an initial satellite image latent vector and a ground panoramic image latent vector;

[0009] In the latent space, performing a Brownian bridge forward process based on the initial satellite image latent vector and the ground panoramic image latent vector to gradually add noise to the ground panoramic image latent vector to obtain a satellite image latent vector;

[0010] performing a Brownian bridge inverse process based on the satellite image latent vector and the shared features to gradually remove noise from the satellite image latent vector and generate a target latent vector;

[0011] The target potential vector is decoded to generate a target ground panoramic image.

[0012] In order to solve the above technical problems, the embodiment of the present application provides an image generation device based on a Brownian bridge diffusion model, comprising:

[0013] An image receiving unit, configured to receive a set of image combinations, wherein the image combination includes a satellite image and a ground panoramic image corresponding to the satellite image;

[0014] a feature extraction unit, configured to extract shared features of the satellite image and the ground panoramic image through a cross-view image joint encoder;

[0015] a coordinate transformation unit, configured to perform polar coordinate transformation on the satellite image, and encode the polar coordinate transformed satellite image and the ground panoramic image into a latent space to obtain an initial satellite image latent vector and a ground panoramic image latent vector;

[0016] a noise adding unit, configured to perform a Brownian bridge forward process in the latent space based on the initial satellite image latent vector and the ground panoramic image latent vector, so as to gradually add noise to the ground panoramic image latent vector to obtain a satellite image latent vector;

[0017] a noise removal unit, configured to perform a Brownian bridge inverse process based on the satellite image latent vector and the shared feature to gradually remove noise from the satellite image latent vector and generate a target latent vector;

[0018] The image generation unit is used to decode the target potential vector to generate a target ground panoramic image.

[0019] To solve the above technical problems, a technical solution adopted by the present invention is: providing a computer device, including one or more processors; a memory for storing one or more programs, so that the one or more processors implement any of the above-mentioned image generation methods based on the Brownian bridge diffusion model.

[0020] To solve the above technical problems, a technical solution adopted by the present invention is: a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, it implements any one of the above-mentioned image generation methods based on the Brownian bridge diffusion model.

[0021] An embodiment of the present invention provides an image generation method, apparatus, device, and medium based on a Brownian bridge diffusion model. The method includes: receiving a set of image combinations, wherein the image combination includes a satellite image and a ground panoramic image corresponding to the satellite image; extracting shared features of the satellite image and the ground panoramic image using a cross-view image joint encoder; performing a polar coordinate transformation on the satellite image, and encoding the polar coordinate transformed satellite image and the ground panoramic image into a latent space to obtain an initial satellite image latent vector and a ground panoramic image latent vector; in the latent space, performing a Brownian bridge forward process based on the initial satellite image latent vector and the ground panoramic image latent vector to gradually add noise to the ground panoramic image latent vector to obtain a satellite image latent vector; performing a Brownian bridge reverse process based on the satellite image latent vector and the shared features to gradually remove noise from the satellite image latent vector to generate a target latent vector; and decoding the target latent vector to generate a target ground panoramic image. The embodiments of the present invention extract shared features from satellite images and ground panoramic images through a cross-view image joint encoder, perform Brownian bridge forward and reverse processes, and simultaneously inject the shared features into the reverse process. This facilitates the generation of detailed and structurally precise ground panoramic images from satellite images, improving image fidelity and semantic consistency. Furthermore, the embodiments of the present application also perform Brownian bridge forward and reverse processes in the latent space, reducing model training and inference costs and improving image conversion efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In order to more clearly illustrate the solutions in this application, a brief introduction will be given below to the drawings required for use in the description of the embodiments of this application. Obviously, the drawings described below are some embodiments of this application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0023] Figure 1 This is a flowchart of an implementation of the image generation method based on the Brownian bridge diffusion model provided in an embodiment of the present application;

[0024] Figure 2 This is a flowchart for implementing the first sub-process in the image generation method based on the Brownian bridge diffusion model provided in an embodiment of the present application;

[0025] Figure 3 This is a flowchart of an implementation process of an image generation method based on a Brownian bridge diffusion model provided by another embodiment of the present application;

[0026] Figure 4 This is a flowchart for implementing the second sub-process in the image generation method based on the Brownian bridge diffusion model provided in an embodiment of the present application;

[0027] Figure 5 This is a flowchart for implementing the third sub-process in the image generation method based on the Brownian bridge diffusion model provided in an embodiment of the present application;

[0028] Figure 6 This is a flowchart for implementing the fourth sub-process in the image generation method based on the Brownian bridge diffusion model provided in an embodiment of the present application;

[0029] Figure 7 This is a flowchart for implementing the fifth sub-process in the image generation method based on the Brownian bridge diffusion model provided in an embodiment of the present application;

[0030] Figure 8 Schematic diagram of an image generation device based on a Brownian bridge diffusion model provided in an embodiment of the present application;

[0031] Figure 9 It is a schematic diagram of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0032] Unless otherwise defined, all technical and scientific terms used herein have the same meanings as commonly understood by those skilled in the art to which this application belongs. The terms used in the specification of the application are for the purpose of describing specific embodiments only and are not intended to limit this application. The terms "including" and "having" and any variations thereof in the specification and claims of this application and the above-mentioned drawings are intended to cover non-exclusive inclusions. The terms "first", "second", etc. in the specification and claims of this application or the above-mentioned drawings are used to distinguish different objects, not to describe a specific order.

[0033] References herein to "embodiments" mean that a particular feature, structure, or characteristic described in connection with the embodiments may be included in at least one embodiment of the present application. The appearance of this phrase in various places in the specification does not necessarily refer to the same embodiment, nor does it constitute an independent or alternative embodiment that is mutually exclusive of other embodiments. It is understood, both explicitly and implicitly, by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0034] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the accompanying drawings.

[0035] The present invention will be described in detail below with reference to the accompanying drawings and embodiments.

[0036] It should be noted that the image generation method based on the Brownian bridge diffusion model provided in the embodiment of the present application is generally executed by a server. Accordingly, the image generation device based on the Brownian bridge diffusion model is generally configured in the server.

[0037] See also Figure 1 , Figure 1 A specific implementation of the image generation method based on the Brownian bridge diffusion model is shown.

[0038] It should be noted that the method of the present invention is not limited to the method of Figure 1 The process sequence shown is limited to the following steps:

[0039] S1: Receive a set of image combinations, where the image combination includes a satellite image and a ground panoramic image corresponding to the satellite image.

[0040] The present embodiment provides an image generation method based on a Brownian bridge diffusion model for converting satellite images into ground-based panoramic images, thereby improving the efficiency and quality of image conversion. Furthermore, the present embodiment has broad application prospects in multiple fields, including but not limited to wide-area virtual environment modeling, augmented reality content creation, 3D game development, advanced simulation training, and cross-view image matching. In the present embodiment, a set of image combinations is obtained, which is a satellite image and a corresponding ground-based panoramic image.

[0041] S2: Extracting shared features of the satellite image and the ground panoramic image through a cross-view image joint encoder.

[0042] Specifically, a cross-view image joint encoder is used to extract shared features from satellite and ground panoramic images. These shared features are then embedded into a shared feature space, providing feature information for the subsequent diffusion model. The cross-view image joint encoder is trained using contrastive learning to generate an encoder for extracting shared features from cross-view images.

[0043] See also Figure 2 , Figure 2 A specific implementation of step S2 is shown, which is described in detail as follows:

[0044] S21: Extracting features of the satellite image and the ground panoramic image respectively to obtain initial satellite image features and initial ground image features.

[0045] S22: performing average pooling on the initial satellite image features and the initial ground image features respectively to obtain basic satellite image features and basic ground image features.

[0046] S23: Embed the basic satellite image features and the basic ground image features into a shared feature space using a pre-trained encoder to obtain the shared features.

[0047] Specifically, a convolutional neural network (CNN), such as ConvNeXt-B, is used as an encoder. This pre-trained encoder extracts features from satellite images and ground panoramic images, respectively, to obtain initial satellite image features and initial ground image features. These initial satellite image features and initial ground image features capture important information from these images, such as shape, texture, and color. To reduce the dimensionality of the feature map while retaining the most important feature information, average pooling is performed on the initial satellite image features and initial ground image features, respectively, to obtain basic satellite image features and basic ground image features. Finally, the basic satellite image features and basic ground image features are embedded into a shared feature space to obtain shared features.

[0048] See also Figure 3 , Figure 3 A specific implementation method before step S23 is shown, which is described in detail as follows:

[0049] S23A: Acquire an original satellite image and an original ground panoramic image corresponding to the original satellite image.

[0050] S23B: Using a convolutional neural network as an encoder, and performing feature extraction and feature pooling on the original satellite image and the original ground panoramic image based on the encoder to obtain a compressed feature vector.

[0051] S23C: Using the InfoNCE loss function as the model loss function, and adopting the GPS coordinate sampling method and the dynamic similarity sampling method to train the encoder based on the compressed feature vector to obtain the pre-trained encoder.

[0052] Specifically, a set of original satellite images and original ground panoramic image pairs are collected, and the original features are obtained through a cross-view image joint encoder. Finally, a compressed feature vector is obtained through average pooling in the spatial dimension. The compressed feature vector is used to calculate the loss and optimize the encoder parameters. The InfoNCE loss function is used as the model loss function, and the encoder is trained based on the compressed feature vector using the GPS coordinate sampling method and the dynamic similarity sampling method. The encoder is optimized so that the similarity of the positive sample pair is higher than that of the negative sample pair. When the training is completed, the pre-trained encoder in the embodiment of the present application is obtained.

[0053] In this embodiment, the Input Noise-Contrast Estimation (INFONCE) loss function is used as the model loss function, which optimizes the model by comparing positive and negative sample pairs in a small batch. In the early stages of the training process, considering that the model's discriminative ability is still immature, this embodiment adopts a GPS coordinate sampling method to select images with similar geographical locations as negative samples. This method not only enhances the model's sensitivity to geographical features, but also improves its ability to recognize geographical proximity. As training progresses, the model's discriminative ability gradually increases. This embodiment introduces a dynamic similarity sampling method. This method dynamically selects images that are highly similar to the query image in feature space as negative samples during training, further mining difficult-to-distinguish samples. Implementation of this method enables the model to focus more on learning to distinguish subtle visual and geographical features in the later stages of training. By focusing on GPS sampling in the early stages of training and gradually transitioning to a dynamic similarity sampling strategy in the later stages, not only is the mining process for difficult samples optimized, but the encoder's ability to distinguish cross-view image features is also significantly improved.

[0054] Among them, the InfoNCE loss function is a commonly used contrastive loss function used to train models to distinguish between positive samples and negative samples. In image matching or feature embedding tasks, InfoNCE can help the model learn more discriminative feature representations. GPS coordinate sampling method: Since there is a geographical correspondence between satellite images and ground panoramic images, GPS coordinates can be used to sample positive sample pairs (i.e., image pairs with close geographical locations) and negative sample pairs (i.e., image pairs with farther geographical locations). This method helps the model learn features related to geographical location. Dynamic similarity sampling method: Dynamically sample negative samples based on the similarity between feature vectors. Specifically, for each positive sample pair, other images with higher similarity to the positive sample feature vector can be selected from the dataset as negative samples to increase the difficulty of training and the generalization ability of the model.

[0055] Training Process: During training, the encoder continuously adjusts its parameters based on feedback from the InfoNCE loss function to minimize the feature distance between positive pairs and maximize the feature distance between negative pairs. In this way, the encoder learns a more effective feature representation, embedding features from satellite and ground-based panoramic images into the same shared feature space.

[0056] After training, the embodiment of the present application obtains a pre-trained encoder that can convert original satellite images and ground panoramic images into compressed feature vectors in the same feature space.

[0057] S3: Perform polar coordinate transformation on the satellite image, and encode the polar coordinate transformed satellite image and the ground panoramic image into a latent space to obtain an initial satellite image latent vector and a ground panoramic image latent vector.

[0058] Specifically, a polar coordinate transformation is performed on the satellite image to partially bridge the view difference between the satellite image and the ground panoramic image, making the two geometrically closer and facilitating subsequent processing. The polar coordinate transformed satellite image and ground panoramic image are then encoded into a latent space to obtain the initial satellite image latent vector and the ground panoramic image latent vector.

[0059] Since training a diffusion model in pixel space will bring high training costs and video memory consumption. Therefore, the embodiment of the present application uses the encoder of the VQGAN (Vector Quantitative Generative Adversarial Network) model to encode the satellite image and the ground panoramic image after polar coordinate transformation into the latent space to obtain a latent vector. Among them, the ground panoramic image is a real ground panoramic image. The latent space can be regarded as a compression of the pixel space. For example, a 256×256×3 image can be encoded into 64×64×3. The VQGAN model encoding and decoding process can well ensure the authenticity of the image, which is beneficial to improving the quality of image conversion. Encoding the image into the latent space reduces the data dimension, reduces the training cost and video memory consumption, and is beneficial to improving the efficiency of converting from satellite images to panoramic images.

[0060] S4: In the latent space, a Brownian bridge forward process is performed based on the initial satellite image latent vector and the ground panoramic image latent vector to gradually add noise to the ground panoramic image latent vector to obtain a satellite image latent vector.

[0061] Specifically, starting with the real ground panoramic image latent vector, a fixed mapping from the ground panoramic image domain to the satellite image domain is constructed. In the latent space, noise is gradually added to the ground panoramic image latent vector through a forward diffusion process, simulating the transformation from the ground panoramic image latent vector to the satellite image latent vector. When the current ground panoramic image latent vector reaches the satellite image latent vector, no more noise is added, and the ground panoramic image latent vector at this point is used as the satellite image latent vector. This process allows the calculation of true noise labels for the subsequent Brownian bridge backward process to guide the training process.

[0062] The Brownian bridge forward process simulates a transformation from a given starting point (such as the latent vector of a real ground panoramic image) to a target domain (such as a satellite image domain) in the latent space by gradually adding noise or other forms of transformation. This process aims to simulate the natural transition of images from one representation to another in the latent space while maintaining or emphasizing the common features or structures between the two image domains.

[0063] See also Figure 4 , Figure 4 A specific implementation of step S4 is shown, which is described in detail as follows:

[0064] S41: Constructing a mapping path from the ground panoramic image latent vector to the initial satellite image latent vector in the latent space.

[0065] S42: gradually adding noise to the ground panoramic image potential vector along the mapping path in a forward diffusion manner, and recording the noise as a true noise label.

[0066] S43: If the current ground panoramic image potential vector reaches the initial satellite image potential vector, stop adding noise and use the current ground panoramic image potential vector as the satellite image potential vector.

[0067] Specifically, in the latent space, a mapping path is constructed from the ground panoramic image latent vector to the initial satellite image latent vector. Along this mapping path, noise is gradually added to the ground panoramic image latent vector. This noise addition process is controllable, following a specific noise distribution (e.g., Gaussian noise), and the intensity and type of the noise can be varied gradually as the transformation proceeds. Noise addition is a key step in simulating image quality degradation. By gradually adding noise, the transition from high quality (ground panoramic image) to low quality (satellite image) can be simulated. After each noise addition step, a corresponding true noise label is generated. This true noise label records the specific information about the noise added to the latent vector in the current step (e.g., noise type, intensity, and distribution). The true noise labels are crucial for the subsequent Brownian bridge reverse process, as they serve as true noise information to guide noise removal in the reverse process. The noise addition and noise label generation steps are then repeated until the current ground panoramic image latent vector reaches the satellite image latent vector. Noise addition is then stopped, and the current ground panoramic image latent vector is used as the satellite image latent vector.

[0068] The iterative process of noise addition requires multiple iterations, and the specific number of iterations depends on the complexity of the mapping relationship and the noise addition strategy.

[0069] In this embodiment, the Langbridge forward pass maps the true ground-based panoramic image latent vector to the satellite image latent vector and generates realistic noise labels to guide the subsequent backward pass. This process is a crucial step in image generation and conversion tasks, helping to model the gradual degradation of image quality in the latent space.

[0070] S5: Performing a Brownian bridge inverse process based on the satellite image latent vector and the shared features to gradually remove noise in the satellite image latent vector and generate a target latent vector.

[0071] Specifically, a Brownian bridge inverse process is performed based on the satellite image latent vector and the shared features, so as to realize the satellite image latent vector as the starting point, gradually predict and remove the noise in the satellite image latent vector, gradually generate the ground panoramic image latent vector, and obtain the target latent vector. In an embodiment of the present application, an attention mechanism is introduced in the execution of the Brownian bridge inverse process, and the shared features of the cross-view images are injected through the cross-attention mechanism. The cross-attention mechanism uses the latent vector in the Brownian bridge inverse process as the query vector, and the shared feature vector extracted by the cross-view joint encoder as the key vector and value vector, and uses scaled dot product attention for calculation. Through the introduction of the cross-attention mechanism, the latent vector in the generation process can extract the high-dimensional cross-view image information present in the shared features, thereby improving the semantic consistency and fidelity of the generated ground panoramic images and satellite images.

[0072] The Brownian Bridge reverse process starts with the satellite image's latent vector and, through a series of carefully designed transformations or network layers, gradually predicts and removes the noise added in the forward process to gradually approximate and generate the latent vector of the target domain image (such as a ground panoramic image). This process is usually equivalent to the Brownian Bridge forward process, where the forward process simulates the loss of image information by adding noise, while the reverse process attempts to recover this information.

[0073] See also Figure 5 , Figure 5 A specific implementation of step S5 is shown, which is described in detail as follows:

[0074] S51: Constructing a reverse mapping path based on the satellite image potential vector.

[0075] S52: gradually removing noise from the satellite image latent vector based on the reverse mapping path, and updating the current satellite image latent vector in combination with the attention mechanism and the shared feature during each step of removing noise from the satellite image latent vector.

[0076] S53: When the reverse mapping path is completed, the target latent vector is generated.

[0077] Specifically, a reverse mapping path is constructed in the latent space, which is the opposite of the forward Brownian bridge process. This reverse mapping path connects the satellite image latent vector and the ground panoramic image latent vector and specifies the noise removal path. Starting from the satellite image latent vector, the previously added noise is gradually removed along the reverse mapping path. The noise removal process is controllable and follows the reverse steps of the noise addition process. During each noise removal step, an attention mechanism is combined to predict and remove a portion of the noise based on the current state of the latent vector and the information in the ground panoramic image latent vector. A cross-attention mechanism is introduced to enhance the semantic consistency and fidelity of the generated image. The cross-attention mechanism allows the latent vector in the generation process to focus on and utilize the shared features extracted by the cross-view joint encoder. These shared features contain key information between the satellite image and the ground panoramic image. Through the cross-attention mechanism, the latent vector in the generation process can fully utilize this information to generate more accurate and realistic ground panoramic images.

[0078] See also Figure 6 , Figure 6 A specific implementation of step S52 is shown, which is described in detail as follows:

[0079] S521: gradually removing noise from the satellite image latent vector based on the reverse mapping path.

[0080] S522: In each step of removing noise from the satellite image latent vector, the current satellite image latent vector is used as a query vector, and the shared features are used as a key vector and a value vector respectively.

[0081] S523: Calculate the attention weights of the query vector and the key vector, and update the value vector based on the attention weights to obtain an updated latent vector.

[0082] Specifically, the noise in the satellite image latent vector is gradually removed based on the reverse mapping path. At each step, the current satellite image latent vector is used as the query vector, and the shared features are used as the key vector and value vector, respectively. Attention weights are calculated for the query and key vectors, and the value vector is updated based on the attention weights to obtain the updated latent vector. A cross-attention mechanism is introduced to enhance the semantic consistency and fidelity of the generated images.

[0083] See also Figure 7 , Figure 7 A specific implementation of step S523 is shown, which is described in detail as follows:

[0084] S5231: Calculate the attention weights of the query vector and the key vector using a scaled dot product attention calculation method.

[0085] S5232: Perform weighted summation on the value vector using the attention weight to obtain the updated potential vector.

[0086] Specifically, the attention weight is calculated using a scaled dot product attention method. The core of this method is to evaluate the similarity between the query vector and the key vector by taking the dot product between them. However, to prevent the dot product result from being too large, causing the Softmax function to enter the gradient vanishing region, the embodiment of the application will scale the dot product result to obtain the attention weight. The value vector is then weighted and summed using the attention weight to obtain the updated potential vector.

[0087] S6: Decode the target potential vector to generate a target ground panoramic image.

[0088] Specifically, the above steps have generated a target latent vector corresponding to the ground panoramic image. This embodiment of the application uses the decoder of the VQGAN model to decode the target latent vector in the latent space back into the image space, generating a target ground panoramic image. This target ground panoramic image is semantically consistent with the input satellite image, while also having rich details and accurate structure.

[0089] In an embodiment of the present application, a set of image combinations is received, wherein the image combination includes a satellite image and a ground panoramic image corresponding to the satellite image; shared features of the satellite image and the ground panoramic image are extracted using a cross-view image joint encoder; the satellite image is polar-transformed and the polar-transformed satellite image and the ground panoramic image are encoded into a latent space to obtain an initial satellite image latent vector and a ground panoramic image latent vector; a Brownian bridge forward process is performed in the latent space based on the initial satellite image latent vector and the ground panoramic image latent vector to gradually add noise to the ground panoramic image latent vector to obtain a satellite image latent vector; a Brownian bridge inverse process is performed based on the satellite image latent vector and the shared features to gradually remove noise from the satellite image latent vector to generate a target latent vector; and the target latent vector is decoded to generate a target ground panoramic image. In this embodiment of the present invention, the shared features of the satellite image and the ground panoramic image are extracted using a cross-view image joint encoder, and the Brownian bridge forward process and the Brownian bridge inverse process are performed. Simultaneously, the shared features are injected into the Brownian bridge inverse process, which facilitates the generation of a ground panoramic image with rich details and precise structure from the satellite image, thereby improving the image fidelity and semantic consistency. In addition, the embodiment of the present application also performs the Brownian bridge forward process and the Brownian bridge reverse process in the latent space, reducing the model training and inference costs and improving the efficiency of image conversion.

[0090] The embodiment of the present application applies the Brownian bridge diffusion model to model the conversion process from satellite images to ground panoramic images as a random Brownian bridge process, and directly learns the conversion between the two domains through a bidirectional diffusion process, thereby achieving the generation of detailed and structurally precise ground panoramic images directly from satellite images, significantly improving the realism and semantic consistency of the image. The embodiment of the present application uses the characteristic of gradual generation through the diffusion model to make the generated image have fewer artifacts and blur. The embodiment of the present application extracts the shared information of the cross-view image pairs as a generation clue through the cross-view image joint encoder, effectively improving the semantic consistency of the generated ground panoramic image and the corresponding satellite image.

[0091] Please refer to Figure 8 , as a response to the above Figure 1 The present application provides an embodiment of an image generation device based on a Brownian bridge diffusion model. Figure 1 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.

[0092] like Figure 8 As shown, the image generation device based on the Brownian bridge diffusion model of this embodiment includes: an image receiving unit 71, a feature extraction unit 72, a coordinate transformation unit 73, a noise adding unit 74, a noise removal unit 75 and an image generation unit 76, wherein:

[0093] An image receiving unit 71 is configured to receive a set of image combinations, wherein the image combination includes a satellite image and a ground panoramic image corresponding to the satellite image;

[0094] a feature extraction unit 72, configured to extract shared features between the satellite image and the ground panoramic image through a cross-view image joint encoder;

[0095] A coordinate transformation unit 73 is configured to perform polar coordinate transformation on the satellite image, and encode the polar coordinate transformed satellite image and the ground panoramic image into a latent space to obtain an initial satellite image latent vector and a ground panoramic image latent vector;

[0096] a noise adding unit 74 configured to perform a Brownian bridge forward process in the latent space based on the initial satellite image latent vector and the ground panoramic image latent vector, so as to gradually add noise to the ground panoramic image latent vector to obtain a satellite image latent vector;

[0097] a noise removal unit 75 for performing a Brownian bridge inverse process based on the satellite image latent vector and the shared feature to gradually remove noise from the satellite image latent vector and generate a target latent vector;

[0098] The image generation unit 76 is configured to decode the target potential vector to generate a target ground panoramic image.

[0099] Furthermore, the feature extraction unit 72 includes:

[0100] an initial feature extraction unit, configured to extract features of the satellite image and the ground panoramic image respectively, to obtain initial satellite image features and initial ground image features;

[0101] an average pooling unit, configured to perform average pooling on the initial satellite image features and the initial ground image features, respectively, to obtain basic satellite image features and basic ground image features;

[0102] The shared feature generation unit is used to embed the basic satellite image features and the basic ground image features into a shared feature space through a pre-trained encoder to obtain the shared features.

[0103] Furthermore, the shared feature generation unit further includes:

[0104] An image acquisition unit, configured to acquire an original satellite image and an original ground panoramic image corresponding to the original satellite image;

[0105] a compressed feature vector generating unit, configured to use a convolutional neural network as an encoder, and perform feature extraction and feature pooling on the original satellite image and the original ground panoramic image based on the encoder to obtain a compressed feature vector;

[0106] A training unit is used to use the InfoNCE loss function as a model loss function, and adopt a GPS coordinate sampling method and a dynamic similarity sampling method to train the encoder based on the compressed feature vector to obtain the pre-trained encoder.

[0107] Furthermore, the noise adding unit 74 includes:

[0108] a mapping path construction unit, configured to construct, in the latent space, a mapping path from the ground panoramic image latent vector to the initial satellite image latent vector;

[0109] a label recording unit, configured to gradually add noise to the ground panoramic image potential vector along the mapping path by forward diffusion, and record the noise as a true noise label;

[0110] The noise stopping adding unit is used to stop adding noise if the current ground panoramic image potential vector reaches the initial satellite image potential vector, and use the current ground panoramic image potential vector as the satellite image potential vector.

[0111] Furthermore, the noise removal unit 75 includes:

[0112] a reverse mapping path construction unit, configured to construct a reverse mapping path based on the satellite image potential vector;

[0113] an attention set mechanism unit, configured to gradually remove noise from the satellite image latent vector based on the reverse mapping path, and update the current satellite image latent vector by combining the attention mechanism and the shared features during each step of removing noise from the satellite image latent vector;

[0114] A target latent vector generating unit is configured to generate the target latent vector when the reverse mapping path is completed.

[0115] Furthermore, the target potential vector generation unit includes:

[0116] a noise step-by-step removal unit, configured to step-by-step remove noise from the satellite image latent vector based on the reverse mapping path;

[0117] a vector validation unit, configured to use the current satellite image latent vector as a query vector and the shared features as a key vector and a value vector, respectively, in each step of removing noise from the satellite image latent vector;

[0118] A vector updating unit is configured to calculate attention weights of the query vector and the key vector, and update the value vector based on the attention weights to obtain an updated latent vector.

[0119] Furthermore, the vector updating unit includes:

[0120] an attention weight calculation unit, configured to calculate the attention weights of the query vector and the key vector using a scaled dot product attention calculation method;

[0121] A weighted summation unit is configured to perform weighted summation on the value vector using the attention weight to obtain the updated potential vector.

[0122] To solve the above technical problems, the present application also provides a computer device. Figure 9 , Figure 9 This is a basic structural block diagram of the computer device in this embodiment.

[0123] The computer device 8 includes a memory 81, a processor 82, and a network interface 83 that are interconnected through a system bus. It should be noted that Figure 9Only a computer device 8 having three components, memory 81, processor 82, and network interface 83, is shown. However, it should be understood that it is not required to implement all of the components shown, and more or fewer components may be implemented instead. Those skilled in the art will understand that a computer device herein is a device that can automatically perform numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes but is not limited to a microprocessor, an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), a digital signal processor (DSP), an embedded device, etc.

[0124] Computer devices can be desktop computers, laptops, PDAs, cloud servers, etc. Computer devices can interact with users through keyboards, mice, remote controls, touchpads, or voice-activated devices.

[0125] The memory 81 includes at least one type of readable storage medium, including flash memory, a hard disk, a multimedia card, a card-type memory (e.g., SD or DX memory), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic storage, a magnetic disk, an optical disk, etc. In some embodiments, the memory 81 may be an internal storage unit of the computer device 8, such as the hard disk or memory of the computer device 8. In other embodiments, the memory 81 may also be an external storage device of the computer device 8, such as a plug-in hard disk equipped on the computer device 8, a smart memory card (SMC), a secure digital (SD) card, a flash memory card, etc. Of course, the memory 81 may also include both the internal storage unit of the computer device 8 and its external storage device. In this embodiment, the memory 81 is generally used to store the operating system and various application software installed on the computer device 8, such as the program code of the image generation method based on the Brownian bridge diffusion model. In addition, the memory 81 can also be used to temporarily store various types of data that have been output or are to be output.

[0126] In some embodiments, the processor 82 may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chip. The processor 82 is generally used to control the overall operation of the computer device 8. In this embodiment, the processor 82 is used to execute program code stored in the memory 81 or process data, such as executing the program code of the above-mentioned Brownian bridge diffusion model-based image generation method to implement various embodiments of the Brownian bridge diffusion model-based image generation method.

[0127] The network interface 83 may include a wireless network interface or a wired network interface. The network interface 83 is generally used to establish a communication connection between the computer device 8 and other electronic devices.

[0128] The present application also provides another embodiment, namely, providing a computer-readable storage medium, which stores a computer program, and the computer program can be executed by at least one processor to enable the at least one processor to perform the steps of the above-mentioned image generation method based on the Brownian bridge diffusion model.

[0129] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus the necessary general hardware platform, and of course can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for enabling a terminal device (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods of each embodiment of the present application.

[0130] Obviously, the embodiments described above are only some of the embodiments of the present application, rather than all of the embodiments. The preferred embodiments of the present application are given in the accompanying drawings, but they do not limit the patent scope of the present application. The present application can be implemented in many different forms. On the contrary, the purpose of providing these embodiments is to make the understanding of the disclosure of the present application more thorough and comprehensive. Although the present application has been described in detail with reference to the aforementioned embodiments, for those skilled in the art, it is still possible to modify the technical solutions described in the aforementioned specific embodiments, or to make equivalent replacements for some of the technical features therein. Any equivalent structure made using the contents of the present application specification and the accompanying drawings, directly or indirectly used in other related technical fields, is also within the scope of patent protection of the present application.

Claims

1. An image generation method based on a Brownian bridge diffusion model, characterized in that: include: receiving a set of image combinations, wherein the image combinations include satellite images and ground panoramic images corresponding to the satellite images; extracting shared features of the satellite image and the ground panoramic image through a cross-view image joint encoder; Performing polar coordinate transformation on the satellite image, and encoding the polar coordinate transformed satellite image and the ground panoramic image into a latent space to obtain an initial satellite image latent vector and a ground panoramic image latent vector; In the latent space, performing a Brownian bridge forward process based on the initial satellite image latent vector and the ground panoramic image latent vector to gradually add noise to the ground panoramic image latent vector to obtain a satellite image latent vector; performing a Brownian bridge inverse process based on the satellite image latent vector and the shared features to gradually remove noise from the satellite image latent vector and generate a target latent vector; The target potential vector is decoded to generate a target ground panoramic image.

2. The image generation method based on the Brownian bridge diffusion model according to claim 1, characterized in that: The extracting shared features of the satellite image and the ground panoramic image by using a cross-view image joint encoder includes: Extracting features of the satellite image and the ground panoramic image respectively to obtain initial satellite image features and initial ground image features; performing average pooling on the initial satellite image features and the initial ground image features respectively to obtain basic satellite image features and basic ground image features; The basic satellite image features and the basic ground image features are embedded into a shared feature space by a pre-trained encoder to obtain the shared features.

3. The image generation method based on the Brownian bridge diffusion model according to claim 2, characterized in that: Before embedding the basic satellite image features and the basic ground image features into a shared feature space using a pre-trained encoder to obtain the shared features, the method further includes: Acquiring original satellite images and original ground panoramic images corresponding to the original satellite images; Using a convolutional neural network as an encoder, and performing feature extraction and feature pooling on the original satellite image and the original ground panoramic image based on the encoder to obtain a compressed feature vector; The InfoNCE loss function is used as the model loss function, and the GPS coordinate sampling method and the dynamic similarity sampling method are used to train the encoder based on the compressed feature vector to obtain the pre-trained encoder.

4. The image generation method based on the Brownian bridge diffusion model according to claim 1, characterized in that: The step of performing a Brownian bridge forward process in the latent space based on the initial satellite image latent vector and the ground panoramic image latent vector to gradually add noise to the ground panoramic image latent vector to obtain a satellite image latent vector includes: In the latent space, constructing a mapping path from the ground panoramic image latent vector to the initial satellite image latent vector; gradually adding noise to the ground panoramic image potential vector along the mapping path by forward diffusion, and recording the noise as a true noise label; If the current ground panoramic image potential vector reaches the initial satellite image potential vector, the noise addition is stopped, and the current ground panoramic image potential vector is used as the satellite image potential vector.

5. The image generation method based on the Brownian bridge diffusion model according to any one of claims 1 to 4, characterized in that: The performing of the Brownian bridge inverse process based on the satellite image latent vector and the shared feature to gradually remove noise in the satellite image latent vector to generate a target latent vector includes: constructing a reverse mapping path based on the satellite image latent vector; gradually removing noise from the satellite image latent vector based on the reverse mapping path, and updating the current satellite image latent vector in combination with the attention mechanism and the shared features during each step of removing noise from the satellite image latent vector; When the reverse mapping path is completed, the target latent vector is generated.

6. The image generation method based on the Brownian bridge diffusion model according to claim 5, characterized in that: The step of gradually removing noise from the satellite image latent vector based on the reverse mapping path, and updating the current satellite image latent vector in combination with the attention mechanism and the shared feature during each step of removing noise from the satellite image latent vector, includes: gradually removing noise from the satellite image latent vector based on the reverse mapping path; In each step of removing noise from the satellite image latent vector, the current satellite image latent vector is used as a query vector, and the shared features are used as a key vector and a value vector respectively; Attention weights of the query vector and the key vector are calculated, and the value vector is updated based on the attention weights to obtain an updated latent vector.

7. The image generation method based on the Brownian bridge diffusion model according to claim 6, characterized in that: The calculating the attention weights of the query vector and the key vector, and updating the value vector based on the attention weights to obtain an updated latent vector, includes: Calculating the attention weights of the query vector and the key vector using a scaled dot product attention calculation method; The value vector is weightedly summed using the attention weight to obtain the updated potential vector.

8. An image generation device based on a Brownian bridge diffusion model, characterized in that: include: An image receiving unit, configured to receive a set of image combinations, wherein the image combination includes a satellite image and a ground panoramic image corresponding to the satellite image; a feature extraction unit, configured to extract shared features of the satellite image and the ground panoramic image through a cross-view image joint encoder; a coordinate transformation unit, configured to perform polar coordinate transformation on the satellite image, and encode the polar coordinate transformed satellite image and the ground panoramic image into a latent space to obtain an initial satellite image latent vector and a ground panoramic image latent vector; a noise adding unit, configured to perform a Brownian bridge forward process in the latent space based on the initial satellite image latent vector and the ground panoramic image latent vector, so as to gradually add noise to the ground panoramic image latent vector to obtain a satellite image latent vector; a noise removal unit, configured to perform a Brownian bridge inverse process based on the satellite image latent vector and the shared feature to gradually remove noise from the satellite image latent vector and generate a target latent vector; The image generation unit is used to decode the target potential vector to generate a target ground panoramic image.

9. A computer device, characterized in that: The method comprises a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, the image generation method based on the Brownian bridge diffusion model according to any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the image generation method based on the Brownian bridge diffusion model according to any one of claims 1 to 7 is implemented.

Images