Image generation method and device based on Brown bridge diffusion model, equipment and medium

Abstract

The invention relates to an image generation method and device based on a Brown bridge diffusion model, equipment and a medium, and the method comprises the steps: receiving a group of image combinations which comprise a satellite image and a ground panoramic image corresponding to the satellite image; extracting sharing features of the satellite image and the ground panoramic image through a cross-view-angle image joint encoder; performing polar coordinate transformation on the satellite image, and encoding the satellite image and the ground panoramic image after polar coordinate transformation into a potential space to obtain an initial satellite image potential vector and a ground panoramic image potential vector; in the potential space, gradually adding noise to the potential vector of the ground panoramic image to obtain a potential vector of a satellite image; noise in the satellite image potential vector is gradually removed based on the satellite image potential vector and the shared features, and a target potential vector is generated; and decoding the target potential vector to generate a target ground panoramic image. According to the invention, the efficiency and quality of converting the satellite image into the ground panoramic image are improved.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to an image generation method, device, equipment and medium based on a Brownian bridge diffusion model. Background Art

[0002] With the rapid development of remote sensing technology, satellite images have become an important tool for us to understand and monitor the earth. However, directly acquired satellite images are often limited by multiple factors such as resolution, viewing angle, and lighting conditions, making it difficult to use them directly as ground panoramic images. Therefore, the existing method of generating ground panoramic images from satellite images has become an important research field.

[0003] Existing satellite image to ground panoramic image generation methods often rely on additional ground image information (such as semantic segmentation map, depth map) for training or reasoning, which is not suitable for real-world application scenarios. In addition, existing methods have the problem of unstable training, which requires careful adjustment of parameters and complex network structure design to maintain the balance of the training process. Therefore, the efficiency and quality of satellite image conversion to ground panoramic images in existing methods are low. Summary of the invention

[0004] The purpose of the embodiments of the present application is to propose an image generation method, device, equipment and medium based on the Brownian bridge diffusion model to achieve efficiency and quality in converting satellite images into ground panoramic images.

[0005] In order to solve the above technical problems, the embodiment of the present application provides an image generation method based on the Brownian bridge diffusion model, including:

[0006] Receive a set of image combinations, wherein the image combination includes a satellite image and a ground panoramic image corresponding to the satellite image;

[0007] Extracting shared features of the satellite image and the ground panoramic image through a cross-view image joint encoder;

[0008] Performing polar coordinate transformation on the satellite image, and encoding the polar coordinate transformed satellite image and the ground panoramic image into a latent space to obtain an initial satellite image latent vector and a ground panoramic image latent vector;

[0009] In the latent space, a Brownian bridge forward process is performed based on the initial satellite image latent vector and the ground panoramic image latent vector to gradually add noise to the ground panoramic image latent vector to obtain a satellite image latent vector;

[0010] Performing a Brownian bridge inverse process based on the satellite image latent vector and the shared feature to gradually remove noise in the satellite image latent vector to generate a target latent vector;

[0011] The target potential vector is decoded to generate a target ground panoramic image.

[0012] In order to solve the above technical problems, the embodiment of the present application provides an image generation device based on a Brownian bridge diffusion model, comprising:

[0013] An image receiving unit, configured to receive a group of image combinations, wherein the image combination includes a satellite image and a ground panoramic image corresponding to the satellite image;

[0014] A feature extraction unit, configured to extract shared features of the satellite image and the ground panoramic image through a cross-view image joint encoder;

[0015] A coordinate transformation unit, used for performing polar coordinate transformation on the satellite image, and encoding the polar coordinate transformed satellite image and the ground panoramic image into a latent space to obtain an initial satellite image latent vector and a ground panoramic image latent vector;

[0016] a noise adding unit, configured to perform a Brownian bridge forward process in the latent space based on the initial satellite image latent vector and the ground panoramic image latent vector, so as to gradually add noise to the ground panoramic image latent vector to obtain a satellite image latent vector;

[0017] a noise removal unit, configured to perform a Brownian bridge inverse process based on the satellite image latent vector and the shared feature to gradually remove noise in the satellite image latent vector and generate a target latent vector;

[0018] The image generation unit is used to decode the target potential vector to generate a target ground panoramic image.

[0019] In order to solve the above technical problems, a technical solution adopted by the present invention is: to provide a computer device, including one or more processors; a memory for storing one or more programs, so that the one or more processors can implement any one of the above-mentioned image generation methods based on the Brownian bridge diffusion model.

[0020] To solve the above technical problems, a technical solution adopted by the present invention is: a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, any one of the above-mentioned image generation methods based on the Brownian bridge diffusion model is implemented.

[0021] The embodiment of the present invention provides an image generation method, device, equipment and medium based on the Brownian bridge diffusion model. The method includes: receiving a group of image combinations, wherein the image combination includes a satellite image and a ground panoramic image corresponding to the satellite image; extracting the shared features of the satellite image and the ground panoramic image through a cross-view image joint encoder; performing polar coordinate transformation on the satellite image, and encoding the polar coordinate transformed satellite image and the ground panoramic image into a latent space to obtain an initial satellite image latent vector and a ground panoramic image latent vector; in the latent space, performing a Brownian bridge forward process based on the initial satellite image latent vector and the ground panoramic image latent vector to gradually add noise to the ground panoramic image latent vector to obtain a satellite image latent vector; performing a Brownian bridge reverse process based on the satellite image latent vector and the shared features to gradually remove the noise in the satellite image latent vector to generate a target latent vector; decoding the target latent vector to generate a target ground panoramic image. The embodiment of the present invention extracts the shared features of satellite images and ground panoramic images through a cross-view image joint encoder, and executes the Brownian bridge forward process and the Brownian bridge reverse process, and injects the shared features into the Brownian bridge reverse process, which is conducive to generating detailed and structurally precise ground panoramic images from satellite images, and improves the image fidelity and semantic consistency. In addition, the embodiment of the present application also executes the Brownian bridge forward process and the Brownian bridge reverse process in the latent space, reducing the model training and reasoning costs, and improving the efficiency of image conversion. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In order to more clearly illustrate the scheme in the present application, a brief introduction is given below to the drawings required for use in the description of the embodiments of the present application. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0023] Figure 1 This is a flowchart of an implementation of the image generation method based on the Brownian bridge diffusion model provided in an embodiment of the present application;

[0024] Figure 2 is a flowchart for implementing the first sub-process in the image generation method based on the Brownian bridge diffusion model provided in an embodiment of the present application;

[0025] Figure 3 is a flowchart of an implementation process of an image generation method based on a Brownian bridge diffusion model provided by another embodiment of the present application;

[0026] Figure 4 is a flowchart for implementing the second sub-process in the image generation method based on the Brownian bridge diffusion model provided in an embodiment of the present application;

[0027] Figure 5 is a flowchart for implementing the third sub-process in the image generation method based on the Brownian bridge diffusion model provided in an embodiment of the present application;

[0028] Figure 6 is a flowchart for implementing the fourth sub-process in the image generation method based on the Brownian bridge diffusion model provided in an embodiment of the present application;

[0029] Figure 7 is a flowchart for implementing the fifth sub-process in the image generation method based on the Brownian bridge diffusion model provided in an embodiment of the present application;

[0030] Figure 8 is a schematic diagram of an image generation device based on a Brownian bridge diffusion model provided in an embodiment of the present application;

[0031] Fig. 9 It is a schematic diagram of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0032] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by technicians in the technical field of the present application; the terms used in the specification of the application herein are only for the purpose of describing specific embodiments and are not intended to limit the present application; the terms "including" and "having" and any variations thereof in the specification and claims of the present application and the above-mentioned drawings are intended to cover non-exclusive inclusions. The terms "first", "second", etc. in the specification and claims of the present application or the above-mentioned drawings are used to distinguish different objects, not to describe a specific order.

[0033] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0034] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the accompanying drawings.

[0035] The present invention is described in detail below with reference to the accompanying drawings and embodiments.

[0036] It should be noted that the image generation method based on the Brownian bridge diffusion model provided in the embodiment of the present application is generally executed by a server, and accordingly, the image generation device based on the Brownian bridge diffusion model is generally configured in the server.

[0037] See also Figure 1 , Figure 1 A specific implementation of an image generation method based on a Brownian bridge diffusion model is shown.

[0038] It should be noted that if there are substantially the same results, the method of the present invention is not limited to Figure 1 The process sequence shown is limited to the following steps:

[0039] S1: Receive a group of image combinations, wherein the image combination includes a satellite image and a ground panoramic image corresponding to the satellite image.

[0040] The embodiment of the present application provides an image generation method based on the Brownian bridge diffusion model, which is used to convert satellite images into ground panoramic images, thereby improving the efficiency and quality of image conversion. In addition, the embodiment of the present application has broad application prospects in multiple fields, including but not limited to wide-area virtual environment modeling, augmented reality content creation, 3D game development, advanced simulation training, and cross-view image matching. In the embodiment of the present application, a set of image combinations is obtained, which is a satellite image and a ground panoramic image corresponding to the satellite image.

[0041] S2: Extracting shared features of the satellite image and the ground panoramic image through a cross-view image joint encoder.

[0042] Specifically, a cross-view image joint encoder is used to extract shared features of satellite images and ground panoramic images, which are embedded in a shared feature space to provide feature information for the subsequent diffusion model. The cross-view image joint encoder is trained using contrastive learning to generate an encoder for extracting shared features of cross-view images.

[0043] See also Figure 2 , Figure 2 A specific implementation of step S2 is shown, which is described in detail as follows:

[0044] S21: extracting features of the satellite image and the ground panoramic image respectively to obtain initial satellite image features and initial ground image features.

[0045] S22: performing average pooling on the initial satellite image features and the initial ground image features respectively to obtain basic satellite image features and basic ground image features.

[0046] S23: embedding the basic satellite image features and the basic ground image features into a shared feature space through a pre-trained encoder to obtain the shared features.

[0047] Specifically, a convolutional neural network is used as an encoder, and the convolutional neural network may be ConvNeXt-B. The encoder is a pre-trained encoder, and the features of the satellite image and the ground panoramic image are respectively extracted by the encoder to obtain the initial satellite image features and the initial ground image features. Among them, the initial satellite image features and the initial ground image features can capture important information in the satellite image and the ground panoramic image, such as shape, texture, color, etc. Then, in order to reduce the dimension of the feature map while retaining the most important feature information, it is necessary to average pool the initial satellite image features and the initial ground image features respectively to obtain the basic satellite image features and the basic ground image features. Finally, the basic satellite image features and the basic ground image features are embedded in the shared feature space to obtain the shared features.

[0048] See also Figure 3 , Figure 3 A specific implementation method before step S23 is shown, which is described in detail as follows:

[0049] S23A: Acquire an original satellite image and an original ground panoramic image corresponding to the original satellite image.

[0050] S23B: Using a convolutional neural network as an encoder, and performing feature extraction and feature pooling on the original satellite image and the original ground panoramic image based on the encoder to obtain a compressed feature vector.

[0051] S23C: Using the InfoNCE loss function as the model loss function, and adopting the GPS coordinate sampling method and the dynamic similarity sampling method to train the encoder based on the compressed feature vector to obtain the pre-trained encoder.

[0052] Specifically, a set of original satellite images and original ground panoramic image pairs are collected, and the original features are obtained through a cross-view image joint encoder, and finally a compressed feature vector is obtained through average pooling in the spatial dimension. Among them, the compressed feature vector is used to calculate the loss and optimize the encoder parameters. The InfoNCE loss function is used as the model loss function, and the encoder is trained based on the compressed feature vector using the GPS coordinate sampling method and the dynamic similarity sampling method. The similarity between the query image and the reference image is calculated by dot product, and the encoder is optimized so that the similarity of the positive sample pair is higher than that of the negative sample pair. When the training is completed, the pre-trained encoder in the embodiment of the present application is obtained.

[0053] In the embodiment of the present application, the Information Noise-Contrast Estimation loss function is used as the model loss function, which optimizes the model by comparing the positive and negative sample pairs in the small batch. In the early stage of the training process, considering that the discriminative power of the model is not yet mature, the embodiment of the present application adopts a sampling method based on GPS coordinates to select images with similar geographical locations as negative samples. This method not only enhances the sensitivity of the model to geographical features, but also improves its cognitive ability to geographical proximity. As the training deepens, the discriminative power of the model gradually increases. The embodiment of the present application introduces a dynamic similarity sampling method, which dynamically selects images that are highly similar to the query image in the feature space as negative samples during the training process, and further mines samples that are difficult to distinguish. The implementation of this method enables the model to focus more on learning to distinguish subtle visual and geographical features in the later stage of training. By focusing on the GPS sampling method in the early stage of training and gradually transitioning to the dynamic similarity sampling strategy in the later stage, not only the mining process of difficult samples is optimized, but also the encoder's ability to distinguish cross-view image features is significantly improved.

[0054] Among them, the InfoNCE loss function is a commonly used contrast loss function, which is used to train the model to distinguish between positive samples and negative samples. In the task of image matching or feature embedding, InfoNCE can help the model learn more discriminative feature representations. GPS coordinate sampling method: Since there is a correspondence between the geographical locations between satellite images and ground panoramic images, GPS coordinates can be used to sample positive sample pairs (i.e., image pairs with similar geographical locations) and negative sample pairs (i.e., image pairs with farther geographical locations). This method helps the model learn features related to geographical location. Dynamic similarity sampling method: Dynamically sample negative samples based on the similarity between feature vectors. Specifically, for each positive sample pair, other images with higher similarity to the positive sample feature vector can be selected from the data set as negative samples to increase the difficulty of training and the generalization ability of the model.

[0055] Training process: During the training process, the encoder continuously adjusts its parameters based on the feedback of the InfoNCE loss function to minimize the feature distance between positive sample pairs and maximize the feature distance between negative sample pairs. In this way, the encoder can learn a more effective feature representation and embed the features of satellite images and ground panoramic images into the same shared feature space.

[0056] After training, the embodiment of the present application obtains a pre-trained encoder, which can convert the original satellite image and the ground panoramic image into a compressed feature vector in the same feature space.

[0057] S3: performing polar coordinate transformation on the satellite image, and encoding the polar coordinate transformed satellite image and the ground panoramic image into a latent space to obtain an initial satellite image latent vector and a ground panoramic image latent vector.

[0058] Specifically, the polar coordinate transformation of the satellite image is used to partially bridge the view difference between the satellite image and the ground panoramic image, making the two closer in geometric structure and facilitating subsequent processing. The polar coordinate transformed satellite image and the ground panoramic image are then encoded into the latent space to obtain the initial satellite image latent vector and the ground panoramic image latent vector.

[0059] Since training a diffusion model in pixel space will bring high training costs and video memory consumption. Therefore, the embodiment of the present application uses the encoder of the VQGAN (Vector Quantitative Generative Adversarial Network) model to encode the satellite image and the ground panoramic image after polar coordinate transformation into the latent space to obtain a latent vector. Among them, the ground panoramic image is a real ground panoramic image. The latent space can be regarded as a compression of the pixel space, for example: an image of 256×256×3 can be encoded into 64×64×3. The VQGAN model encoding and decoding process can well ensure the authenticity of the image, which is conducive to improving the quality of image conversion. Encoding the image into the latent space reduces the data dimension, reduces the training cost and video memory consumption, and is conducive to improving the efficiency of converting from satellite images to panoramic images.

[0060] S4: In the latent space, a Brownian bridge forward process is performed based on the initial satellite image latent vector and the ground panoramic image latent vector to gradually add noise to the ground panoramic image latent vector to obtain a satellite image latent vector.

[0061] Specifically, starting from the real ground panoramic image latent vector, a fixed mapping from the ground panoramic image domain to the satellite image domain is constructed. In the latent space, noise is gradually added to the ground panoramic image latent vector through the forward diffusion process to simulate the conversion from the ground panoramic image latent vector to the satellite image latent vector. If the current ground panoramic image latent vector reaches the satellite image latent vector, no noise is added and the ground panoramic image latent vector at this time is used as the satellite image latent vector. In this process, the real noise label can be calculated for the subsequent Brownian bridge reverse process to guide the training process.

[0062] The forward process of the Brownian bridge is to simulate a process of transformation to a target domain (such as a satellite image domain) in the latent space from a given starting point (such as the latent vector of a real ground panoramic image) by gradually adding noise or other forms of transformation. This process aims to simulate the natural transition of an image from one representation to another in the latent space, while maintaining or emphasizing the common features or structures between the two image domains.

[0063] See also Figure 4 , Figure 4 A specific implementation of step S4 is shown, which is described in detail as follows:

[0064] S41: In the latent space, construct a mapping path from the ground panoramic image latent vector to the initial satellite image latent vector.

[0065] S42: gradually adding noise to the ground panoramic image potential vector along the mapping path in a forward diffusion manner, and recording the noise as a true noise label.

[0066] S43: If the current ground panoramic image potential vector reaches the initial satellite image potential vector, stop adding noise and use the current ground panoramic image potential vector as the satellite image potential vector.

[0067] Specifically, in the latent space, a mapping path is constructed from the ground panoramic image latent vector to the initial satellite image latent vector; noise is gradually added to the ground panoramic image latent vector along the mapping path. This noise addition process is controllable and follows a certain noise distribution (such as Gaussian noise), and the intensity and type of noise can gradually change as the conversion process proceeds. Noise addition is a key step in simulating image quality degradation. By gradually adding noise, the conversion process from high quality (ground panoramic image) to low quality (satellite image) can be simulated. After adding noise at each step, a corresponding true noise label is generated. The true noise label records the specific information of the noise added to the latent vector in the current step (such as the type, intensity, distribution, etc. of the noise). The true noise labels are crucial for the subsequent Brownian bridge reverse process because they will serve as true noise information to guide noise removal in the reverse process. Then repeat the noise addition and noise label generation steps until the current ground panoramic image latent vector reaches the satellite image latent vector, stop adding noise, and use the current ground panoramic image latent vector as the satellite image latent vector.

[0068] The iterative process of adding noise requires multiple iterations, and the specific number of iterations depends on the complexity of the mapping relationship and the strategy of adding noise.

[0069] In the embodiment of the present application, the Langqiao forward process realizes the mapping from the real ground panoramic image latent vector to the satellite image latent vector, and generates the real noise label for guiding the subsequent reverse process. This process is an important part of the image generation and conversion task, which helps to simulate the gradual degradation process of image quality in the latent space.

[0070] S5: Performing a Brownian bridge reverse process based on the satellite image latent vector and the shared feature to gradually remove noise in the satellite image latent vector and generate a target latent vector.

[0071] Specifically, the Brownian bridge reverse process is performed based on the satellite image latent vector and the shared features, so as to realize the satellite image latent vector as the starting point, gradually predict and remove the noise in the satellite image latent vector, gradually generate the ground panoramic image latent vector, and obtain the target latent vector. In an embodiment of the present application, an attention mechanism is introduced in the execution of the Brownian bridge reverse process, and the shared features of the cross-view images are injected through the cross-attention mechanism. The cross-attention mechanism uses the latent vector in the Brownian bridge reverse process as the query vector, and the shared feature vector extracted by the cross-view joint encoder as the key vector and value vector, and uses scaled dot product attention for calculation. Through the introduction of the cross-attention mechanism, the latent vector in the generation process can extract the high-dimensional cross-view image information present in the shared features, thereby improving the semantic consistency and fidelity of the generated ground panoramic images and satellite images.

[0072] Among them, the Brownian bridge reverse process refers to the process of gradually approximating and generating the latent vector of the target domain image (such as a ground panoramic image) by gradually predicting and removing the noise added in the forward process from the satellite image potential vector through a series of carefully designed transformations or network layers. This process usually corresponds to the Brownian bridge forward process, where the forward process simulates the loss of image information by adding noise, while the reverse process attempts to restore this information.

[0073] See also Figure 5 , Figure 5 A specific implementation of step S5 is shown, which is described in detail as follows:

[0074] S51: Constructing a reverse mapping path based on the satellite image potential vector.

[0075] S52: gradually removing the noise in the satellite image latent vector based on the reverse mapping path, and updating the current satellite image latent vector in combination with the attention mechanism and the shared feature in each step of removing the noise in the satellite image latent vector.

[0076] S53: When the reverse mapping path is executed, the target potential vector is generated.

[0077] Specifically, in the latent space, a reverse mapping path is constructed that is opposite to the forward process of the Brownian bridge. The reverse mapping path connects the satellite image latent vector and the ground panoramic image latent vector and specifies the path for noise removal. Starting from the satellite image latent vector, the previously added noise is gradually removed along the reverse mapping path. The noise removal process is controllable and follows the steps opposite to the noise addition process. When removing noise at each step, the attention mechanism is combined to predict and remove part of the noise based on the state of the current latent vector and the information of the ground panoramic image latent vector. The cross-attention mechanism is introduced to enhance the semantic consistency and fidelity of the generated image. The cross-attention mechanism allows the latent vector in the generation process to pay attention to and utilize the shared features extracted by the cross-view joint encoder. These shared features contain the key information between the satellite image and the ground panoramic image. Through the cross-attention mechanism, the latent vector in the generation process can make full use of this information to generate more accurate and realistic ground panoramic images.

[0078] See also Figure 6 , Figure 6 A specific implementation of step S52 is shown, which is described in detail as follows:

[0079] S521: gradually removing noise in the satellite image latent vector based on the reverse mapping path.

[0080] S522: In each step of removing noise from the satellite image latent vector, the current satellite image latent vector is used as a query vector, and the shared features are used as a key vector and a value vector respectively.

[0081] S523: Calculate the attention weights of the query vector and the key vector, and update the value vector based on the attention weights to obtain an updated potential vector.

[0082] Specifically, the noise in the satellite image latent vector is gradually removed based on the reverse mapping path; and in each step of removing the noise in the satellite image latent vector, the current satellite image latent vector is used as the query vector, and the shared features are used as the key vector and value vector respectively; the attention weights of the query vector and the key vector are calculated, and the value vector is updated based on the attention weights to obtain the updated latent vector. The semantic consistency and fidelity of the generated image are enhanced by introducing the cross-attention mechanism.

[0083] See also Figure 7 , Figure 7 A specific implementation of step S523 is shown, which is described in detail as follows:

[0084] S5231: Calculate the attention weights of the query vector and the key vector using a scaled dot product attention calculation method.

[0085] S5232: Perform weighted summation on the value vector using the attention weight to obtain the updated potential vector.

[0086] Specifically, the attention weight is calculated using the method of scaling dot product attention. The core of this method is to evaluate the similarity between the query vector and the key vector by the dot product between them, but in order to prevent the dot product result from being too large and causing the Softmax function to enter the gradient vanishing region, the embodiment of the present application will scale the dot product result to obtain the attention weight. Then the value vector is weighted and summed by the attention weight to obtain the updated potential vector.

[0087] S6: Decode the target potential vector to generate a target ground panoramic image.

[0088] Specifically, the above steps have generated a target latent vector corresponding to the ground panoramic image. The embodiment of the present application uses the decoder of the VQGAN model to decode the target latent vector in the latent space back to the image space to generate a target ground panoramic image. The target ground panoramic image is semantically consistent with the input satellite image, and has rich details and accurate structure.

[0089] In an embodiment of the present application, a group of image combinations is received, wherein the image combination includes a satellite image and a ground panoramic image corresponding to the satellite image; the shared features of the satellite image and the ground panoramic image are extracted through a cross-view image joint encoder; the satellite image is polar transformed, and the polar transformed satellite image and the ground panoramic image are encoded into a latent space to obtain an initial satellite image latent vector and a ground panoramic image latent vector; in the latent space, a Brownian bridge forward process is performed based on the initial satellite image latent vector and the ground panoramic image latent vector to gradually add noise to the ground panoramic image latent vector to obtain a satellite image latent vector; a Brownian bridge reverse process is performed based on the satellite image latent vector and the shared features to gradually remove the noise in the satellite image latent vector to generate a target latent vector; the target latent vector is decoded to generate a target ground panoramic image. The embodiment of the present invention extracts the shared features of the satellite image and the ground panoramic image through a cross-view image joint encoder, and performs a Brownian bridge forward process and a Brownian bridge reverse process, and injects the shared features into the Brownian bridge reverse process at the same time, which is conducive to generating a ground panoramic image with rich details and precise structure from the satellite image, and improves the image fidelity and semantic consistency. In addition, the embodiment of the present application also performs a Brownian bridge forward process and a Brownian bridge reverse process in the latent space, thereby reducing the model training and reasoning costs and improving the efficiency of image conversion.

[0090] The embodiment of the present application applies the Brownian bridge diffusion model to model the conversion process from satellite images to ground panoramic images as a random Brownian bridge process, and directly learns the conversion between the two domains through a bidirectional diffusion process, thereby achieving the generation of detailed and structurally precise ground panoramic images directly from satellite images, significantly improving the realism and semantic consistency of the image. The embodiment of the present application uses the characteristic of gradual generation through the diffusion model to make the generated image have fewer artifacts and blur. The embodiment of the present application extracts the shared information of the cross-view image pairs through the cross-view image joint encoder as a generation clue, effectively improving the semantic consistency of the generated ground panoramic image and the corresponding satellite image.

[0091] Please refer to Figure 8 , as a response to the above Figure 1 The present application provides an embodiment of an image generation device based on a Brownian bridge diffusion model. Figure 1 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.

[0092] like Figure 8 As shown, the image generation device based on the Brownian bridge diffusion model of this embodiment includes: an image receiving unit 71, a feature extraction unit 72, a coordinate transformation unit 73, a noise adding unit 74, a noise removal unit 75 and an image generation unit 76, wherein:

[0093] An image receiving unit 71 is configured to receive a set of image combinations, wherein the image combination includes a satellite image and a ground panoramic image corresponding to the satellite image;

[0094] A feature extraction unit 72, configured to extract shared features of the satellite image and the ground panoramic image through a cross-view image joint encoder;

[0095] A coordinate transformation unit 73 is used to perform polar coordinate transformation on the satellite image, and encode the polar coordinate transformed satellite image and the ground panoramic image into a latent space to obtain an initial satellite image latent vector and a ground panoramic image latent vector;

[0096] A noise adding unit 74 is used to perform a Brownian bridge forward process in the latent space based on the initial satellite image latent vector and the ground panoramic image latent vector, so as to gradually add noise to the ground panoramic image latent vector to obtain a satellite image latent vector;

[0097] A noise removal unit 75, configured to perform a Brownian bridge inverse process based on the satellite image latent vector and the shared feature to gradually remove noise in the satellite image latent vector and generate a target latent vector;

[0098] The image generation unit 76 is used to decode the target potential vector to generate a target ground panoramic image.

[0099] Furthermore, the feature extraction unit 72 includes:

[0100] An initial feature extraction unit, used to extract features of the satellite image and the ground panoramic image respectively, to obtain initial satellite image features and initial ground image features;

[0101] An average pooling unit, used to perform average pooling on the initial satellite image features and the initial ground image features respectively to obtain basic satellite image features and basic ground image features;

[0102] The shared feature generating unit is used to embed the basic satellite image features and the basic ground image features into a shared feature space through a pre-trained encoder to obtain the shared features.

[0103] Furthermore, the shared feature generation unit also includes:

[0104] An image acquisition unit, used for acquiring an original satellite image and an original ground panoramic image corresponding to the original satellite image;

[0105] A compressed feature vector generating unit, configured to use a convolutional neural network as an encoder, and perform feature extraction and feature pooling on the original satellite image and the original ground panoramic image based on the encoder to obtain a compressed feature vector;

[0106] A training unit is used to use the InfoNCE loss function as a model loss function, and adopt a GPS coordinate sampling method and a dynamic similarity sampling method to train the encoder based on the compressed feature vector to obtain the pre-trained encoder.

[0107] Further, the noise adding unit 74 includes:

[0108] A mapping path construction unit, used to construct a mapping path from the ground panoramic image potential vector to the initial satellite image potential vector in the potential space;

[0109] a label recording unit, configured to gradually add noise to the ground panoramic image potential vector along the mapping path by forward diffusion, and record the noise as a true noise label;

[0110] The noise stopping adding unit is used to stop adding noise if the current ground panoramic image potential vector reaches the initial satellite image potential vector, and use the current ground panoramic image potential vector as the satellite image potential vector.

[0111] Further, the noise removal unit 75 includes:

[0112] A reverse mapping path construction unit, used to construct a reverse mapping path based on the satellite image potential vector;

[0113] an attention set mechanism unit, configured to gradually remove noise from the satellite image latent vector based on the reverse mapping path, and update the current satellite image latent vector by combining the attention mechanism and the shared feature in the process of removing noise from the satellite image latent vector at each step;

[0114] A target latent vector generating unit is used to generate the target latent vector when the reverse mapping path is executed.

[0115] Furthermore, the target potential vector generation unit includes:

[0116] a noise step-by-step removal unit, configured to step-by-step remove noise from the satellite image latent vector based on the reverse mapping path;

[0117] A vector confirmation unit, configured to use the current satellite image latent vector as a query vector and the shared features as a key vector and a value vector respectively in each step of removing noise from the satellite image latent vector;

[0118] A vector updating unit is used to calculate the attention weights of the query vector and the key vector, and update the value vector based on the attention weights to obtain an updated potential vector.

[0119] Further, the vector updating unit comprises:

[0120] An attention weight calculation unit, configured to calculate the attention weights of the query vector and the key vector by using a scaled dot product attention calculation method;

[0121] A weighted summation unit is used to perform weighted summation on the value vector using the attention weight to obtain the updated potential vector.

[0122] To solve the above technical problems, the present application also provides a computer device. Fig. 9 , Fig. 9 This is a basic structural block diagram of the computer device in this embodiment.

[0123] The computer device 8 includes a memory 81, a processor 82, and a network interface 83 which are interconnected through a system bus. It should be noted that: Fig. 9Only a computer device 8 having three components, a memory 81, a processor 82, and a network interface 83, is shown in the figure, but it should be understood that it is not required to implement all the components shown, and more or fewer components may be implemented instead. Among them, those skilled in the art can understand that the computer device here is a device that can automatically perform numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes but is not limited to microprocessors, application-specific integrated circuits (ASICs), programmable gate arrays (FPGAs), digital processors (DSPs), embedded devices, etc.

[0124] Computer devices can be computing devices such as desktop computers, notebooks, PDAs, and cloud servers. Computer devices can interact with users through keyboards, mice, remote controls, touch pads, or voice control devices.

[0125] The memory 81 includes at least one type of readable storage medium, and the readable storage medium includes flash memory, hard disk, multimedia card, card-type memory (for example, SD or DX memory, etc.), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, disk, optical disk, etc. In some embodiments, the memory 81 can be an internal storage unit of the computer device 8, such as a hard disk or memory of the computer device 8. In other embodiments, the memory 81 can also be an external storage device of the computer device 8, such as a plug-in hard disk equipped on the computer device 8, a smart memory card (SmartMedia Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. Of course, the memory 81 can also include both the internal storage unit of the computer device 8 and its external storage device. In this embodiment, the memory 81 is generally used to store the operating system and various application software installed on the computer device 8, such as the program code of the image generation method based on the Brownian bridge diffusion model, etc. In addition, the memory 81 can also be used to temporarily store various types of data that have been output or are to be output.

[0126] The processor 82 may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chips in some embodiments. The processor 82 is generally used to control the overall operation of the computer device 8. In this embodiment, the processor 82 is used to run the program code stored in the memory 81 or process data, such as running the program code of the image generation method based on the Brownian bridge diffusion model to implement various embodiments of the image generation method based on the Brownian bridge diffusion model.

[0127] The network interface 83 may include a wireless network interface or a wired network interface, and the network interface 83 is generally used to establish a communication connection between the computer device 8 and other electronic devices.

[0128] The present application also provides another implementation, namely, providing a computer-readable storage medium, which stores a computer program, and the computer program can be executed by at least one processor to enable the at least one processor to perform the steps of the above-mentioned image generation method based on the Brownian bridge diffusion model.

[0129] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, a disk, or an optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.) to execute the methods of each embodiment of the present application.

[0130] Obviously, the embodiments described above are only some embodiments of the present application, rather than all embodiments. The preferred embodiments of the present application are given in the accompanying drawings, but they do not limit the patent scope of the present application. The present application can be implemented in many different forms. On the contrary, the purpose of providing these embodiments is to make the understanding of the disclosure of the present application more thorough and comprehensive. Although the present application is described in detail with reference to the aforementioned embodiments, for those skilled in the art, it is still possible to modify the technical solutions recorded in the aforementioned specific implementation methods, or to replace some of the technical features therein with equivalents. Any equivalent structure made using the contents of the specification and drawings of this application, directly or indirectly used in other related technical fields, is similarly within the scope of patent protection of this application.

Claims

1. An image generation method based on a Brownian bridge diffusion model, characterized in that: include: Receive a set of image combinations, wherein the image combination includes a satellite image and a ground panoramic image corresponding to the satellite image; Extracting shared features of the satellite image and the ground panoramic image through a cross-view image joint encoder; Performing polar coordinate transformation on the satellite image, and encoding the polar coordinate transformed satellite image and the ground panoramic image into a latent space to obtain an initial satellite image latent vector and a ground panoramic image latent vector; In the latent space, a Brownian bridge forward process is performed based on the initial satellite image latent vector and the ground panoramic image latent vector to gradually add noise to the ground panoramic image latent vector to obtain a satellite image latent vector; Performing a Brownian bridge inverse process based on the satellite image latent vector and the shared feature to gradually remove noise in the satellite image latent vector to generate a target latent vector; The target potential vector is decoded to generate a target ground panoramic image.

2. The image generation method based on the Brownian bridge diffusion model according to claim 1, characterized in that: The extracting shared features of the satellite image and the ground panoramic image by using a cross-view image joint encoder includes: Extracting features of the satellite image and the ground panoramic image respectively to obtain initial satellite image features and initial ground image features; Performing average pooling on the initial satellite image features and the initial ground image features respectively to obtain basic satellite image features and basic ground image features; The basic satellite image features and the basic ground image features are embedded into a shared feature space through a pre-trained encoder to obtain the shared features.

3. The image generation method based on the Brownian bridge diffusion model according to claim 2, characterized in that: Before embedding the basic satellite image features and the basic ground image features into a shared feature space by a pre-trained encoder to obtain the shared features, the method further includes: Acquiring an original satellite image and an original ground panoramic image corresponding to the original satellite image; A convolutional neural network is used as an encoder, and based on the encoder, feature extraction and feature pooling are performed on the original satellite image and the original ground panoramic image to obtain a compressed feature vector; The InfoNCE loss function is used as the model loss function, and the GPS coordinate sampling method and the dynamic similarity sampling method are used to train the encoder based on the compressed feature vector to obtain the pre-trained encoder.

4. The image generation method based on the Brownian bridge diffusion model according to claim 1, characterized in that: The step of performing a Brownian bridge forward process in the latent space based on the initial satellite image latent vector and the ground panoramic image latent vector to gradually add noise to the ground panoramic image latent vector to obtain a satellite image latent vector includes: In the latent space, constructing a mapping path from the ground panoramic image latent vector to the initial satellite image latent vector; gradually adding noise to the ground panoramic image potential vector along the mapping path by forward diffusion, and recording the noise as a true noise label; If the current ground panoramic image potential vector reaches the initial satellite image potential vector, the noise addition is stopped, and the current ground panoramic image potential vector is used as the satellite image potential vector.

5. The image generation method based on the Brownian bridge diffusion model according to any one of claims 1 to 4, characterized in that: The performing of the Brownian bridge reverse process based on the satellite image potential vector and the shared feature to gradually remove noise in the satellite image potential vector to generate a target potential vector includes: constructing a reverse mapping path based on the satellite image latent vector; Based on the reverse mapping path, the noise in the satellite image latent vector is gradually removed, and in each step of removing the noise in the satellite image latent vector, the current satellite image latent vector is updated by combining the attention mechanism and the shared feature; When the reverse mapping path is executed to completion, the target latent vector is generated.

6. The image generation method based on the Brownian bridge diffusion model according to claim 5, characterized in that: The step of gradually removing the noise in the satellite image potential vector based on the reverse mapping path, and updating the current satellite image potential vector in combination with the attention mechanism and the shared feature in each step of removing the noise in the satellite image potential vector, includes: gradually removing noise in the satellite image latent vector based on the reverse mapping path; In each step of removing noise from the satellite image latent vector, the current satellite image latent vector is used as a query vector, and the shared features are used as a key vector and a value vector respectively; Attention weights of the query vector and the key vector are calculated, and the value vector is updated based on the attention weights to obtain an updated potential vector.

7. The image generation method based on the Brownian bridge diffusion model according to claim 6, characterized in that: The calculating the attention weights of the query vector and the key vector, and updating the value vector based on the attention weights to obtain an updated potential vector, includes: Calculating the attention weights of the query vector and the key vector using a scaled dot product attention calculation method; The value vector is weightedly summed using the attention weight to obtain the updated potential vector.

8. An image generation device based on a Brownian bridge diffusion model, characterized in that: include: An image receiving unit, configured to receive a group of image combinations, wherein the image combination includes a satellite image and a ground panoramic image corresponding to the satellite image; A feature extraction unit, configured to extract shared features of the satellite image and the ground panoramic image through a cross-view image joint encoder; A coordinate transformation unit, used for performing polar coordinate transformation on the satellite image, and encoding the polar coordinate transformed satellite image and the ground panoramic image into a latent space to obtain an initial satellite image latent vector and a ground panoramic image latent vector; a noise adding unit, configured to perform a Brownian bridge forward process in the latent space based on the initial satellite image latent vector and the ground panoramic image latent vector, so as to gradually add noise to the ground panoramic image latent vector to obtain a satellite image latent vector; a noise removal unit, configured to perform a Brownian bridge inverse process based on the satellite image latent vector and the shared feature to gradually remove noise in the satellite image latent vector and generate a target latent vector; The image generation unit is used to decode the target potential vector to generate a target ground panoramic image.

9. A computer device, characterized in that: The method comprises a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, the image generation method based on the Brownian bridge diffusion model as described in any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the image generation method based on the Brownian bridge diffusion model according to any one of claims 1 to 7 is implemented.

Images