All-directional image super-resolution reconstruction method based on latitude-aware latent diffusion model

Abstract

The application relates to the field of computer vision and image processing, and discloses an omnidirectional image super-resolution reconstruction method based on a latitude-aware latent diffusion model, which comprises the following steps: acquiring a low-resolution omnidirectional image and obtaining a guide image through an interpolation method upsampling; constructing a latitude-aware map and inputting the latitude-aware map into a latitude-aware network after being fused with image features to extract multi-scale image features; using an image encoder to encode a high-resolution image and the guide image to obtain a latent representation and a latent residual; constructing a latent diffusion sequence according to a diffusion scheduling function; starting from a final diffusion representation, using a denoising network to reversely sample and restore the latent representation; inputting a decoder to be decoded into a high-resolution image; and training and optimizing through a joint loss function. The technical scheme adopts the technical scheme of introducing a latitude map as an auxiliary control signal and fusing spatial semantic features through a multi-scale control module, so that the texture restoration intensity can be adaptively adjusted in different latitude regions.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of computer vision and image processing, in particular to an omnidirectional image super-resolution reconstruction method based on latitude-aware latent diffusion model. BACKGROUND

[0002] In real life and practical applications, high-definition and structurally stable panoramic images are gradually becoming a key requirement in many scenarios. For example, in virtual reality systems, high-precision omnidirectional images not only affect the immersive experience of users, but also directly determine the effect of subsequent three-dimensional modeling and spatial perception tasks. In the fields of geographic remote sensing, medical imaging and digital city construction, the original images often have low resolution limitations and cannot meet the needs of high-precision analysis and automatic recognition. Therefore, how to realize high-quality reconstruction and detail enhancement of omnidirectional images while maintaining consistent image semantics has become a technical problem to be solved.

[0003] Current mainstream image super-resolution methods mainly construct end-to-end mapping models based on convolutional neural networks, directly learning the reconstruction relationship between low-resolution images and high-resolution images. Such methods have made great progress on natural image datasets, especially in the visual perception level. Some methods introduce attention mechanisms to enhance the expression ability of local features, which can improve the local texture restoration effect. Some models use perceptual loss to optimize the similarity of image semantic layers, which helps to improve the high-level visual consistency of images.

[0004] Although existing methods perform well in general image processing tasks, they still have obvious limitations when dealing with omnidirectional images, a special type of data. First, mainstream models generally equate spherical projection images to ordinary image processing, without considering the fact that different latitude regions have geometric distortion, resulting in incomplete texture recovery and obvious deformation in the polar region, lacking adaptive regulation ability for regional differences. Second, most methods ignore the evolution information in the latent space and only make direct regression in the image space, lacking gradual reconstruction ability and easily causing texture jumping or context misplacement problems. In addition, traditional training mechanisms often use a single loss function for optimization, which lacks fine constraints on structural consistency and information coordination when facing complex network structures with main path and control path. Therefore, the omnidirectional image super-resolution reconstruction method based on latitude-aware latent diffusion model is proposed to solve the above problems. SUMMARY

[0005] In view of the deficiencies of the prior art, the omnidirectional image super-resolution reconstruction method based on latitude-aware latent diffusion model is provided, which solves the problems of ignoring spatial differences, insufficient adaptive adjustment, and difficulty in structural coordination in the prior art.

[0006] To achieve the above object, the application is implemented by the following technical solutions: the omnidirectional image super-resolution reconstruction method based on the latitude-aware latent diffusion model comprises the following steps:

[0007] S1, obtaining a low-resolution omnidirectional image and up-sampling a guide image by an interpolation method;

[0008] S2, constructing a latitude-aware map and inputting the latitude-aware map and image features into a latitude-aware network after fusion, and extracting multi-scale image perception features;

[0009] S3, using an image encoder to encode a high-resolution image and an up-sampled guide image respectively to obtain original latent representation and guide latent representation, and calculating a difference between the original latent representation and the guide latent representation to obtain a latent residual representation;

[0010] S4, according to a set diffusion scheduling function and Markov transition rule, superimposing the original latent representation and the residual representation and adding Gaussian noise to construct a multi-stage latent diffusion sequence;

[0011] S5, starting from the final diffusion representation, relying on a denoising neural network to perform reverse iterative sampling to restore the original latent representation;

[0012] S6, inputting the reconstructed latent representation into a decoder to decode the high-resolution image;

[0013] S7, training and optimizing the model by a joint loss function, wherein the loss function comprises a latent reconstruction loss, an image reconstruction loss and a perception loss.

[0014] The application provides an omnidirectional image super-resolution reconstruction method based on a latitude-aware latent diffusion model.

[0015] 1, the application introduces a latitude map as an auxiliary control signal, and fuses spatial semantic features through a multi-scale control module, which achieves the effect of adaptively adjusting the texture restoration intensity in different latitude regions, compared with the strategy of uniformly processing omnidirectional image regions in the prior art, which solves the problems of blurred structure recovery in the polar region and insufficient enhancement in the equatorial region.

[0016] 2, the application constructs a latent space reverse sampling mechanism based on a diffusion model, which can effectively capture the cross-scale semantic distribution features in the low-dimensional space, and achieves a stable process of gradually evolving high-resolution features rather than one-time reconstruction.

[0017] 3、The decoding network with consistent structure combines the design method of residual and skip connection, restores the texture details of the image, ensures the consistency of the global structure and the full transmission of the channel information, and is significantly better than the hierarchical isolated decoding structure in multi-level feature fusion compared with the traditional decoder, and alleviates the problem of detail loss in the reconstruction process.

[0018] 4、The joint loss function training mechanism proposed in the application includes latent consistency, pixel reconstruction, perceptual difference and control alignment loss, realizes end-to-end consistency supervision from semantic space to image space, and compared with the method relying on single MSE or perceptual loss in the past, the multi-dimensional joint constraint method is more suitable for complex image patterns, and solves the defects of slow training convergence and weak generalization ability. BRIEF DESCRIPTION OF DRAWINGS

[0019] Figure 1 The method flowchart of the application is shown. DETAILED DESCRIPTION

[0020] The technical solutions in the embodiments of the application will be clearly and completely described below with reference to the drawings in the application specification. Obviously, the described embodiments are only a part of the embodiments of the application, not all the embodiments. Based on the embodiments in the application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the application.

[0021] Please refer to the drawings in the application specification Figure 1 The embodiment of the application provides an omnidirectional image super-resolution reconstruction method based on a latitude perception latent diffusion model, which comprises the following steps:

[0022] S1, obtaining a low-resolution omnidirectional image and obtaining a guide image by an interpolation method;

[0023] Specifically, in the omnidirectional image super-resolution reconstruction method proposed in the application, step S1 as a key starting stage is mainly responsible for constructing a guide image according to the input low-resolution omnidirectional image. The guide image will be used as a structure guiding condition in the subsequent latent diffusion modeling process and throughout the information restoration link. Therefore, in order to ensure that the model has stability and consistency in the preliminary understanding of image content, it is necessary to quickly align the input image size to the target high-resolution scale in a low-complexity manner.

[0024] Generally, the generation of the guide image does not directly improve the detail quality of the image, but provides a lightweight prior structure as a coordination interface between modules. This step not only provides initial pixel features for the subsequent latent encoder, but also to some extent preserves the spatial perception structure, thereby assisting the model to avoid structure misjudgment phenomenon.

[0025] Specifically, in the present embodiment, the input image is a low-resolution omnidirectional image acquired via a panoramic camera, whose image size is set as . The goal is to obtain a corresponding high-resolution reconstructed image with a size of , where , , and is the magnification factor, usually selected as 2, 4, or 8.

[0026] In one possible implementation, the original image is upsampled using the Bilinear Interpolation method to obtain a preliminary guide image , which satisfies:

[0027] ;

[0028] wherein: represents the bilinear interpolation up-sampling operation on the input image with a factor of .

[0029] As an option, Bicubic, Laplacian Pyramid Upsampling, or other parameter-free reconstruction methods can also be used, provided that the output image size is strictly consistent with the target resolution and does not introduce model learnable parameters.

[0030] In some embodiments, to enhance the structural fidelity of the interpolated guide image, preprocessing methods such as light equalization, brightness normalization, or edge enhancement processing can also be used to make the structural features more prominent, facilitating the subsequent latent encoder to extract corresponding features.

[0031] Further, the guide image generated in the present embodiment will be input into the downstream latent representation modeling module as a structural guide, in conjunction with the subsequently extracted guide latent representation .

[0032] It should be noted that although itself does not contain additional texture details, it plays a crucial role in model design. Its role is not to generate content, but to "provide positioning", i.e., to provide a positional structure reference for the subsequent model to recover the latent residual. This process is similar to providing a "weak structural skeleton" in a blurred space.

[0033] It is worth noting that the construction of this guide image is a one-time static process that does not require training. Its design goal is to minimize computational overhead while retaining sufficient spatial semantic features to assist the model in accurately reconstructing the target image details.

[0034] In addition, in some other embodiments, to match the omnidirectional image characteristics, the input image With guided images The number of channels is usually set to 3 (i.e., RGB channels), but for other sensor images such as thermal infrared and depth maps, the number of channels can be extended to 1 or more.

[0035] The above interpolation strategy does not explicitly learn the image content, but only relies on the local weight estimation of space, so the generated There will be blurring in the edge areas, which is an important basis for correction by potential diffusion modeling in subsequent steps.

[0036] S2, constructing a latitude perception map and fusing it with image features, then inputting it into the latitude perception network to extract multi-scale image perception features;

[0037] Specifically, after generating the guidance image, the present invention proceeds to the next stage, step S2: latitude map construction and feature fusion. This step follows the structure guidance map generated in step S1 and further combines the geometric deformation differences in the vertical dimension of the omnidirectional image to construct a latitude map for perceiving the distortion characteristics of the longitude and latitude structure. This latitude map is then integrated into the image feature representation, improving the model's responsiveness to spatial location information.

[0038] In general, due to its spherical projection characteristics, omnidirectional images exhibit uneven distribution of spatial distortion in the longitudinal direction (i.e., height dimension) of the image. Pixels near the top or bottom of the image correspond to the polar regions of the sphere, while the middle portion approximately corresponds to the equator of the sphere. This type of nonlinear spatial deformation is often ignored in traditional planar image processing, resulting in structural dislocation or blurring in the polar regions of the reconstructed model. Therefore, in the present invention, a latitude map is introduced as a geometric compensation mechanism for spatial position to construct a spatial perception structure prior.

[0039] Specifically, in this embodiment, the height dimension of the input image is first Perform position indexing and set the image size to ,in Indicates the image height, Indicates the image width.

[0040] Constructing a 2D latitude map , which is defined as follows:

[0041] ;

[0042] in: Indicates the image Liedi The latitude weighted value of the row pixel; Indicates the height of the image in pixels; denotes the horizontal coordinate index of the image; denotes the vertical coordinate index of the image; is the constant of pi, controlling the amplitude range of the mapping, smoothly varying in the interval [-1, 1]; the function form introduces as a normalized position term, ensuring the latitude map is symmetrically distributed in the image center.

[0043] As an option, the may be further normalized to compress its range to the interval [0, 1], with the specific form being:

[0044] ;

[0045] But in this embodiment, to enhance the contrast sensitivity of positive and negative regions, the original cosine mapping value is directly used.

[0046] In a possible implementation, the latitude map will be expanded and duplicated to multiple channels, spliced with the backbone feature. For example, for the backbone network feature , the latitude feature may be constructed and broadcasted and duplicated as , and then concatenated with :

[0047] ;

[0048] In this embodiment, the latitude map can be input to the subsequent feature extraction network together with the structure guide map or its feature representation.

[0049] As an implementation, the latitude map can also be input to the control branch to assist in constructing a modulation factor with spatial bias for controlling the backbone convolution path.

[0050] In some embodiments, the latitude coefficient can be further applied to the attention weight generation of each feature channel to form a weighted feature map, so as to realize the regional adaptation mechanism of “polar region suppression, equatorial enhancement”, thereby improving the recognition ability of the model in the edge region.

[0051] In addition, in this embodiment, the constructed latitude map is considered as a non-learnable structure prior term and does not participate in the backward propagation weight update process, and is mainly used to guide the model to automatically adapt the spatial perception response ability in each region of the image.

[0052] Generally, to maintain structural consistency, the construction process of the latitude map directly corresponds to the resolution of the input image, in other words, if the input image is , then the latitude map should also be , to avoid dimension misalignment problems in subsequent splicing operations.

[0053] In order to enhance the feature alignment capability, in some implementations, the latitude map can also be preliminarily encoded through a convolutional module, so that it is transformed from the original position map into an implicit semantic map before participating in the fusion. However, this method is not the preferred path in this method because it introduces learnable parameters and destroys the stability of the spatial prior.

[0054] Furthermore, in the fusion structure, the latitude map is not only combined with the encoding features Splicing, also with the potential guide features in step S3 They are input into the multi-layer convolution module together to form a joint feature set that integrates contextual information, spatial location, and structural guidance.

[0055] This feature set will be used as conditional information to input into the noise prediction network in the subsequent potential diffusion modeling process. In the process, it plays the role of global spatial structure modulation.

[0056] S3. Use an image encoder to encode the high-resolution image and the upsampled guide image respectively to obtain the original latent representation and the guided latent representation, and calculate the difference between the original latent representation and the guided latent representation to obtain the latent residual representation;

[0057] Specifically, after completing the generation of the guidance map and the construction of the latitudinal map, the present invention enters the latent representation construction phase, i.e., step S3. The primary goal of this phase is to extract the multi-layered representation of the image in the latent space and construct latent residual features by comparing the features of the guidance image with the high-quality image. These residual features play a key role in information compensation and restoration in the subsequent latent diffusion process and are the fundamental source of the model's detail restoration performance.

[0058] Typically, the explicit pixel information of an image contains a significant amount of redundancy, while the latent space can express the image's semantics and structure in a compressed manner. This method exploits this property by encoding the high-resolution image and the guidance image separately to obtain a latent representation, and characterizes detail loss through the residual relationship between the two. Compared to direct pixel modeling, this method boasts greater representation sparsity and modeling stability, facilitating the effective implementation of diffusion-based prediction mechanisms.

[0059] Specifically, in this embodiment, the input high-resolution image is first set to , the guide image is , both are the same size, .

[0060] Build two latent encoder networks separately: main encoder and boot encoder , for extracting latent features of high-resolution image and guidance image. Wherein,

[0061] ;

[0062] ;

[0063] Wherein: represents the encoded representation of the target high-resolution image in the latent space, which is the original latent representation; represents the encoded representation of the guidance image in the latent space, which is the guidance latent representation; , , wherein, is the encoding down-sampling factor, usually 4 or 8; is the number of latent space channels, usually set to 64, 128 or other appropriate values;

[0064] The encoder structure generally adopts a lightweight convolution stack, such as a multi-layer residual module or a Conv-BN-ReLU structure, to avoid information overfitting.

[0065] In order to obtain the expression difference value of high-frequency details, the latent residual tensor is calculated:

[0066] ;

[0067] Wherein: represents the encoded representation of the high-resolution image in the latent space, which is the original latent representation; represents the encoded representation of the up-sampled image in the latent space, which is the guidance latent representation; represents the latent residual feature map between the two, which is the latent residual representation.

[0068] This residual tensor reflects the missing detail features in the guidance image, especially having stronger response ability to texture, edge and local structure information.

[0069] In some embodiments, in order to prevent the feature scale from being too large to cause training instability, the latent feature can be normalized before calculating the residual. For example, the mean and standard deviation of each channel are normalized:

[0070] , ;

[0071] ;

[0072] Wherein: , respectively represent the mean and standard deviation calculated by channel.

[0073] As an option, attention mechanisms such as channel attention or spatial attention modules can be added to the residual features to further improve local feature sensitivity.

[0074] In one possible implementation, the latent feature extraction network not only receives the image input, but also integrates the latitude map features generated in step S2. For example, at the encoder input, with latitude map Splicing, the form is:

[0075] ;

[0076] ;

[0077] This design enables the encoder to have spatial bias capabilities, which helps the model focus on the geometric characteristics of the polar regions and improves the robustness of the potential representation to structural distortions.

[0078] In addition, multi-scale structures can be added during feature encoding to capture the local and global feature levels of the image. Common structures include:

[0079] Dilated Conv to expand the receptive field;

[0080] Pyramid pooling (SPP module) is used to extract scale-invariant semantics;

[0081] The deformable convolution kernel mechanism (Deformable Conv) is used to enhance the adaptability to polar distorted areas.

[0082] In this embodiment, the potential residual It will be used as an information perturbation term in the subsequent potential diffusion process, and its participation level will be dynamically adjusted according to the number of steps. Its construction method will directly affect the stability and prediction accuracy of the subsequent diffusion sequence generation.

[0083] In order to enhance the robustness of feature representation, the encoder structure is recommended to share some parameters with the denoising network, or adopt a symmetric structure to achieve collaborative constraints on the information path.

[0084] S4. According to the set diffusion scheduling function and Markov transition rule, the original latent representation and the residual representation are superimposed and Gaussian noise is added to construct a multi-stage latent diffusion sequence;

[0085] Specifically, after constructing the latent representation and latent residual, the method proceeds to the diffusion sequence generation stage in the latent space, i.e., step S4. The core of this stage is to transform the target latent representation into a controlled perturbation sequence. Through a stepwise diffusion operation, the information degradation path is simulated, providing noise conditions and dynamic guidance cues for the subsequent reverse reconstruction process.

[0086] Generally, the traditional diffusion model tends to add Gaussian noise diffusion operation in pixel space, but the diffusion strategy proposed in the present application does not directly act on the image pixels, but is established in the latent feature space, which has stronger structural consistency and modeling efficiency. In this stage, the model introduces latent residuals as disturbance benchmarks, and combines Gaussian randomness to iteratively construct latent variable sequences at multiple steps , thereby completing the step-by-step mapping of high-dimensional semantics to low-quality representations.

[0087] Specifically, in the present embodiment, the acquired latent representation and the latent residual are used to construct the diffusion sequence.

[0088] Define the step index , where represents the total diffusion step number, which is generally a fixed value such as 50, 100 or 200. To control the disturbance amplitude of each step, a time scheduling factor is introduced, which is defined as:

[0089] ;

[0090] where: is the diffusion strength of the th step; is the initial disturbance amplitude; is the maximum disturbance amplitude; is the scheduling curvature control parameter, which determines the nonlinear trend of the diffusion process; is the current time step; is the total number of diffusion steps.

[0091] On the basis of the above scheduling, the latent diffusion state is constructed, which is defined as follows:

[0092] ;

[0093] where: is the diffusion latent feature of the th step; is the original target latent representation; is the latent residual constructed in the previous stage; controls the degree of participation of residual information; represents the standard normal Gaussian noise; is the noise amplitude coefficient, which controls the strength of the noise disturbance; represents the standard deviation scaling item bound with the scheduling factor.

[0094] It should be particularly pointed out that the above formula will explicitly introduce the residual with random Gaussian terms Fusion is performed to some extent, making the potential state of each step have both structure guidance and noise generalization ability, providing sufficient conditions for subsequent anti-diffusion.

[0095] As an implementation, the state transition between each diffusion step can be modeled by a Markov chain. The forward transition probability of the latent diffusion state is defined as:

[0096]

[0097] where: represents the conditional probability distribution of the latent state; , represents the increment of the diffusion intensity of the adjacent step; represents the covariance matrix of this step; is the unit covariance matrix; represents the probability density function of the random variable with mean and covariance .

[0098] This Markov diffusion mechanism shows that the current state can be obtained by applying a bias in the residual direction to the previous state and adding a noise sample. This transition rule provides a theoretical basis for the denoising prediction model in subsequent reverse sampling.

[0099] In some embodiments, to improve the stability and continuity of the sequence, an autoregressive adjustment mechanism can also be introduced, that is, when generating , introduce historical multiple state information such as , , construct a second-order or third-order Markov chain, and further strengthen the consistency of the sequence.

[0100] As an option, the diffusion scheduling factors and can be fixed by pre-setting the function form, or can be predicted and generated by an external control module (such as a time encoder or attention network) as learnable parameters, to realize a more flexible scheduling mechanism.

[0101] Further, in actual implementation, the sampler module can generate all diffusion steps of the sequence at once, or can take a step-by-step iterative generation mode to expand layer by layer according to the time step, which is beneficial to save video memory and optimize the allocation of computing resources.

[0102] After the diffusion sequence is generated, the final state ​The initial input of the reverse denoising sampling in step S5 is used to restore the latent representation approximation and reconstruct the clear image content.

[0103] S5, from the final diffusion representation, the original latent representation is restored by relying on the denoising neural network for reverse iterative sampling;

[0104] Specifically, after completing the forward generation of the diffusion sequence, the present application enters step S5, that is, the reverse denoising sampling stage. This stage takes the last state in the latent diffusion space as the starting point, and uses the trained conditional denoising prediction model to perform reverse reconstruction, gradually eliminating noise and residual signals through step-by-step reasoning to restore the original state of the latent representation.

[0105] Generally, the reverse process of the diffusion model is driven by a parameterized denoising network, which gradually guides the latent state from the noise state to a clean representation with clear structure. In the present application, the reverse denoising path not only relies on the diffusion state , but also combines the guided latent representation , the original low-resolution image , the latitude map and the current step size to form a multi-condition control mechanism to realize the collaborative control of spatial structure, semantic prior and dynamic disturbance information.

[0106] Specifically, in this embodiment, the parameterized conditional denoising network is defined as , which includes the current latent state , the guided latent feature , the step size , the low-resolution image and its corresponding latitude map , and the output is the residual prediction under the current step size.

[0107] The sampling mean of the first step of reverse sampling is defined as:

[0108] ;

[0109] Where: represents the reconstruction mean of the first step; , is the diffusion scheduling factor defined in step S4; represents the increment of the diffusion intensity of the adjacent step; represents the conditional noise estimation network, which includes a multi-scale residual block, a time modulation module and a control condition fusion module;

[0110] All inputs are kept at the same spatial scale as the diffusion state.

[0111] The covariance introduced at sampling time is:

[0112] ;

[0113] where: is the Gaussian noise covariance matrix of the th step sampling; is the noise coefficient, taken from the forward process definition; is the identity covariance matrix, used to control the uniform distribution of noise.

[0114] On this basis, the sampling process can be written as:

[0115] ;

[0116] This process recovers from the current latent state to , and finally obtains the estimated latent original state at .

[0117] In one possible implementation, the time step can be fed into the model through position encoding, for example, using a sine-cosine embedding or a learnable embedding based on Learnable Embedding, so that the model can dynamically adapt to different stages of recovery strategies.

[0118] To enhance the expression ability of control information in the network, in this embodiment, a pixel-aware control module (Pixel-aware Control Block, PACB) is introduced, and the core structure form is:

[0119] ;

[0120] where: represents the final fused output feature map; represents the intermediate feature map of the control flow branch; represents the fused feature map after splicing the context feature and the control feature; represents the convolution operation with zero initialized weights; represents the element-wise multiplication operation.

[0121] This structure realizes the dynamic adjustment of the control path to the backbone feature, and fuses the residual information to realize controllable sampling.

[0122] Generally, the PACB structure is placed after the multi-layer residual module to exert control guidance at the medium-high layer stage of feature expression, so as to ensure the effectiveness of information injection.

[0123] In some embodiments, the PACB module can work in conjunction with a time embedding to form a conditional time-aware PACB to further improve the temporal consistency and control response stability.

[0124] It is worth noting that the above reverse sampling process is in the form of conditional generation, and the model can accept different structure guiding conditions to maintain the diversity of the generated space while having a certain degree of reconstruction fidelity.

[0125] After sampling is completed, the final estimated latent representation The input is input to the decoder defined in step S6 for recovery output in the image domain.

[0126] S6, input the reconstructed latent representation into the decoder to decode into a high-resolution image;

[0127] Specifically, after completing the reverse denoising sampling operation on the diffusion latent sequence, the final latent representation estimate value The method of the present application enters step S6, i.e., the process of restoring the latent feature to the image space. This step uses a structured decoding network to upsample and restore the latent feature, and finally outputs a super-resolution image with a size consistent with the high-resolution target image.

[0128] Generally, the latent space and the image space differ significantly in terms of dimension distribution and information organization form, so they cannot be directly mapped. A set of decoding networks with good structural consistency and clear feature response are needed to reconstruct abstract semantics into specific texture content.

[0129] In the present application, the design of the decoder needs to meet two goals: one is to maintain the corresponding relationship between the reconstructed image and the original high-resolution image in structure and details, and the other is to realize end-to-end mapping without introducing excessive inference calculation burden.

[0130] Specifically, in this embodiment, the input is the latent representation obtained in step S5 wherein represents the number of feature channels, , , is a feature scaling factor.

[0131] The final target image is set to wherein , is the original high-resolution target size.

[0132] To complete this mapping operation, a decoder whose output expression is:

[0133] ;

[0134] In a possible implementation, the decoder is composed of multiple up-sampling modules, each module including an up-sampling operation, a convolution layer, a normalization layer, and an activation function.

[0135] Generally, the up-sampling operation can adopt one of the following forms:

[0136] Transposed Convolution;

[0137] Sub-pixel Convolution, namely, PixelShuffle structure;

[0138] nearest-neighbor interpolation or bilinear interpolation + convolution combination.

[0139] In this embodiment, the PixelShuffle structure is preferably adopted to reduce the grid artifact phenomenon. Specifically, assuming that the up-sampling multiple is , the decoding network includes two layers of consecutive pixel rearrangement modules, each layer performing spatial magnification.

[0140] The operation of each decoding block can be represented as:

[0141] ;

[0142] wherein: represents the input feature map of the layer; achieves rearrangement of the channel dimension to the spatial dimension; is a standard convolution operation, with a kernel size of 3; is an activation function, used to enhance the non-linear expression capability.

[0143] In some embodiments, a 1x1 convolution can also be connected at the end of the decoder for channel mapping, converting the final feature map into an RGB image format:

[0144] ;

[0145] To further improve the reconstruction quality, a residual connection structure is introduced in the decoding stage in this embodiment, so that the low-level features and high-level features form a fusion path:

[0146] ;

[0147] wherein: is a spatial alignment operation consistent with the decoding stride, guaranteeing the size compatibility of the skip connection.

[0148] As an option, an attention module, such as channel attention or spatial attention, can be embedded in the decoder structure to improve the ability to restore texture details, especially for the detailed structure recovery of polar region areas.

[0149] In some embodiments, to enhance the stability and generalization of the decoder, a symmetric structure design can also be used, so that the decoder is mirror-aligned with the encoder structure, thereby realizing the structural consistency of the information channel.

[0150] Further, considering the spatially uneven distribution characteristics of the omnidirectional image, in the present embodiment, the input features received by the decoder The modulation information of the latitude map in step S2 has been included, so there is no need to repeatedly add geometric priors in the decoding stage, but the polar region weight compensation and equatorial region detail enhancement are completed through feature expression transmission.

[0151] The final generated The high-resolution image In the training stage, supervised comparison is performed to construct the loss function, and after the training is completed, the decoder has the ability to directly restore clear images.

[0152] S7, the model is trained and optimized through a joint loss function, and the loss function includes latent reconstruction loss, image reconstruction loss and perceptual loss.

[0153] Specifically, after completing the latent space decoding and obtaining the final reconstructed image, to further optimize the collaborative effect of each module of the model and improve the accuracy and stability of the reconstruction result, the present application enters step S7, i.e., the joint loss training stage. This step realizes the simultaneous improvement of the modeling accuracy of the latent space and the quality of the image space reconstruction by constructing a multi-level, cross-space loss function system, which is a key part of the entire network training link.

[0154] Generally, the traditional image super-resolution method mainly uses the pixel space mean square error as the only supervision signal, which is difficult to effectively constrain the feature evolution process in the latent space. In the present application, by simultaneously introducing the latent space alignment loss, the image space reconstruction error, the perceptual consistency constraint and the structure control module supervision signal, the joint regulation of the modeling process from multiple dimensions is realized.

[0155] Specifically, in the present embodiment, four main loss terms are constructed, which are:

[0156] Latent space reconstruction loss (Latent Consistency Loss);

[0157] Image space mean square error loss (MSE Loss);

[0158] Perceptual loss (Perceptual Loss);

[0159] Control Alignment Loss.

[0160] First, the latent space reconstruction loss is defined as follows:

[0161] ;

[0162] where: is the original target latent representation; is the diffusion state at the th step; is the reverse denoising sampling prediction network; is the total number of diffusion steps; and are the low-resolution image and the latitude map, respectively; denotes the L2 norm.

[0163] This loss term is used to constrain the approximation accuracy of the target latent state at each step in the reverse denoising process.

[0164] Second, the image space mean square error loss is defined as follows:

[0165] ;

[0166] where: is the real high-resolution image; is the reconstructed image output by the decoder; this term is mainly used to evaluate the accuracy of the reconstructed image at the pixel level.

[0167] To enhance the expression ability of structural perception, a perception loss is further introduced:

[0168] ;

[0169] where: is a fixed perception feature extraction network, such as the intermediate layer output of VGG-19; this loss focuses on the consistency of the image in the high-level semantic feature space, and is used to supplement the limitations of the pixel space.

[0170] In addition, to maintain the consistency of the backbone path and the control path response, a control consistency loss is designed in this embodiment, and the specific form is:

[0171] ;

[0172] where: denotes the output of the pixel perception control module; is the corresponding intermediate feature map in the backbone path; denotes the gradient stop operation, which is used to prevent the backbone gradient from flowing back to the control path; represents the consistency constraint loss for the pixel-perception control block output; this loss guides the control module output to align with the backbone semantic features, ensuring stable and reliable control signals.

[0173] Finally, the above losses are combined to form the total training objective function:

[0174] ;

[0175] wherein: represents the consistency constraint loss for the pixel-perception control block output; represents the joint loss function for the overall model training; represents the reconstruction loss in the latent space; represents the pixel reconstruction error loss in the image space; represents the perception loss based on the difference in deep image features; is the weighting coefficient for each loss term; the coefficients of each term can be adjusted through cross-validation or experience setting to ensure the balance between network convergence and reconstruction quality.

[0176] In some embodiments, a step-by-step scheduling strategy can be adopted, i.e., the perception loss and control loss weights are weakened in the early training period, relying only on the latent space and pixel space supervision; and after the model is stable and convergent, the proportion of perception and control items is gradually increased to enhance the model structure understanding ability and response flexibility.

[0177] As an option, the loss composition structure can also be dynamically adjusted based on the task scenario, for example, emphasizing pixel accuracy in high-precision industrial imaging scenarios, and paying more attention to structural perception characteristics in visual perception tasks.

[0178] During the training process, the Adam optimizer is used in this embodiment, and the initial learning rate is usually set to and is dynamically decreased when there is no improvement in the preset training number or validation set indicators.

[0179] All training processes are carried out under a multi-GPU parallel architecture, and a mixed precision training mechanism is used to improve training efficiency and stability.

[0180] Although embodiments of the present application have been shown and described, it will be understood by those of ordinary skill in the art that various changes, modifications, substitutions and alterations can be made without departing from the principles and spirit of the present application, and the scope of the present application is defined by the appended claims and their equivalents.

Claims

1. An omnidirectional image super-resolution reconstruction method based on a latitude-aware latent diffusion model, characterized in that: The following steps are involved: Obtain a low-resolution omnidirectional image and upsample it to obtain a guidance image through interpolation; Construct a latitude perception map and fuse it with image features before inputting it into the latitude perception network to extract multi-scale image perception features; Use the image encoder to encode the high-resolution image and the upsampled guidance image respectively to obtain the original latent representation and the guided latent representation, and calculate the difference between the two to obtain the latent residual representation; The acquisition of the original potential representation and the potential residual includes: Use the encoder to train real high-resolution images Encode, get representation ; Upsample the image Encoding is guided representation ; The residuals of the two are expressed as: ； in: Represent the encoding representation of high-resolution images in the latent space; Represents the encoded representation of the upsampled image in the latent space; Represents the potential residual feature map between the two; According to the set diffusion scheduling function and Markov transition rule, the original latent representation and the residual representation are superimposed and Gaussian noise is added to construct a multi-stage latent diffusion sequence; The multi-stage potential diffusion sequence is constructed as follows: Set the number of diffusion steps , minimum diffusion intensity , maximum diffusion intensity , exponential control factor of diffusion scheduling ; Construct a diffusion intensity scheduling function: ； In the Step 2 generates latent representation: ； in: For the Diffusion intensity of the step; is the minimum value of diffusion intensity; is the maximum value of diffusion intensity; is the exponential control factor of diffusion scheduling; is the total number of diffusion steps; is the current diffusion step index, and its value range is ; is the Gaussian noise scaling factor; is a Gaussian random noise tensor with zero mean and unit variance; is the original target latent representation; is the potential residual constructed in the previous stage; Represents the standard deviation scaling term bound to the scheduling factor; Starting from the final diffusion representation, reverse iterative sampling is performed based on the denoising neural network to restore the original potential representation; The reconstructed latent representation is input into the decoder to decode into a high-resolution image; The model is trained and optimized through a joint loss function, which includes potential reconstruction loss, image reconstruction loss and perceptual loss.

2. The omnidirectional image super-resolution reconstruction method based on the latitude-aware latent diffusion model according to claim 1 is characterized in that: The constructing of the latitude perception map includes: Construct a cosine function weighted graph for the vertical coordinates of the input image to express the latitude distortion information; The latitude map is used as a weight map and is concatenated with the image features in the channel dimension to input the latitude perception network; The latitude-aware network extracts the structural and geometric features of the image to assist in the reconstruction of the latent representation; Among them, the latitude map The weight of the position is: ； in: Indicates the image Liedi The latitude weighted value of the row pixel; Indicates the height of the image in pixels; Represents the horizontal coordinate index of the image; Indicates the vertical coordinate index of the image; is the circumference constant of pi.

3. The omnidirectional image super-resolution reconstruction method based on the latitude-aware latent diffusion model according to claim 1, characterized in that: The Markov transition rule is defined as: ； in: The transition probability density function representing the potential diffusion; represents the potential representation of the previous diffusion step; Represents the encoded representation of the upsampled image in the latent space; represents the potential residual feature map; represents the increment of diffusion intensity between adjacent steps; Indicates the Diffusion intensity of the step; Indicates the Diffusion intensity of the step; represents the covariance matrix of this step; is the identity matrix with the same dimension as the latent representation; Indicates the mean , the covariance is Gaussian distribution of random variables The probability density function of .

4. The omnidirectional image super-resolution reconstruction method based on the latitude-aware latent diffusion model according to claim 1, characterized in that: The reverse iterative sampling process includes: At every step The denoising network predicts the mean and covariance of the next potential state, and the prediction form is: ； ； in: represents the mean vector used for sampling; represents the covariance matrix used for sampling; Indicates the Diffusion intensity of the step; Indicates the Diffusion intensity of the step; represents the intensity increment between adjacent diffusion steps; Indicates the The potential diffusion representation of the step; Representations guide latent representations; is the noise scaling factor; is the unit covariance matrix; For parameters The denoising neural network function is used to estimate the noise residual; is the original low-resolution omnidirectional image; is the latitude weighted graph corresponding to the image space; Indicates the current diffusion step index.

5. The omnidirectional image super-resolution reconstruction method based on the latitude-aware latent diffusion model according to claim 1, characterized in that: The denoising neural network comprises: The input contains the current diffusion potential representation , guide the latent representation , low-resolution images , Latitude map , current step count ; The network consists of a temporal modulation module and a pixel-aware control module, the feature enhancement of which is expressed as: ； in: Represents the final fused output feature map; Intermediate feature graph representing control flow branches; Represents the fused feature map after concatenating the context feature and the control feature; represents a convolution operation with zero-initialized weights; Represents the corresponding element-wise multiplication operation.

6. The omnidirectional image super-resolution reconstruction method based on the latitude-aware latent diffusion model according to claim 1, characterized in that: The decoder comprises: Input is the potential representation after reverse sampling ; It includes a decoding module composed of a multi-layer deconvolution structure to recover the latent representation into image space representation; The output is a reconstructed high-resolution image .

7. The omnidirectional image super-resolution reconstruction method based on the latitude-aware latent diffusion model according to claim 1, characterized in that: The loss function includes: Latent space reconstruction loss: ； Image space reconstruction loss: ； Perceptual Loss: ； in: represents the L2 norm, i.e. the Euclidean distance; Denoising neural network for predicting latent representations; represents the denoising reconstruction loss in the latent space; is the total number of diffusion steps; Represents the original representation of high-resolution images in the latent space; Indicates spread to The potential representation of the step; represents the guided latent representation obtained by bilinear interpolation image encoding; is the original low-resolution omnidirectional image; is the latitude weighted graph corresponding to the image space; Indicates the current diffusion step index; represents the mean square error loss in pixel space; Represents a true high-resolution image; represents the reconstructed image generated by the decoder; represents the perceptual loss, which is used to measure the difference of images in the deep feature space; Represents the deep feature representation of an image extracted by a pre-trained image perception network.

8. The omnidirectional image super-resolution reconstruction method based on the latitude-aware latent diffusion model according to claim 7, characterized in that: The total loss function of the loss function is defined as: ； in: represents the consistency constraint loss for the pixel-aware control block output; represents the joint loss function used for overall model training; represents the reconstruction loss of the latent space; Represents the pixel reconstruction error loss in the image space; Represents the perceptual loss based on the difference of deep image features; is the weighting coefficient of each loss item.

Images