Extreme magnification image resampling method and device based on generative hidden space learning

Abstract

The invention relates to an extreme magnification image resampling method and device based on generative hidden space learning, and the method comprises the steps: mapping a high-resolution image to a feature space, so as to obtain a corresponding high-resolution feature; inputting the high-resolution feature into a decoupling feature resampling network to obtain a low-resolution image and a resampling feature; generating a multi-scale feature corresponding to the resampling feature according to the adapter module, obtaining a noise prediction result of a pre-trained noise prediction network, finely adjusting the noise prediction result through the multi-scale feature to generate a perception optimization feature corresponding to the resampling feature, and inputting the perception optimization feature into a preset decoder, and outputting a target magnification resampling image corresponding to the high-resolution image. Therefore, the problems that in the prior art, when an extreme magnification image is resampled, sufficient details and an accurate semantic structure are difficult to generate, a model is prone to falling into an under-optimal training state, and the perception quality of a resampling result is greatly influenced are solved.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to an extreme magnification image resampling method and device based on generative latent space learning. Background Art

[0002] With the explosive growth of ultra-high-resolution multimedia data, image resampling has become a key technology to improve data storage and transmission efficiency, optimize resource usage in real-time applications, and ensure correct display of images on devices with different resolutions. In the process of image resampling, high-resolution images are usually downsampled to low-resolution images with consistent content for transmission or display; then, upsampling algorithms are used to restore the downsampled images to their original resolution.

[0003] Unlike super-resolution tasks, image resampling can jointly optimize the downsampling and upsampling processes, thereby retaining as much information as possible that is critical to the upsampling process; unlike image compression tasks, image resampling needs to ensure that the intermediate results are images that conform to human vision rather than binary code streams.

[0004] In recent years, many image resampling methods have been proposed, including traditional methods and deep learning-based methods. However, these methods are mainly designed for smaller resampling ratios (4x or 8x). When dealing with more extreme resampling ratios (16x or 32x), existing methods often have difficulty generating sufficient details and accurate semantic structures. At the same time, existing methods usually combine latent space feature loss and pixel loss for model training, but since the model must make a trade-off between feature loss and pixel loss, a large number of artifacts are generated, which easily leads to the model falling into a suboptimal training state.

[0005] In summary, the existing technology has difficulty in generating sufficient details and accurate semantic structures when resampling images at extreme magnifications, and it is easy to cause the model to fall into a suboptimal training state, which greatly affects the perceptual quality of the resampling results and needs to be urgently solved. Summary of the invention

[0006] The present application provides a method and device for extreme magnification image resampling based on generative latent space learning to solve the problems that the prior art has difficulty in generating sufficient details and accurate semantic structures when resampling extreme magnification images, and easily causes the model to fall into a suboptimal training state, greatly affecting the perceptual quality of the resampling results.

[0007] The first aspect of the present application provides an extreme magnification image resampling method based on generative latent space learning, which is applied to the offline training stage and includes the following steps: obtaining a first target magnification image that meets a preset high-resolution requirement, and mapping the first target magnification image to a feature space to obtain a magnification feature corresponding to the first target magnification image; inputting the magnification feature into a preset decoupled feature resampling network, and combining it with a preset resampling loss function to generate a second target magnification map that meets a preset low-resolution requirement and a resampling feature corresponding to the magnification feature; generating a multi-scale feature corresponding to the resampling feature according to a preset adapter module, and obtaining a noise prediction result of a pre-trained noise prediction network, and fine-tuning the noise prediction result through the multi-scale feature and a preset perceptual optimization loss function to generate a perceptual optimization feature corresponding to the resampling feature, and inputting the perceptual optimization feature into a preset decoder to output a target magnification resampled image corresponding to the first target magnification image.

[0008] Optionally, in one embodiment of the present application, the rate feature is input into a preset decoupled feature resampling network, and combined with a preset resampling loss function to generate a second target rate map that meets the preset low resolution requirement and a resampling feature corresponding to the rate feature, including: based on a preset hybrid encoder, simultaneously performing feature processing operations on the first target rate image and the rate feature to respectively generate the processing results corresponding to the first target rate image and the rate feature; fusing the processing results corresponding to the first target rate image and the rate feature to obtain a first intermediate expression corresponding to the first target rate image; obtaining a forward function of a domain converter in the decoupled feature resampling network to convert the first intermediate expression into a pixel domain through the forward function, and performing a quantization operation on the first intermediate expression in the pixel domain to generate the second target rate map; converting the second target rate map through an inverse function of the domain converter to generate a second intermediate expression; and mapping the second intermediate expression to the resampling feature based on the decoder in the decoupled feature resampling network and the resampling loss function.

[0009] Optionally, in one embodiment of the present application, the multi-scale features corresponding to the resampled features are generated according to a preset adapter module, and the noise prediction results of the pre-trained noise prediction network are obtained, and the noise prediction results are fine-tuned by the multi-scale features and the preset perceptual optimization loss function to generate the perceptual optimization features corresponding to the resampled features, including: inputting the resampled features into a preset adapter to generate the multi-scale features corresponding to the resampled features; and conditionally controlling the encoding layer and the decoding layer in the noise prediction network at the same time by the multi-scale features and the perceptual optimization loss function to fine-tune the noise prediction results and generate the perceptual optimization features corresponding to the multi-scale features.

[0010] Optionally, in one embodiment of the present application, the mathematical expression of the resampling loss function is:

[0011] L stage1 =λ1L rec. +λ2L gui.

[0012] Among them, L rec. Represents the preset reconstruction loss function; L gui. represents the preset guided item loss function; λ1 represents the weight corresponding to the reconstruction item loss function; λ2 represents the weight corresponding to the guided item loss function.

[0013] Optionally, in one embodiment of the present application, the mathematical expression of the perceptual optimization loss function is:

[0014]

[0015] Wherein, z represents the magnification feature; ∈ represents the true value noise; Represents the resampled features; ∈ θ represents the noise prediction network; z t Represents the rate characteristics with noise.

[0016] The second aspect of the present application provides an extreme magnification image resampling method based on generative latent space learning, which is applied to the online reasoning stage and includes the following steps: obtaining an image to be resampled that meets a preset high-resolution requirement, and mapping the image to be resampled to a feature space to obtain a target resolution feature corresponding to the image to be resampled; inputting the target resolution feature into a preset decoupled feature resampling network to generate a target magnification map that meets a preset low-resolution requirement and a latent space feature corresponding to the target resolution feature, and based on a preset generative diffusion model, perceptually optimizing the latent space feature to obtain a perceptual optimization feature corresponding to the latent space feature; performing an interpolation and fusion operation on the latent space feature and the perceptual optimization feature corresponding to the latent space feature to obtain a fused feature, and inputting the fused feature or the perceptual optimization feature corresponding to the latent space feature into a pre-trained decoder to output a target resampled image corresponding to the image to be resampled.

[0017] The third aspect of the present application provides an extreme magnification image resampling device based on generative latent space learning, which is applied to the offline training stage, including: an encoding module, used to obtain a first target magnification image that meets the preset high-resolution requirement, and map the first target magnification image to a feature space to obtain a magnification feature corresponding to the first target magnification image; a resampling module, used to input the magnification feature into a preset decoupled feature resampling network, and combine the preset resampling loss function to generate a second target magnification map that meets the preset low-resolution requirement and a resampling feature corresponding to the magnification feature; a perceptual optimization module, used to generate a multi-scale feature corresponding to the resampling feature according to a preset adapter module, and obtain a noise prediction result of a pre-trained noise prediction network, and fine-tune the noise prediction result through the multi-scale feature and the preset perceptual optimization loss function to generate a perceptual optimization feature corresponding to the resampling feature, and input the perceptual optimization feature into a preset decoder to output a target magnification resampled image corresponding to the first target magnification image.

[0018] Optionally, in one embodiment of the present application, the resampling module includes: a feature processing unit, which is used to perform feature processing operations on the first target magnification image and the magnification feature simultaneously based on a preset hybrid encoder to generate processing results corresponding to the first target magnification image and the magnification feature respectively; a fusion unit, which is used to fuse the processing results corresponding to the first target magnification image and the magnification feature to obtain a first intermediate expression corresponding to the first target magnification image; a quantization unit, which is used to obtain a forward function of the domain converter in the decoupled feature resampling network to convert the first intermediate expression into the pixel domain through the forward function, and perform a quantization operation on the first intermediate expression in the pixel domain to generate the second target magnification map; a first conversion unit, which is used to convert the second target magnification map through the inverse function of the domain converter to generate a second intermediate expression; a mapping unit, which is used to map the second intermediate expression to the resampling feature based on the decoder in the decoupled feature resampling network and the resampling loss function.

[0019] Optionally, in one embodiment of the present application, the perceptual optimization module includes: a generation unit, used to input the resampled features into a preset adapter to generate multi-scale features corresponding to the resampled features; a fine-tuning unit, used to simultaneously perform conditional control on the encoding layer and the decoding layer in the noise prediction network through the multi-scale features and the perceptual optimization loss function to fine-tune the noise prediction results and generate perceptual optimization features corresponding to the multi-scale features.

[0020] Optionally, in one embodiment of the present application, the mathematical expression of the resampling loss function is:

[0021] L stage1 =λ1L rec. +λ2L gui.

[0022] Among them, L rec . represents the preset reconstruction loss function; L gui. represents the preset guided item loss function; λ1 represents the weight corresponding to the reconstruction item loss function; λ2 represents the weight corresponding to the guided item loss function.

[0023] Optionally, in one embodiment of the present application, the mathematical expression of the perceptual optimization loss function is:

[0024]

[0025] Wherein, z represents the magnification feature; ∈ represents the true value noise; Represents the resampled features; ∈ θ represents the noise prediction network; zt Represents the rate characteristics with noise.

[0026] The fourth aspect of the present application provides an extreme magnification image resampling device based on generative latent space learning, which is applied to the online reasoning stage and includes the following steps: an acquisition module, used to acquire an image to be resampled that meets a preset high-resolution requirement, and map the image to be resampled to a feature space to obtain a target resolution feature corresponding to the image to be resampled; a feature optimization module, used to input the target resolution feature into a preset decoupled feature resampling network to generate a target magnification map that meets a preset low-resolution requirement and a latent space feature corresponding to the target resolution feature, and based on a preset generative diffusion model and the target magnification map, perceptually optimize the latent space feature to obtain a perceptual optimization feature corresponding to the latent space feature; a decoding module, used to perform an interpolation and fusion operation on the latent space feature and the perceptual optimization feature corresponding to the latent space feature to obtain a fused feature, and input the fused feature or the perceptual optimization feature corresponding to the latent space feature into a pre-trained decoder to output a target resampled image corresponding to the image to be resampled.

[0027] The fifth aspect of the present application provides an electronic device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the extreme magnification image resampling method based on generative latent space learning as described in the above embodiments.

[0028] The sixth aspect embodiment of the present application provides a computer-readable storage medium, which stores a computer program, which, when executed by a processor, implements the above-mentioned extreme magnification image resampling method based on generative latent space learning.

[0029] Therefore, the embodiments of the present application have the following beneficial effects:

[0030] The embodiment of the present application can obtain a first target magnification image that meets the preset high-resolution requirement, and map the first target magnification image to the feature space to obtain the magnification feature corresponding to the first target magnification image; input the magnification feature into the preset decoupled feature resampling network, and combine the preset resampling loss function to generate a second target magnification map that meets the preset low-resolution requirement and the resampling feature corresponding to the magnification feature; according to the preset adapter module, generate multi-scale features corresponding to the resampling feature, and obtain the noise prediction result of the pre-trained noise prediction network, and fine-tune the noise prediction result through the multi-scale feature and the preset perceptual optimization loss function to generate the perceptual optimization feature corresponding to the resampling feature, and input the perceptual optimization feature into the preset decoder to output the target magnification resampling image corresponding to the first target magnification image. The present application transfers the resampling operation from the pixel space to the latent space where a pre-trained autoencoder is located, thereby generating more realistic and human-vision-compliant texture details and achieving a balance between fidelity and authenticity. This solves the problem that the existing technology is difficult to generate sufficient details and accurate semantic structures when resampling images at extreme magnifications, and easily causes the model to fall into a suboptimal training state, greatly affecting the perceptual quality of the resampling results.

[0031] Additional aspects and advantages of the present application will be given in part in the description below, and in part will become apparent from the description below, or will be learned through the practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0033] Figure 1 A flowchart of an extreme magnification image resampling method based on generative latent space learning applied in an offline training phase according to an embodiment of the present application;

[0034] Figure 2 A schematic diagram of the execution logic of a two-stage training process provided for one embodiment of the present application;

[0035] Figure 3 A schematic diagram of the structure of a decoupled feature resampling network provided for one embodiment of the present application;

[0036] Figure 4 A schematic diagram of the main resampling process of an extreme magnification image resampling method based on generative latent space learning provided in one embodiment of the present application;

[0037] Figure 5 A schematic diagram of the logic architecture of an extreme magnification image resampling method based on generative latent space learning provided for one embodiment of the present application;

[0038] Figure 6 A flowchart of an extreme magnification image resampling method based on generative latent space learning applied to an online reasoning stage according to an embodiment of the present application;

[0039] Figure 7 This is an example diagram of an extreme magnification image resampling device based on generative latent space learning applied in an offline training phase according to an embodiment of the present application;

[0040] Figure 8 This is an example diagram of an extreme magnification image resampling device based on generative latent space learning applied to an online reasoning stage according to an embodiment of the present application;

[0041] Fig. 9 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application.

[0042] Among them, 10-an extreme ratio image resampling device based on generative latent space learning applied in the offline training stage, 20-an extreme ratio image resampling device based on generative latent space learning applied in the online reasoning stage; 101-an encoding module, 102-a resampling module, 103-a perception optimization module; 201-an acquisition module, 202-a feature optimization module, 203-a decoding module; 901-a memory, 902-a processor, 903-a communication interface. DETAILED DESCRIPTION

[0043] The embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present application, and should not be construed as limiting the present application.

[0044] The following describes the extreme magnification image resampling method and device based on generative latent space learning of the embodiment of the present application with reference to the accompanying drawings. In view of the problems mentioned in the above background technology, the present application provides an extreme magnification image resampling method based on generative latent space learning, in which a first target magnification image that meets the preset high resolution requirement is obtained, and the first target magnification image is mapped to the feature space to obtain the magnification feature corresponding to the first target magnification image; the magnification feature is input into the preset decoupled feature resampling network, and combined with the preset resampling loss function to generate a second target magnification map that meets the preset low resolution requirement and a resampling feature corresponding to the magnification feature; according to the preset adapter module, a multi-scale feature corresponding to the resampling feature is generated, and the noise prediction result of the pre-trained noise prediction network is obtained, and the noise prediction result is fine-tuned by the multi-scale feature and the preset perceptual optimization loss function to generate a perceptual optimization feature corresponding to the resampling feature, and the perceptual optimization feature is input into the preset decoder to output the target magnification resampling image corresponding to the first target magnification image. This application transfers the resampling operation from the pixel space to the latent space where a pre-trained autoencoder is located, thereby generating more realistic texture details that conform to human vision and achieving a balance between fidelity and authenticity. This solves the problem that the existing technology is difficult to generate sufficient details and accurate semantic structures when resampling images at extreme magnifications, and it is easy to cause the model to fall into a suboptimal training state, greatly affecting the perceived quality of the resampling results.

[0045] Specifically, Figure 1 A flowchart of an extreme magnification image resampling method based on generative latent space learning applied in an offline training phase provided in an embodiment of the present application.

[0046] like Figure 1 As shown, the extreme magnification image resampling method based on generative latent space learning includes the following steps:

[0047] In step S101, a first target magnification image that meets a preset high resolution requirement is acquired, and the first target magnification image is mapped to a feature space to obtain a magnification feature corresponding to the first target magnification image.

[0048] In step S102, the magnification feature is input into a preset decoupled feature resampling network, and combined with a preset resampling loss function to generate a second target magnification map that meets the preset low-resolution requirement and a resampling feature corresponding to the magnification feature.

[0049] The embodiment of the present application can first obtain a high-resolution image (i.e., a first target magnification image) x∈R H×W×3 , and use the pre-trained encoder ε to map it from the pixel space to the feature space to obtain high-resolution features (i.e., magnification features) Where s represents the encoder downsampling ratio, and C represents the number of encoder output channels.

[0050] Secondly, the embodiment of the present application can input the high-resolution feature z into the decoupled feature rescaling module (DFRM), thereby simultaneously obtaining a low-resolution image (i.e., the second target magnification map) (where N represents the target downsampling ratio) and the resampling characteristics Provide reliable data guidance and basis for achieving extreme magnification image resampling.

[0051] Optionally, in one embodiment of the present application, the magnification feature is input into a preset decoupled feature resampling network, and combined with a preset resampling loss function to generate a second target magnification map that meets the preset low resolution requirement and a resampling feature corresponding to the magnification feature, including simultaneously performing feature processing operations on the first target magnification image and the magnification feature based on a preset hybrid encoder to respectively generate the first target magnification image and the processing results corresponding to the magnification feature; fusing the processing results corresponding to the first target magnification image and the magnification feature to obtain a first intermediate expression corresponding to the first target magnification image; obtaining a forward function of the domain converter in the decoupled feature resampling network to convert the first intermediate expression into the pixel domain through the forward function, and performing a quantization operation on the first intermediate expression in the pixel domain to generate a second target magnification map; converting the second target magnification map through the inverse function of the domain converter to generate a second intermediate expression; and mapping the second intermediate expression to a resampling feature based on the decoder and resampling loss function in the decoupled feature resampling network.

[0052] It should be noted that if Figure 2 As shown, the embodiment of the present application adopts a two-stage training strategy, corresponding to the feature resampling stage (ie, the first stage) and the perception optimization stage (ie, the second stage).

[0053] In the first stage, the embodiment of the present application can construct a decoupled feature resampling network, whose specific structure is as follows: Figure 3 As shown in the figure, the network includes an asymmetric encoder-decoder and a set of domain converters composed of reversible neural networks, wherein the encoder-decoder is used to resample the input high-resolution features to the target size; and the domain converter is used to convert the low-resolution features into low-resolution images; the decoupled feature resampling network can simultaneously accept high-resolution images x and high-resolution features z to perform resampling operations, and output low-resolution images y and resampled features

[0054] Specifically, the execution steps of the decoupled feature resampling network are as follows:

[0055] Step 1: Use a hybrid encoder G e Process the high-resolution image x and high-resolution feature z at the same time, and fully integrate the information of the two to obtain an intermediate expression (i.e., the first intermediate expression)

[0056] Step 2: Use the forward function F of the domain converter to convert the intermediate expression w to the pixel domain, and output it as a low-resolution image (i.e., the second target magnification image) after quantization. Then, the inverse function F of the domain converter is used -1 Reconvert the low-resolution image y into an intermediate representation (i.e., the second intermediate representation)

[0057] Step 3: Use decoder G d The intermediate expression Remap to the resampled features, i.e., resampled features

[0058] Therefore, the embodiments of the present application complete the bidirectional mapping between the potential features of the image and the low-resolution image of the target size by decoupling the feature resampling operation, so that a resampling result with higher fidelity can be output, thereby solving the coupling relationship between the resampling operation and the feature-pixel mapping operation.

[0059] Optionally, in one embodiment of the present application, the mathematical expression of the resampling loss function is:

[0060] L stage1 =λ1L rec. +λ2L gui.

[0061] Among them, L rec. Represents the preset reconstruction loss function; L gui . represents the preset guided item loss function; λ1 represents the weight corresponding to the reconstruction item loss function; λ2 represents the weight corresponding to the guided item loss function.

[0062] In the actual implementation process, the loss function (i.e., the resampling loss function) used in the first stage of the embodiment of the present application can be expressed as:

[0063] L stage1 =λ1L rec. +λ2L gui.

[0064] Among them, L rec. represents the preset reconstruction loss function, which can constrain the network to have the ability to accurately reconstruct the high-resolution feature z based on the intermediate expression w with or without quantization; L gui.Represents the preset guided loss function, which can align the content of the low-resolution image y and the high-resolution image x without affecting the resampling operation as much as possible; λ1 represents the weight corresponding to the reconstruction loss function; λ2 represents the weight corresponding to the guided loss function.

[0065] The mathematical expressions of the above reconstruction loss function and guidance loss function are:

[0066] L rec. =|G d (G e (x,z))-z|+|G d (F -1 (F(G g (x,z))))-z|

[0067] L gui. =|sg(F(G e (x,z)))-Bicubic(x))|

[0068] Here, sg(·) represents the gradient clipping operation.

[0069] Therefore, the embodiment of the present application trains the DFRM in the first stage, and combines the resampling loss function constructed by the reconstruction term loss function and the guide term loss function, so as to promote the resampled features The goal is to get as close as possible to the input high-resolution features z, while ensuring that the low-resolution image y shares the same content with the original high-resolution image x.

[0070] In step S103, according to the preset adapter module, multi-scale features corresponding to the resampling features are generated, and the noise prediction results of the pre-trained noise prediction network are obtained, and the noise prediction results are fine-tuned by multi-scale features and a preset perceptual optimization loss function to generate perceptual optimization features corresponding to the resampling features, and the perceptual optimization features are input into a preset decoder to output a target magnification resampled image corresponding to the first target magnification image.

[0071] Furthermore, the embodiments of the present application can use a pre-trained generative diffusion model to resample features. Perform perceptual optimization, that is, the embodiment of the present application can resample the features Input into an adapter Ada to generate multi-scale features, and use these features to fine-tune the pre-trained noise prediction network ∈ θ The noise prediction result is used to generate the perceptual optimization feature corresponding to the resampled feature; then, the embodiment of the present application can generate the perceptual optimization feature Feed into the pre-trained decoder In order to obtain the resampled high-resolution image (i.e. the target magnification resampled image)

[0072] Therefore, the embodiment of the present application is to predict the noise network ε θ And the adapter module Ada is trained to promote the perception of optimized features as close as possible to the input high-resolution feature z.

[0073] Optionally, in one embodiment of the present application, according to a preset adapter module, multi-scale features corresponding to the resampled features are generated, and the noise prediction results of the pre-trained noise prediction network are obtained, and the noise prediction results are fine-tuned by multi-scale features and a preset perceptual optimization loss function to generate perceptual optimization features corresponding to the resampled features, including: inputting the resampled features into a preset adapter to generate multi-scale features corresponding to the resampled features; conditionally controlling the encoding layer and the decoding layer in the noise prediction network at the same time by the multi-scale features and the perceptual optimization loss function to fine-tune the noise prediction results and generate perceptual optimization features corresponding to the multi-scale features.

[0074] It should be noted that the first stage of decoupled feature resampling can already output a resampling result with high fidelity, but it lacks the constraints of prior knowledge and can often only obtain a relatively fuzzy reconstruction result. Therefore, the embodiment of the present application can design a second stage of training to perform perceptual optimization on the resampling result.

[0075] In the second stage, in order to solve the extremely pathological problems existing in extreme-rate resampling, the embodiment of the present application introduces a pre-trained generative diffusion model as prior knowledge to restrict the upsampling process, thereby solving the problems of insufficient detail texture and semantic errors caused by conventional upsampling algorithms.

[0076] Specifically, the execution process of the second stage of the perception optimization process is as follows:

[0077] Step 1: Resample features Input into the adapter Ada to generate the same noise prediction network ∈ θ Adaptive multi-scale features:

[0078]

[0079] in, represents the extracted features at the i-th scale; K represents the number of scales;

[0080] Step 2: Multi-scale features are added to the pre-trained noise prediction network ∈ θIn the output of each layer of , thereby assisting it in performing the diffusion denoising process; considering that the image resampling task has a high requirement for fidelity, the embodiment of the present application performs a noise prediction network ∈ θ The encoding layer and decoding layer of are controlled conditionally at the same time:

[0081]

[0082] in, and They represent the noise prediction network ∈ θ The encoding and decoding layers are at the output of the i-th layer.

[0083] Therefore, the embodiments of the present application retain more complete semantic information about the high-resolution image by transferring the resampling operation from the pixel space to the latent space of a pre-trained autoencoder; at the same time, the pre-trained generative diffusion model is used to further enhance the perceptual quality of the reconstruction result and refine the features of the first stage reconstruction, thereby generating more realistic texture details that conform to human vision.

[0084] Optionally, in one embodiment of the present application, the mathematical expression of the perceptual optimization loss function is:

[0085]

[0086] Among them, z represents the rate feature; ∈ represents the true value noise; Represents the resampled features; ∈ θ represents the noise prediction network; z t Represents the rate characteristics with noise.

[0087] In the specific implementation process, according to the relevant theory of the denoising diffusion probability model, the loss function of the embodiment of the present application in the second stage (ie, the perceptual optimization loss function) can be expressed as:

[0088]

[0089] Among them, z represents the rate feature; ∈ represents the true value noise; Represents the resampled features; ∈ θ represents the noise prediction network; z t Represents a noisy magnification feature (i.e., a noisy high-resolution feature).

[0090] Therefore, the embodiments of the present application predict the distance between the predicted noise and the actual noise by constraining the noise, so as to output a reconstruction result with higher fidelity.

[0091] It should be noted that, in response to the current problems of insufficient texture and semantic errors in the field of extreme-magnification image resampling, the embodiments of the present application transfer the resampling operation from the pixel space to the latent space where a pre-trained autoencoder is located by decoupling the two processes of feature resampling and perceptual optimization, wherein the decoupled feature resampling network accepts high-resolution features as input and outputs low-resolution images and resampled features at the same time; the perceptual optimization process takes the resampled features as input and makes full use of the powerful priors stored in the pre-trained generative diffusion to generate rich details and accurate semantics.

[0092] Those skilled in the art should understand that traditional resampling methods are performed directly in the pixel space, which usually leads to blurry reconstruction results; at the same time, some methods based on pre-trained generative models introduce feature loss in the latent space, but also introduce pixel loss; during training, these two losses will produce a certain adversarial effect, resulting in suboptimal results.

[0093] Therefore, one of the core innovations of the embodiment of the present application is to transfer the resampling operation from the pixel space to the latent space where the pre-trained autoencoder is located, such as Figure 4 As shown, the resampling operation is performed completely in the latent space. Therefore, the embodiment of the present application has two advantages:

[0094] 1. The resampling operator in the latent space has a larger receptive field, thus allowing more information that is beneficial to upsampling to be embedded in the low-resolution image;

[0095] 2. The resampling operation in the latent space is conducive to maintaining the semantic integrity of the resampled features, thereby better guiding the generation process of the diffusion model.

[0096] The following is a specific example to illustrate the two-stage training process and effect of the extreme magnification image resampling method based on generative latent space learning applied to the offline training stage of the present application.

[0097] A specific embodiment of the present application may use DIV2K and Flickr2K as training data. Since traditional image restoration datasets such as Set5, Set14, and BSD100 have low resolution and are not suitable for evaluating image resampling methods at extreme magnifications, the specific embodiment of the present application may use four other high-resolution datasets (i.e., DIV2K, Urban100, CLIC2020, and DIV8K) as test sets to evaluate the performance of the model:

[0098] In the first stage, the specific embodiment of the present application freezes the pre-trained autoencoder and iteratively trains the DFRM for 200K times; the initial learning rate is set to 2e-4 and is halved every 50K iterations; after completing the training of DFRM, the noise prediction network and the adapter network can be trained. In order to balance the training cost and model performance, the specific embodiment of the present application only trains the linear layer in the noise prediction network; in the second stage, a constant learning rate of 5e-5 is used for iterative training for 150K times; for the two-stage training process, the specific embodiment of the present application uses Adam as the optimizer.

[0099] The specific embodiments of the present application can use pixel-level indicators PSNR, SSIM, perception indicators LPIPS, DISTS, and reference-free indicators MUSIQ and CLIPIQA and other image quality assessment indicators to quantitatively evaluate the effectiveness of the extreme magnification image resampling method based on generative latent space learning of the present application. The experimental results of the above four data sets and two resampling ratios (16 times and 32 times) also show the superiority of the extreme magnification image resampling method based on generative latent space learning of the present application.

[0100] In summary, if Figure 5 As shown, different from the encoding-decoding scheme of the traditional resampling method, the embodiment of the present application mainly includes two operations of decoupled feature resampling and conditional diffusion model perceptual optimization: first, the input high-resolution image is encoded into the latent space to perform the resampling operation; secondly, the feature resampling operation accepts the high-resolution features, downsamples them to the target size and maps them to the pixel space, and maps them back to the original size; then, the resampled features are sent to the conditional diffusion model for optimization to improve the perceptual quality of the reconstruction results; finally, the perceptually optimized features are sent to the decoder and remapped back to the pixel space, thereby obtaining the resampled high-resolution image.

[0101] Therefore, the embodiments of the present application can make full use of the powerful natural image priors stored in the pre-trained generative diffusion model to alleviate the pathological characteristics of image resampling; at the same time, the embodiments of the present application transfer the resampling process from the pixel space to the latent space of a pre-trained autoencoder, thereby providing better perceptual reconstruction quality.

[0102] According to the extreme magnification image resampling method based on generative latent space learning applied to the offline training stage proposed in the embodiment of the present application, a first target magnification image that meets the preset high resolution requirement is obtained, and the first target magnification image is mapped to the feature space to obtain the magnification feature corresponding to the first target magnification image; the magnification feature is input into the preset decoupled feature resampling network, and combined with the preset resampling loss function, to generate a second target magnification map that meets the preset low resolution requirement and the resampling feature corresponding to the magnification feature; according to the preset adapter module, a multi-scale feature corresponding to the resampling feature is generated, and the noise prediction result of the pre-trained noise prediction network is obtained, and the noise prediction result is fine-tuned by the multi-scale feature and the preset perceptual optimization loss function to generate the perceptual optimization feature corresponding to the resampling feature, and the perceptual optimization feature is input into the preset decoder to output the target magnification resampling image corresponding to the first target magnification image. The present application transfers the resampling operation from the pixel space to the latent space where a pre-trained autoencoder is located, thereby generating more realistic and human-vision-compliant texture details and achieving a balance between fidelity and authenticity.

[0103] Figure 6 A flowchart of an extreme magnification image resampling method based on generative latent space learning applied to the online reasoning stage provided in an embodiment of the present application.

[0104] like Figure 6 As shown, the extreme magnification image resampling method based on generative latent space learning includes the following steps:

[0105] In step S601, an image to be resampled that meets a preset high resolution requirement is obtained, and the image to be resampled is mapped to a feature space to obtain a target resolution feature corresponding to the image to be resampled.

[0106] In step S602, the target resolution feature is input into a preset decoupled feature resampling network to generate a target magnification map that meets the preset low resolution requirements and a latent space feature corresponding to the target resolution feature, and based on a preset generative diffusion model, the latent space feature is perceptually optimized to obtain a perceptually optimized feature corresponding to the latent space feature.

[0107] In step S603, an interpolation fusion operation is performed on the latent space features and the perceptual optimization features corresponding to the latent space features to obtain fused features, and the fused features or the perceptual optimization features corresponding to the latent space features are input into a pre-trained decoder to output a target resampled image corresponding to the image to be resampled.

[0108] It should be noted that after completing the two-stage training process, the embodiment of the present application can jointly perform reasoning using the decoupled feature resampling model and the pre-trained generative diffusion model.

[0109] Specifically, the reasoning process of the embodiment of the present application mainly includes the following steps:

[0110] Step 1: Given a high-resolution image (i.e., image to be resampled) x∈R H×W×3 , use the pre-trained encoder ε to map it from pixel space to feature space to obtain high-resolution features (i.e., target resolution features)

[0111] Step 2: Input the high-resolution image x and high-resolution feature z into the decoupled feature resampling network, and obtain the low-resolution image (i.e., the target magnification map) at the same time. And the resampled features (i.e. latent space features)

[0112]

[0113] Step 3: Use the pre-trained generative diffusion model to resample features Perform perceptual optimization to obtain perceptual optimization features Make the decoded image closer to the manifold where the natural image is located;

[0114] Step 4: Optimize the perceptual features Feed into the pre-trained decoder In order to obtain the resampled high-resolution image (i.e., the target resampled image) At the same time, the embodiment of the present application can also output the latent space features of DFRM and perceptually optimized features Interpolation fusion is performed to obtain fused features, and the fused features are input into the decoder to output the target resampled image corresponding to the image to be resampled, thereby achieving a smooth transition between fidelity and authenticity of the upsampling result.

[0115] According to the extreme magnification image resampling method based on generative latent space learning applied to the online reasoning stage proposed in the embodiment of the present application, by obtaining the image to be resampled that meets the preset high-resolution requirement, and mapping the image to be resampled to the feature space, a target resolution feature corresponding to the image to be resampled is obtained; the target resolution feature is input into the preset decoupled feature resampling network to generate a target magnification map that meets the preset low-resolution requirement and a latent space feature corresponding to the target resolution feature, and based on the preset generative diffusion model, the latent space feature is perceptually optimized to obtain a perceptual optimization feature corresponding to the latent space feature; the latent space feature and the perceptual optimization feature corresponding to the latent space feature are interpolated and fused to obtain a fused feature, and the fused feature or the perceptual optimization feature corresponding to the latent space feature is input into a pre-trained decoder to output a target resampled image corresponding to the image to be resampled. The present application generates more realistic texture details that conform to human vision by transferring the resampling operation from the pixel space to the latent space where a pre-trained autoencoder is located, thereby achieving a balance between fidelity and authenticity.

[0116] Secondly, the extreme magnification image resampling device based on generative latent space learning proposed in accordance with an embodiment of the present application is described with reference to the accompanying drawings.

[0117] Figure 7 It is a block diagram of an extreme ratio image resampling device based on generative latent space learning applied in an offline training stage according to an embodiment of the present application.

[0118] like Figure 7 As shown, the extreme magnification image resampling device 10 based on generative latent space learning includes: an encoding module 101, a resampling module 102 and a perceptual optimization module 103.

[0119] The encoding module 101 is used to obtain a first target magnification image that meets a preset high resolution requirement, and map the first target magnification image to a feature space to obtain a magnification feature corresponding to the first target magnification image.

[0120] The resampling module 102 is used to input the magnification feature into a preset decoupled feature resampling network, and combine it with a preset resampling loss function to generate a second target magnification map that meets the preset low-resolution requirement and a resampling feature corresponding to the magnification feature.

[0121] The perceptual optimization module 103 is used to generate multi-scale features corresponding to the resampled features according to the preset adapter module, and obtain the noise prediction results of the pre-trained noise prediction network, and fine-tune the noise prediction results through the multi-scale features and the preset perceptual optimization loss function to generate perceptual optimization features corresponding to the resampled features, and input the perceptual optimization features into a preset decoder to output a target magnification resampled image corresponding to the first target magnification image.

[0122] Optionally, in one embodiment of the present application, the resampling module 102 includes: a feature processing unit, a fusion unit, a quantization unit, a first conversion unit and a mapping unit.

[0123] Among them, the feature processing unit is used to perform feature processing operations on the first target magnification image and the magnification feature simultaneously based on a preset hybrid encoder to generate processing results corresponding to the first target magnification image and the magnification feature respectively.

[0124] The fusion unit is used to fuse the first target magnification image and the processing results corresponding to the magnification feature to obtain a first intermediate expression corresponding to the first target magnification image.

[0125] A quantization unit is used to obtain a forward function of a domain converter in a decoupled feature resampling network to convert the first intermediate expression into a pixel domain through the forward function, and to perform a quantization operation on the first intermediate expression in the pixel domain to generate a second target magnification map.

[0126] The first conversion unit is used to convert the second target magnification map through an inverse function of the domain converter to generate a second intermediate expression.

[0127] A mapping unit is used to map the second intermediate expression into a resampled feature based on a decoder and a resampling loss function in a decoupled feature resampling network.

[0128] Optionally, in one embodiment of the present application, the perceptual optimization module 300 includes: a generation unit and a fine-tuning unit.

[0129] The generating unit is used to input the resampled features into a preset adapter to generate multi-scale features corresponding to the resampled features.

[0130] The fine-tuning unit is used to simultaneously conditionally control the encoding layer and the decoding layer in the noise prediction network through multi-scale features and the perceptual optimization loss function to fine-tune the noise prediction results and generate perceptual optimization features corresponding to the multi-scale features.

[0131] Optionally, in one embodiment of the present application, the mathematical expression of the resampling loss function is:

[0132] L stage1 =λ1Lrec .+λ2L gui .

[0133] Among them, L rec. Represents the preset reconstruction loss function; L gui. represents the preset guided item loss function; λ1 represents the weight corresponding to the reconstruction item loss function; λ2 represents the weight corresponding to the guided item loss function.

[0134] Optionally, in one embodiment of the present application, the mathematical expression of the perceptual optimization loss function is:

[0135]

[0136] Among them, z represents the rate feature; ∈ represents the true value noise; Represents the resampled features; ∈ θ represents the noise prediction network; z t Represents the rate characteristics with noise.

[0137] It should be noted that the aforementioned explanation of the embodiment of the extreme ratio image resampling method based on generative latent space learning applied to the offline training stage is also applicable to the extreme ratio image resampling device based on generative latent space learning applied to the offline training stage of this embodiment, and will not be repeated here.

[0138] According to the embodiment of the present application, an extreme magnification image resampling device based on generative latent space learning applied to the offline training stage includes an encoding module for obtaining a first target magnification image that meets the preset high resolution requirement, and mapping the first target magnification image to a feature space to obtain a magnification feature corresponding to the first target magnification image; a resampling module for inputting the magnification feature into a preset decoupled feature resampling network, and combining a preset resampling loss function to generate a second target magnification map that meets the preset low resolution requirement and a resampling feature corresponding to the magnification feature; a perceptual optimization module for generating multi-scale features corresponding to the resampling feature according to a preset adapter module, and obtaining the noise prediction result of the pre-trained noise prediction network, and fine-tuning the noise prediction result through the multi-scale feature and the preset perceptual optimization loss function to generate a perceptual optimization feature corresponding to the resampling feature, and inputting the perceptual optimization feature into a preset decoder to output a target magnification resampled image corresponding to the first target magnification image. The present application transfers the resampling operation from the pixel space to a latent space where a pre-trained autoencoder is located, thereby generating more realistic and human-vision-compliant texture details and achieving a balance between fidelity and authenticity.

[0139] Figure 8It is a block diagram of an extreme ratio image resampling device based on generative latent space learning applied to the online reasoning stage of an embodiment of the present application.

[0140] like Figure 8 As shown, the extreme magnification image resampling device 20 based on generative latent space learning applied in the online reasoning stage includes: an acquisition module 201, a feature optimization module 202 and a decoding module 203.

[0141] The acquisition module 201 is used to acquire the image to be resampled that meets the preset high resolution requirement, and map the image to be resampled to the feature space to obtain the target resolution feature corresponding to the image to be resampled.

[0142] The feature optimization module 202 is used to input the target resolution feature into a preset decoupled feature resampling network to generate a target magnification map that meets the preset low resolution requirements and a latent space feature corresponding to the target resolution feature, and based on the preset generative diffusion model and the target magnification map, perform perceptual optimization on the latent space feature to obtain a perceptual optimization feature corresponding to the latent space feature.

[0143] The decoding module 203 is used to perform interpolation and fusion operations on the latent space features and the perceptual optimization features corresponding to the latent space features to obtain fused features, and input the fused features or the perceptual optimization features corresponding to the latent space features into a pre-trained decoder to output a target resampled image corresponding to the image to be resampled.

[0144] It should be noted that the aforementioned explanation of the embodiment of the extreme ratio image resampling method based on generative latent space learning applied to the online reasoning stage is also applicable to the extreme ratio image resampling device based on generative latent space learning applied to the online reasoning stage of this embodiment, and will not be repeated here.

[0145] According to the embodiment of the present application, an extreme magnification image resampling device based on generative latent space learning applied to the online reasoning stage includes an acquisition module for acquiring an image to be resampled that meets the preset high-resolution requirement, and mapping the image to be resampled to a feature space to obtain a target resolution feature corresponding to the image to be resampled; a feature optimization module for inputting the target resolution feature into a preset decoupled feature resampling network to generate a target magnification map that meets the preset low-resolution requirement and a latent space feature corresponding to the target resolution feature, and based on a preset generative diffusion model and a target magnification map, perceptually optimizing the latent space feature to obtain a perceptual optimization feature corresponding to the latent space feature; a decoding module for interpolating and fusing the latent space feature and the perceptual optimization feature corresponding to the latent space feature to obtain a fused feature, and inputting the fused feature or the perceptual optimization feature corresponding to the latent space feature into a pre-trained decoder to output a target resampled image corresponding to the image to be resampled. The present application achieves a balance between fidelity and authenticity by transferring the resampling operation from the pixel space to a latent space where a pre-trained autoencoder is located, thereby generating texture details that are more realistic and in line with human vision.

[0146] Fig. 9 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. The electronic device may include:

[0147] A memory 901 , a processor 902 , and a computer program stored in the memory 901 and executable on the processor 902 .

[0148] When the processor 902 executes the program, the extreme magnification image resampling method based on generative latent space learning provided in the above embodiment is implemented.

[0149] Furthermore, the electronic device further comprises:

[0150] The communication interface 903 is used for communication between the memory 901 and the processor 902 .

[0151] The memory 901 is used to store computer programs that can be executed on the processor 902 .

[0152] The memory 901 may include a high-speed RAM memory, and may also include a non-volatile memory (non-volatile memory), such as at least one disk memory.

[0153] If the memory 901, the processor 902 and the communication interface 903 are implemented independently, the communication interface 903, the memory 901 and the processor 902 can be connected to each other through a bus and communicate with each other. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component (PCI) bus or an Extended Industry Standard Architecture (EISA) bus. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Fig. 9 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0154] Optionally, in a specific implementation, if the memory 901, the processor 902 and the communication interface 903 are integrated on a chip, the memory 901, the processor 902 and the communication interface 903 can communicate with each other through an internal interface.

[0155] The processor 902 may be a central processing unit (CPU), or an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present application.

[0156] An embodiment of the present application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the above-mentioned extreme magnification image resampling method based on generative latent space learning.

[0157] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or N embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.

[0158] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include at least one of the features. In the description of this application, "N" means at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0159] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, fragment or portion of code comprising one or N executable instructions for implementing the steps of a custom logical function or process, and the scope of the preferred embodiments of the present application includes alternative implementations in which functions may not be performed in the order shown or discussed, including performing functions in a substantially simultaneous manner or in reverse order depending on the functions involved, which should be understood by technicians in the technical field to which the embodiments of the present application belong.

[0160] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or in combination with these instruction execution systems, devices or apparatuses. For the purpose of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in combination with these instruction execution systems, devices or apparatuses. More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or N wirings (electronic devices), a portable computer disk box (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), a fiber optic device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically by optically scanning the paper or other medium and then editing, interpreting or processing in other suitable ways as necessary and then storing it in a computer memory.

[0161] It should be understood that the various parts of the present application can be implemented by hardware, software, firmware or a combination thereof. In the above embodiment, the N steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. If implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0162] A person skilled in the art may understand that all or part of the steps in the method for implementing the above-mentioned embodiment may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, which, when executed, includes one or a combination of the steps of the method embodiment.

[0163] In addition, each functional unit in each embodiment of the present application may be integrated into a processing module, or each unit may exist physically separately, or two or more units may be integrated into one module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer-readable storage medium.

[0164] The storage medium mentioned above may be a read-only memory, a magnetic disk or an optical disk, etc. Although the embodiments of the present application have been shown and described above, it can be understood that the above embodiments are exemplary and cannot be understood as limiting the present application. A person of ordinary skill in the art may change, modify, replace and modify the above embodiments within the scope of the present application.

Claims

1. A method for extreme magnification image resampling based on generative latent space learning, applied in the offline training stage, characterized in that: The following steps are involved: Acquire a first target magnification image that meets a preset high resolution requirement, and map the first target magnification image to a feature space to obtain a magnification feature corresponding to the first target magnification image; Inputting the magnification feature into a preset decoupled feature resampling network, and combining it with a preset resampling loss function to generate a second target magnification map that meets a preset low-resolution requirement and a resampling feature corresponding to the magnification feature; According to a preset adapter module, a multi-scale feature corresponding to the resampled feature is generated, and a noise prediction result of a pre-trained noise prediction network is obtained, and the noise prediction result is fine-tuned by the multi-scale feature and a preset perceptual optimization loss function to generate a perceptual optimization feature corresponding to the resampled feature, and the perceptual optimization feature is input into a preset decoder to output a target magnification resampled image corresponding to the first target magnification image.

2. The method according to claim 1, characterized in that The step of inputting the magnification feature into a preset decoupled feature resampling network and combining it with a preset resampling loss function to generate a second target magnification map that meets a preset low-resolution requirement and a resampling feature corresponding to the magnification feature includes: Based on a preset hybrid encoder, a feature processing operation is simultaneously performed on the first target magnification image and the magnification feature to generate processing results corresponding to the first target magnification image and the magnification feature respectively; fusing the first target magnification image and the processing results corresponding to the magnification feature to obtain a first intermediate expression corresponding to the first target magnification image; Obtaining a forward function of a domain converter in the decoupled feature resampling network, so as to convert the first intermediate expression into a pixel domain through the forward function, and performing a quantization operation on the first intermediate expression in the pixel domain to generate the second target magnification map; Converting the second target ratio map by an inverse function of the domain converter to generate a second intermediate expression; Based on a decoder in the decoupled feature resampling network and the resampling loss function, the second intermediate expression is mapped to the resampled feature.

3. The method according to claim 2, characterized in that The method of generating a multi-scale feature corresponding to the resampled feature according to a preset adapter module, obtaining a noise prediction result of a pre-trained noise prediction network, and fine-tuning the noise prediction result by using the multi-scale feature and a preset perceptual optimization loss function to generate a perceptual optimization feature corresponding to the resampled feature includes: Inputting the resampled features into a preset adapter to generate multi-scale features corresponding to the resampled features; The encoding layer and the decoding layer in the noise prediction network are conditionally controlled simultaneously through the multi-scale features and the perceptual optimization loss function to fine-tune the noise prediction result and generate perceptual optimization features corresponding to the multi-scale features.

4. The method according to claim 1, characterized in that: The mathematical expression of the resampling loss function is: L stage1 =λ1L rec. +λ2L gui . Among them, L rec. Represents the preset reconstruction loss function; L gui. represents the preset guided item loss function; λ1 represents the weight corresponding to the reconstruction item loss function; λ2 represents the weight corresponding to the guided item loss function.

5. The method according to claim 1, characterized in that The mathematical expression of the perceptual optimization loss function is: Wherein, z represents the magnification feature; ∈ represents the true value noise; Represents the resampled features; ∈ θ represents the noise prediction network; z t Represents the rate characteristics with noise.

6. A method for extreme magnification image resampling based on generative latent space learning, applied to the online reasoning stage, characterized in that: The following steps are involved: Acquire an image to be resampled that meets a preset high resolution requirement, and map the image to be resampled to a feature space to obtain a target resolution feature corresponding to the image to be resampled; The target resolution feature is input into a preset decoupled feature resampling network to generate a target magnification map that meets a preset low resolution requirement and a latent space feature corresponding to the target resolution feature, and based on a preset generative diffusion model, the latent space feature is perceptually optimized to obtain a perceptually optimized feature corresponding to the latent space feature; An interpolation and fusion operation is performed on the latent space feature and the perceptual optimization feature corresponding to the latent space feature to obtain a fused feature, and the fused feature or the perceptual optimization feature corresponding to the latent space feature is input into a pre-trained decoder to output a target resampled image corresponding to the image to be resampled.

7. An extreme magnification image resampling device based on generative latent space learning, applied in the offline training stage, characterized in that: include: An encoding module, used for acquiring a first target magnification image that meets a preset high resolution requirement, and mapping the first target magnification image to a feature space to obtain a magnification feature corresponding to the first target magnification image; A resampling module, used to input the magnification feature into a preset decoupled feature resampling network, and combine it with a preset resampling loss function to generate a second target magnification map that meets the preset low-resolution requirement and a resampling feature corresponding to the magnification feature; A perceptual optimization module is used to generate a multi-scale feature corresponding to the resampled feature according to a preset adapter module, and obtain a noise prediction result of a pre-trained noise prediction network, and fine-tune the noise prediction result through the multi-scale feature and a preset perceptual optimization loss function to generate a perceptual optimization feature corresponding to the resampled feature, and input the perceptual optimization feature into a preset decoder to output a target magnification resampled image corresponding to the first target magnification image.

8. An extreme magnification image resampling device based on generative latent space learning, applied to the online reasoning stage, characterized in that: include: An acquisition module is used to acquire an image to be resampled that meets a preset high-resolution requirement, and map the image to be resampled to a feature space to obtain a target resolution feature corresponding to the image to be resampled; A feature optimization module, used for inputting the target resolution feature into a preset decoupled feature resampling network to generate a target magnification map that meets a preset low-resolution requirement and a latent space feature corresponding to the target resolution feature, and performing perceptual optimization on the latent space feature based on a preset generative diffusion model and the target magnification map to obtain a perceptual optimization feature corresponding to the latent space feature; A decoding module is used to perform interpolation and fusion operations on the latent space features and the perceptual optimization features corresponding to the latent space features to obtain fused features, and input the fused features or the perceptual optimization features corresponding to the latent space features into a pre-trained decoder to output a target resampled image corresponding to the image to be resampled.

9. An electronic device, characterized in that: include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the extreme magnification image resampling method based on generative latent space learning as claimed in any one of claims 1 to 5 or claim 6.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: The program is executed by a processor to implement the extreme magnification image resampling method based on generative latent space learning as described in any one of claims 1 to 5 or any one of claim 6.

Images