Image reconstruction training method, electronic equipment, storage medium and program product
By introducing a high-frequency information injection module into the encoder and adjusting the output dimension of the latent space, the problem of high-frequency detail loss in VAE during image generation is solved, and better image reconstruction results are achieved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- CHINA UNITED NETWORK COMM GRP CO LTD
- Filing Date
- 2024-10-29
- Publication Date
- 2026-05-12
AI Technical Summary
Existing variational autoencoders (VAEs) tend to lose small or complex high-frequency details, such as small faces and text, when generating images, resulting in poor reconstruction results.
By introducing a high-frequency information injection module into the encoder, merging low-frequency and high-frequency components, adjusting the number of input channels, and performing multi-dimensional encoding in the latent space, the output distribution characteristics are used for superposition and dimensionality reduction, thereby increasing the output dimension of the latent space and preserving more details.
It effectively reduces the loss of details during image reconstruction, adapts to various mainstream text-based image network structures, and improves image reconstruction results.
Smart Images
Figure CN122021731A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of computer technology, and in particular to an image reconstruction training method, electronic device, storage medium, and program product. Background Technology
[0002] Variational autoencoders (VAEs) are generative models based on probabilistic coding. They combine the ideas of autoencoders and probabilistic coding, achieving a more flexible and controllable sample generation process by modeling the latent representation. A VAE mainly consists of three parts: an encoder, a latent layer, and a decoder. The encoder maps the input data onto the distribution parameters of the latent space, and the decoder maps the samples obtained from the latent layer back to the original input space.
[0003] The core idea of multimodal text-to-image (TPE) models is to generate corresponding images using linguistic descriptions (text). A typical TPE model consists of two main parts: a text encoder and an image generator. The text encoder converts the input text into vector representations, which capture the semantic information within the text. The image generator then uses these semantic vectors to generate the corresponding images.
[0004] Visual Image Generators (VAEs) are an important component of multimodal text-to-image models. VAEs can function as part of an image generator, used to generate images based on semantic vectors provided by a text encoder. Specifically, the VAE's encoder maps the input text description (or some preprocessed text vector) to a latent space, and then the decoder samples from this latent space to generate the corresponding image.
[0005] Currently, because neural networks are a Gaussian smoothing process, high-frequency details are gradually lost during information transmission and feature extraction. As a result, VAEs may permanently lose small or complex high-frequency details, such as small faces or text, during inference, leading to weak reconstruction results. Summary of the Invention
[0006] This application provides an image reconstruction training method, electronic device, storage medium, and program product to reduce high-frequency detail loss and improve image reconstruction results.
[0007] In a first aspect, embodiments of this application provide an image reconstruction training method, the method comprising:
[0008] Obtain feature information from the original image, wherein the feature information includes merged low-frequency components and high-frequency components;
[0009] Determine whether the number of input channels is a preset value, wherein the number of input channels is either an initial number of input channels or a target number of input channels, and the initial number of input channels is determined based on hardware resources;
[0010] If not, then obtain the initial output value, wherein the initial output value is obtained through feature information, and the number of output values of the initial output value is the same as the number of initial input channels;
[0011] The output values are sorted and summed pairwise to obtain new output values and target input channel number, and the target input channel number is re-evaluated to determine whether it is the preset value.
[0012] When the target number of input channels is the preset value, the feature information is sampled to obtain the latent space embedding component;
[0013] The dimension of the latent space embedding component is continuously increased to the initial number of input channels, and the increased-dimensional latent space embedding component is decoded to output the reconstructed image.
[0014] In one possible implementation, the method for determining whether the number of input channels is a preset value further includes:
[0015] When the number of input channels is the same as the initial number of input channels, the feature information is sampled to obtain latent space embedding components, and the latent space embedding components are decoded to output a reconstructed image.
[0016] In one possible implementation, obtaining the feature information of the original image includes:
[0017] Extract low-frequency components from the original image;
[0018] Extract the high-frequency sub-components from the original image, and up-convert the high-frequency sub-components to obtain the high-frequency components;
[0019] The low-frequency components and the high-frequency components are combined to obtain the feature information of the original image.
[0020] In one possible implementation, the merging of the low-frequency component and the high-frequency component to obtain the feature information of the original image includes:
[0021] The components in each identical neural network layer are merged sequentially and output to the next layer. The merged components include the merged components of the previous neural network layer and the high-frequency components of that layer. The low-frequency components are assigned to the first neural network layer, and the high-frequency components of each neural network layer are assigned according to a preset rule.
[0022] The merged components of the last neural network layer are obtained to obtain the feature information of the original image.
[0023] In one possible implementation, sorting the output values and adding them pairwise to obtain new output values and the target number of input channels includes:
[0024] The output values include paired mean and variance;
[0025] Sort multiple output values based on the mean of each output value;
[0026] The mean and variance of the output values are added in pairs without repetition to obtain a new output value and the target number of input channels.
[0027] In one possible implementation, after sequentially and non-repeatingly adding the mean and variance of the output values pairwise to obtain new output values and the target number of input channels, the method further includes:
[0028] The feature vectors of the latent space embedding components are determined, and the first loss function is constructed based on the feature vectors. The first loss function is used to represent the reconstruction loss function.
[0029] Based on the high-frequency layer components of each neural network layer, construct the distribution loss function corresponding to each neural network layer;
[0030] The KL divergence is determined based on the distribution loss function, and a second loss function is determined based on the KL divergence, wherein the second loss function is used to represent the multi-level KL divergence function;
[0031] The loss function is determined based on the first loss function and the second loss function;
[0032] The new output value is monitored through the loss function.
[0033] In one possible implementation, sampling the feature information to obtain latent space embedding components includes:
[0034] Based on the aforementioned feature information, several feature vectors for the input channel are obtained;
[0035] The set of latent space embedding components is determined based on several feature vectors of the input channel.
[0036] Secondly, embodiments of this application provide an image reconstruction training apparatus, the apparatus comprising:
[0037] The high-frequency injection module is used to obtain the high-frequency components of the original image;
[0038] An encoder is used to acquire the low-frequency components of the original image;
[0039] The encoder is also used to acquire feature information of the original image, wherein the feature information includes merged low-frequency components and high-frequency components;
[0040] The encoder is also used to determine whether the number of input channels is a preset value, wherein the number of input channels is an initial number of input channels or a target number of input channels, and the initial number of input channels is determined according to hardware resources;
[0041] If not, then obtain the initial output value, wherein the initial output value is obtained through feature information, and the number of output values of the initial output value is the same as the number of initial input channels;
[0042] The output values are sorted and summed pairwise to obtain new output values and target input channel number, and the target input channel number is re-evaluated to determine whether it is the preset value.
[0043] When the target number of input channels is the preset value, the feature information is sampled to obtain the latent space embedding component;
[0044] The decoder is used to continuously increase the dimension of the latent space embedding components to the initial number of input channels, and decode the increased latent space embedding components to output the reconstructed image.
[0045] In one possible implementation, the encoder is further configured to sample the feature information to obtain latent space embedding components when the number of input channels is the initial number of input channels;
[0046] The decoder is also used to decode the latent space embedding component to output a reconstructed image.
[0047] In one possible implementation, the encoder is also used to extract low-frequency components from the original image;
[0048] The high-frequency injection module is also used to extract high-frequency sub-components from the original image and up-frequency the high-frequency sub-components to obtain high-frequency components.
[0049] The encoder is also used to combine the low-frequency component and the high-frequency component to obtain the feature information of the original image.
[0050] In one possible implementation, the high-frequency injection module is further configured to sequentially merge the merged components in each identical neural network layer and output them to the next layer, wherein the merged components include the merged components of the previous neural network layer and the high-frequency layer components of that layer, the low-frequency components are allocated to the first neural network layer, and the high-frequency layer components of each neural network layer are allocated according to a preset rule.
[0051] The encoder is also used to obtain the merged components of the last neural network layer to obtain the feature information of the original image.
[0052] In one possible implementation, the output values include paired mean and variance;
[0053] The encoder is also used to sort multiple output values according to the mean of each output value;
[0054] The encoder is also used to sequentially and non-repeatedly add the mean and variance of the output values pairwise to obtain new output values and the target number of input channels.
[0055] In one possible implementation, the encoder is further configured to determine the feature vector of the latent space embedding component and construct the first loss function based on the feature vector, the first loss function being used to represent the reconstruction loss function;
[0056] The encoder is further configured to construct a distribution loss function corresponding to each neural network layer based on the high-frequency layer components of each neural network layer;
[0057] The encoder is further configured to determine KL divergence based on the distribution loss function, and to determine a second loss function based on the KL divergence, wherein the second loss function is used to represent a multi-level KL divergence function;
[0058] The encoder is further configured to determine the loss function based on the first loss function and the second loss function;
[0059] The encoder is also used to monitor the new output value through the loss function.
[0060] In one possible implementation, the encoder is further configured to acquire several feature vectors of the input channel based on the feature information;
[0061] The encoder is also used to determine the set of latent space embedding components based on several feature vectors of the input channel.
[0062] Thirdly, embodiments of this application provide an image reconstruction training device, including: a memory and a processor;
[0063] The memory stores computer-executed instructions;
[0064] The processor executes computer execution instructions stored in the memory, causing the processor to perform the image reconstruction training method as described in the first aspect and / or various possible embodiments of the first aspect.
[0065] Fourthly, embodiments of this application provide a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement the image reconstruction training method as described in the first aspect and / or various possible embodiments of the first aspect.
[0066] Fifthly, embodiments of this application provide a computer program product, including a computer program that, when executed by a processor, implements the image reconstruction training method as described in the first aspect and / or various possible implementations of the first aspect.
[0067] This application provides an image reconstruction training method, electronic device, storage medium, and program product. The method involves acquiring feature information from the original image, including merged low-frequency and high-frequency components; determining if the number of input channels is a preset value; if not, acquiring an initial output value; sorting the output values and adding them pairwise to obtain a new output value and a target number of input channels; if so, sampling feature information to obtain latent space embedding components; continuously increasing the dimension of the latent space embedding components to the initial number of input channels; and decoding the increased-dimensional latent space embedding components to output a reconstructed image. By fusing high-frequency information in the encoder and increasing the output dimension of the latent space, and utilizing the characteristics of the output distribution to perform multi-dimensional encoding of the latent space through dimensionality reduction, this method retains more details of the original image while adapting to the structural requirements of various mainstream text-based image networks, reducing the loss of image details. Attached Figure Description
[0068] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.
[0069] Figure 1 A flowchart illustrating an image reconstruction training method provided in this application. Figure 1 ;
[0070] Figure 2 A flowchart illustrating an image reconstruction training method provided in this application. Figure 2 ;
[0071] Figure 3 A flowchart illustrating an image reconstruction training method provided in this application. Figure 3 ;
[0072] Figure 4 This is a schematic diagram of the structure of an image reconstruction training device provided in this application;
[0073] Figure 5 This is a schematic diagram of the structure of an image reconstruction training device provided in this application.
[0074] The accompanying drawings illustrate specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concept of this application to those skilled in the art through reference to particular embodiments. Detailed Implementation
[0075] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.
[0076] First, let me explain the terms used in this application:
[0077] An autoencoder (AE) is an artificial neural network used for unsupervised learning tasks (i.e., tasks without labeled data or ground truth). Its main purpose is dimensionality reduction, feature extraction, and data compression. It consists of two main components: an encoder and a decoder. The encoder first compresses the input data into the latent space, forming a low-dimensional embedding representation, and aggregates the input information as much as possible to reduce information loss between the two, thus allowing subsequent networks to reconstruct the input data as accurately as possible. The decoder is designed for this purpose; it reconstructs the original input data by accepting the low-dimensional embeddings from the latent space. The encoder and decoder typically consist of one or more neural network layers, which can be fully connected, convolutional, or recurrent neural networks.
[0078] Variational autoencoders (VAEs) are generative models that learn data distributions to generate new data samples. A VAE works by encoding data into a latent space, sampling data from that space, and then using a decoder to transform the data from the latent space into actual data. Unlike traditional autoencoders, VAEs use variational inference to estimate the latent representation of the data.
[0079] Latent Space: In a VAE, the latent space is a space of random vectors used to represent the data. By encoding the data as vectors in the latent space, the VAE can learn the distribution of the data.
[0080] Latent: Refers to latent variables. In the latent space, the high-dimensional characteristics of data are encoded into a more concise form. This latent space is composed of latent variables, each of which can be seen as a representation of a certain latent characteristic of the data.
[0081] Skip connections: The basic idea of skip connections is to directly connect the input to the output in certain layers of a network, allowing information to skip between different layers. This connection is typically implemented through an addition operation, adding the input and output together. This direct passing helps maintain gradient stability, making the network easier to train.
[0082] Reconstruction Loss: Reconstruction loss is a metric used in VAE to measure the difference between the decoder output and the original data.
[0083] KL divergence (KL Divergence): KL divergence is a metric used to measure the difference between two probability distributions. In VAEs, it is used to measure the difference between the latent vector distribution generated by the encoder and the standard normal distribution. KL divergence is an asymmetric measure of the difference between two probability distributions P and Q. This metric is particularly suitable for evaluating the information loss that occurs when a theoretical (or fitted) distribution Q is used to approximate a true distribution P. The larger the KL divergence, the greater the difference between the two distributions; conversely, the smaller the difference, the smaller the difference. When the two distributions are identical, the KL divergence is 0.
[0084] Decoder: The decoder is part of a VAE (Visual Image Generation) and is used to decode latent vectors into actual data. In image generation tasks, the decoder is typically a convolutional neural network (CNN) used to transform latent vectors into images.
[0085] Visual Image Generators (VAEs) are an important component of multimodal text-to-image models. VAEs can function as part of an image generator, used to generate images based on semantic vectors provided by a text encoder. Specifically, the VAE's encoder maps the input text description (or some preprocessed text vector) to a latent space, and then the decoder samples from this latent space to generate the corresponding image.
[0086] Currently, because neural networks are a Gaussian smoothing process, high-frequency details are gradually lost during information transmission and feature extraction. As a result, VAEs may permanently lose small or complex high-frequency details, such as small faces or text, during inference, leading to weak reconstruction results.
[0087] To address the aforementioned issues, this application provides an image reconstruction training method. By introducing an additional high-frequency information injection module at the input end to reduce high-frequency loss, and by utilizing the distribution characteristics of the encoding part of the variational autoencoder, a multi-channel hybrid latent coding is introduced. This reduces compression loss while maintaining compatibility with commonly used diffusion models, ensuring complete preservation of high-frequency information and achieving the desired effect. Furthermore, a multi-channel loss function is introduced for training, allowing the entire model to adapt to the needs of various latent channels during inference.
[0088] The technical solution of this application and how the technical solution of this application solves the above-mentioned technical problems are described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will be described below with reference to the accompanying drawings.
[0089] Figure 1 Flowchart of the image reconstruction training method provided in this application Figure 1 .like Figure 1 As shown in this embodiment, an image reconstruction training method includes:
[0090] S101: Obtain feature information of the original image.
[0091] The feature information includes merged low-frequency and high-frequency components. The low-frequency component represents the low-frequency part of the visual information, and the high-frequency component represents the high-frequency part of the visual information. In computer vision, RGB images captured by visual sensors can contain actual information about real-life scenes, which can be divided into, for example, two parts: a low-frequency part and a high-frequency part.
[0092] Understandably, the low-frequency component is responsible for capturing information about the appearance, scene, and overall picture of the image; while the high-frequency component is responsible for capturing detailed information such as sharp edges, complex textures, and easily overlooked information such as facial details and hand edges.
[0093] In one possible implementation, after an image is input, the encoder separates the low-frequency component. A high-frequency injection module injects high-frequency information into the encoder output. This injection module can inject into different layers or all layers of the encoder, depending on different requirements. It calculates the high-frequency component corresponding to that layer and merges it with the original output of that layer (i.e., the low-frequency component of the original output). The merged result is then input to the next layer. Thus, the feature information consists of the merged low-frequency and high-frequency components. By integrating the injection module, the original high-frequency information can be better preserved, achieving the original design effect.
[0094] S102: Determine whether the number of input channels is the preset value; if not, proceed to step S103; if yes, proceed to step S105.
[0095] The number of input channels refers to either the initial number of input channels or the target number of input channels, with the initial number determined based on hardware resources. Input channels represent the dimensions or number of features of the input data. The number of input channels can be obtained from metadata extracted from the image, or determined by examining data features. In existing image coding, multiple channels, i.e., the initial number of input channels, can be designed; the specific number can be designed according to actual needs, and this solution does not impose any restrictions.
[0096] Understandably, in the original VAE, after encoding by the variational encoder, the output channel number is 4. This is also the basis for the structural design of the mainstream SD network and other Diffusion image-generating networks like UNet. As mentioned above, to achieve the versatility of VAE, the preset input channel number can be set to 4. In subsequent processing, if the initial or target input channel number differs from the preset value, the channel number is merged to the preset value by merging them in pairs, thereby reducing the loss of image detail during the output process.
[0097] S103: Get the initial output value.
[0098] The initial output value is obtained through feature information, and the number of initial output values is the same as the number of initial input channels.
[0099] Understandably, based on the extracted feature information, an initial number of output values equal to the number of input channels is generated. For example, an encoder can extract corresponding low-frequency and high-frequency components based on the image's feature information and obtain the corresponding output values. The resulting output values are a series of Gaussian distributions, containing multiple pairs of mean and variance.
[0100] S104: Sort the output values and add them together in pairs to get new output values and target number of input channels.
[0101] The output values are a series of Gaussian distributions, including the mean and variance. By sorting the output values according to the mean, multiple sorted Gaussian distributions can be obtained. Due to the additivity of Gaussian distributions, adjacent two sorted distributions can be added together to obtain the corresponding merged distribution.
[0102] Understandably, if there are X output values, meaning there are currently X Gaussian distributions, we can sort these X Gaussian distributions according to their means, and then add adjacent Gaussian distributions together to obtain X / 2 Gaussian distributions, which in turn yield X / 2 output values. Since the number of output values is the same as the number of input channels, we can determine that the current target number of input channels is X / 2.
[0103] An example of the above specific process is to set the number of input channels of the encoder to... (N>=1), at this time the corresponding output will be... The mean and variance of the output Gaussian distribution; the output Sort the distributions. Due to the additivity of the Gaussian distribution, add the sorted distributions pairwise to obtain the number of... A merged distribution.
[0104] After obtaining the new output value and the target number of input channels, repeat step S102.
[0105] In order to ensure the system's versatility, the number of target input channels is judged to see if it meets the preset value. The specific judgment process is the same as described above and will not be repeated here.
[0106] Understandably, through the above merging process, the target number of input channels is half the initial number of channels, that is... For each target number of input channels, the above process can be repeated (N-1) times, that is, iterated multiple times, so that the final number of output channels of the network is consistent with the original VAE output channel number of SD, that is, until the number of output latent channels is reduced to the preset number of input channels.
[0107] S105: When the number of input channels is the target number of input channels, sample feature information to obtain latent space embedding components.
[0108] The latent space embedding component is a set of feature vectors with different numbers of output channels obtained during the above iterative merging process.
[0109] Specifically, let S represent the set of multidimensional coding latent spaces, F represent the CNN coding network, and x represent the input vector. The multidimensional coding latent space can be represented as:
[0110] .
[0111] S106: Continuously increase the dimension of the latent space embedding components to the initial number of input channels, and decode the increased latent space embedding components to output the reconstructed image.
[0112] Among them, a multi-dimensional output latent space was constructed. Then, the decoder will symmetrically increase the dimensionality of the channels from the lowest dimension to the initial number of input channels, so that the input and output are consistent and compatible. After passing through the final decoder network, the reconstructed image is output.
[0113] The purpose of this step is to perform feature aggregation and principal component analysis on the high-dimensional latent space, obtaining lower-dimensional features layer by layer while preserving the characteristics of the low-dimensional latent space. This allows for complete preservation of information from the input image, reducing compression loss and detail loss, and supports compatibility with existing diffusion model series. The decoder network can process the input vector through a series of upsampling layers (such as transposed convolutional layers or upsampling operations), convolutional layers, and possible activation functions. The combination of these layers aims to progressively increase the spatial dimensions of the data (i.e., width and height) while reducing the number of channels, in order to reconstruct an image that increasingly approximates the original input data.
[0114] Understandably, after obtaining multiple output layers, N-1 convolutional layers are added at the decoder to continuously increase the dimensionality of the z-components, i.e., the latent space embedding components, until the output dimension reaches the desired level. At this point, the output component is decoded through the entire decoder, and the corresponding reconstructed result is output.
[0115] S107: When the number of input channels is the same as the initial number of input channels, sample feature information to obtain latent space embedding components.
[0116] This involves matching the number of channels in the input data with the initial number of input channels expected by the model. When the initial number of input channels is the preset value, it means that the number of channels matches, and no dimensionality increase or other operations are needed to ensure universality.
[0117] Understandably, the sampled feature information is further processed to form latent space embedding components. For example, the feature vector can be mapped through a fully connected layer to reduce its dimensionality and capture the most important information. The embedding component can be a fixed-size vector with a dimension much smaller than the original input data, but it can capture enough information to support downstream tasks.
[0118] S108: Decode the latent space embedding component to output the reconstructed image.
[0119] The VAE is an encoder network that maps input data to latent space embeddings, and a decoder network that reconstructs the original data from the latent space embeddings.
[0120] Understandably, the structure of a decoder network can be symmetrical to that of the encoder network, but in the opposite direction. The decoder network can map the latent space embedding components back to the dimensions and shape of the original data. For example, for image data, the decoder should output an image of the same size, number of channels, and resolution as the input image.
[0121] Specifically, latent space embedding components can be used as input to the decoder network. The output of the decoder network is a reconstructed image that should be visually similar to the original input image.
[0122] The image reconstruction training method provided in this application introduces a high-frequency injection module in the encoder section to merge low-frequency and high-frequency components, thereby obtaining corresponding feature information. It then determines the number of input channels; if it is not a preset value, it obtains an initial output value, sorts the output values, and adds them pairwise to obtain a new output value and the target number of input channels, iterating until the preset value is reached. Feature information is sampled to obtain latent space embedding components. The dimension of the latent space embedding components is continuously increased to the initial number of input channels, and the increased-dimensional latent space embedding components are decoded to output the reconstructed image. By fusing high-frequency information in the encoder section, increasing the latent dimension of the latent space output, and utilizing the characteristics of the output distribution to superimpose and reduce the output dimension in a multi-dimensional encoding process of the latent space, more details of the original image are preserved while adapting to the structure of current mainstream text-based image networks, reducing the problem of image detail loss.
[0123] Figure 2 Flowchart of the image reconstruction training method provided in this application Figure 2 This embodiment is in Figure 1 Based on the examples, the image reconstruction training method is described in detail. For example... Figure 2 As shown in this embodiment, an image reconstruction training method includes:
[0124] S201: Extract low-frequency components from the original image.
[0125] The low-frequency component is responsible for capturing the appearance of the image, which may include information about the scene and the overall picture.
[0126] S202: Extract the high-frequency sub-components from the original image and up-convert the high-frequency sub-components to obtain the high-frequency components.
[0127] The high-frequency component is responsible for capturing detailed information in the image, such as sharp edges, complex textures, and easily overlooked details like facial features and hand edges. The system can extract high-frequency details from the image through Fourier transform, obtaining the corresponding feature vectors, i.e., high-frequency sub-components.
[0128] Understandably, high-frequency sub-components can be up-converted, making higher frequencies higher and lower frequencies lower, to reduce the loss of high-frequency details. Methods for up-converting high frequencies include Fourier spectrum shifting, or any other encoding method that can achieve high-frequency information; this scheme does not impose any restrictions.
[0129] S203: Merge the merged components in each identical neural network layer sequentially and output them to the next layer.
[0130] In order to restore the image, the high-frequency components need to be injected into the original separated low-frequency components after being separated. That is, the calculated high-frequency components are merged with the original output of the current layer.
[0131] Understandably, merging can be achieved through addition, splicing, or other forms of fusion, and this solution does not impose any restrictions on this.
[0132] S204: Obtain the merged components of the last neural network layer to obtain the feature information of the original image.
[0133] In this embodiment, the retention of high-frequency information can be achieved by adding a new auxiliary module, namely the high-frequency injection module. Although only low-frequency information is retained after the original input, the injection of high-frequency components ensures the integrity of the original image information.
[0134] Understandably, steps S202-S204 design a high-frequency injection module to separate the high-frequency details of the input image into separate channels and continuously shift them to higher frequencies. During the smoothing and downsampling process of the network, the information in the auxiliary module will gradually migrate from higher frequencies to higher frequencies.
[0135] The purpose of this step is that, although the original RGB input loses high-frequency details after passing through the CNN, the information from the auxiliary module ensures that the high-frequency information is fully preserved, thus achieving the effect of the original design. By injecting information through the auxiliary module, the image feature information is more complete, reducing the loss of image details.
[0136] S205: Determine whether the number of input channels is the preset value; if not, proceed to step S206; if yes, proceed to step S209.
[0137] S206: Get the initial output value.
[0138] S207: Sort multiple output values based on the mean of each output value.
[0139] S208: Add the mean and variance of the output values pairwise without repetition to obtain a new output value and the target number of input channels.
[0140] Steps S205-S208 are similar to steps S102-S104 above, and will not be described in detail here.
[0141] S209: When the number of input channels is the target number of input channels, obtain the number of feature vectors for the number of input channels based on feature information.
[0142] In the channel iteration of the above steps, a feature vector is obtained once for each iteration, which means that multiple sets of feature vectors with different numbers of output channels can be obtained.
[0143] S210: Determine the set of latent space embedding components based on several feature vectors of the input channel.
[0144] For each input feature vector, its corresponding latent space embedding components are extracted, and these components are combined into a set. This set can contain low-dimensional representations of all input feature vectors.
[0145] Understandably, the latent space can store the feature components of each iteration. When the final number of output channels of the network is consistent with the number of VAE output channels of the original SD, the iteration is completed, and multiple sets of feature vectors with different numbers of output channels can be obtained.
[0146] Specifically, let S represent the set of multidimensional coding latent spaces, F represent the CNN coding network, and x represent the input vector. The multidimensional coding latent space can be represented as:
[0147]
[0148] S211: Continuously increase the dimension of the latent space embedding components to the initial number of input channels, and decode the increased dimension latent space embedding components to output the reconstructed image.
[0149] Step S211 is similar to step S106 above, and will not be described in detail here.
[0150] S212: When the number of input channels is the initial number of input channels, obtain the number of feature vectors for the input channels based on feature information.
[0151] S213: Determine the set of latent space embedding components based on several feature vectors of the input channel.
[0152] S214: Decode the latent space embedding component to output the reconstructed image.
[0153] Steps S212-S214 are similar to steps S107-S108 above, and will not be described in detail here.
[0154] The image reconstruction training method provided in this application reduces high-frequency loss by separating high-frequency details of the input image into separate channels and continuously shifting them to higher frequencies. This allows for the complete preservation of high-frequency information, better reconstructing the high-frequency components of the entire image, and minimizing the loss of image details. Simultaneously, it employs a multi-dimensional encoding process to increase the latent dimension of the latent space output and utilize the characteristics of the output distribution to perform dimensionality reduction. This method allows for feature aggregation and principal component analysis of the high-dimensional latent space, obtaining lower-dimensional features layer by layer while maintaining the characteristics of the low-dimensional latent space. This approach preserves more details of the original image while adapting to the structure of current mainstream text-based image networks.
[0155] Figure 3 Flowchart of the image reconstruction training method provided in this application Figure 2 In this embodiment Figure 2 Based on the example, the process of adding a loss function to supervise the new output value after obtaining the new output value and the target number of input channels is explained in detail. For example... Figure 3 As shown in this embodiment, an image reconstruction training method includes:
[0156] S301: Determine the feature vectors of the latent space embedding components and construct the first loss function based on the feature vectors.
[0157] Here, the first loss function represents the reconstruction loss function, which measures the difference between the generated data and the original data. The points in the latent space, i.e., the embedded components, are low-dimensional representations of the original feature vectors. The original feature vectors, with different numbers of input channels, can be input into the encoder network, and the encoder's output is the embedded components in the latent space. These components can be continuous, low-dimensional vectors used to capture the key features of the input feature vectors.
[0158] Understandably, the reconstruction loss function can be used to measure the accuracy of reconstructing the original feature vector from the embedded components in the latent space. The embedded components in the latent space are input into the decoder network, and the output of the decoder is the reconstructed feature vector, which should be as close as possible to the original input feature vector. At this point, an appropriate loss function can be used, such as mean squared error (MSE) or cross-entropy loss, to calculate the difference between the reconstructed feature vector and the original feature vector, i.e., the reconstruction loss.
[0159] In this embodiment, the original VAE of SD can utilize the output latent space feature vector to construct a distribution-based loss function so that the output features conform to the preset latent space characteristics.
[0160] S302: Construct the distributed loss function for each neural network layer based on the high-frequency components of each neural network layer.
[0161] In this process, after constructing the multidimensional output latent space, the decoder symmetrically increases the dimensionality of each channel starting from the lowest dimension. The reconstructed image is then output through the final decoder network. Therefore, during training, a distribution loss function is constructed for each element in the multidimensional encoding latent space, i.e., each neural network layer. The distribution loss function is used in deep learning to capture the characteristics of high-frequency components in neural network layers, determine the specific frequency components of the data learned by the model, and avoid overfitting or other undesirable characteristics.
[0162] Understandably, high-frequency components can be extracted from the output of each neural network layer. For example, Fourier transform or other frequency domain analysis methods can be used to separate components of different frequencies, i.e., to separate the high-frequency parts. A distribution loss function can be constructed to regularize these components. In this embodiment, distribution matching is used to regularize the high-frequency components.
[0163] S303: Determine the KL divergence based on the distributed loss function, and determine the second loss function based on the KL divergence.
[0164] The second loss function is used to represent the multi-level KL divergence function.
[0165] Understandably, during training, each element in the multidimensional encoding latent space... Distributed loss function construction will be performed. .
[0166] In VAE, KL divergence is used as a regularization term to ensure that the difference between the distribution of the latent space (usually a standard normal distribution) and the latent distribution inferred by the model is minimized. In this case, this regularization term is also the second loss function.
[0167] S304: Determine the loss function based on the first loss function and the second loss function.
[0168] The loss function includes a semi-reconstruction loss and a KL divergence regularization term. The reconstruction loss measures the difference between the generated data and the original data, while the KL divergence regularization term ensures that the distribution of the latent variables approximates the prior distribution (usually a standard normal distribution). Training by combining these two loss functions ensures that all outputs of the multidimensional encoded latent space conform to the preset latent space characteristics.
[0169] Understandably, the reconstruction loss remains consistent with the original VAE, while the KL divergence is transformed into a multi-level KL divergence function corresponding to the multi-dimensional encoding latent space, resulting in the following loss function:
[0170]
[0171] in, Indicates the losses incurred during reconstruction. This represents the multi-level KL divergence loss.
[0172] It should be noted that during deployment, the output of a specific part can be used as the input of the diffusion model according to the requirements of the UNet part of the target model, thereby adapting to the needs of different models. This solution does not impose any restrictions on this.
[0173] S305: Monitor the new output value through the loss function.
[0174] The loss function measures the difference between the model's predicted value and the actual value. This difference is used to guide the model's training process and updates the model's weights through the backpropagation algorithm, thereby reducing prediction error.
[0175] Understandably, we can use the currently trained model to predict new data, obtain predicted values, and then substitute the predicted and actual values into the previously defined loss function to calculate the loss value. This loss value reflects the model's predictive performance on the new data.
[0176] Specifically, the calculated loss value can be used to evaluate the model's performance on new data. For example, a smaller loss value indicates that the model's predictions are more accurate; a larger loss value may require further model adjustments.
[0177] The image reconstruction training method provided in this application, by adding a loss function to supervise and train the output value, can make all outputs of the multidimensional encoded latent space more consistent with the preset latent space characteristics, and can adapt to the needs of multiple latent channels during inference.
[0178] Based on the above description, the image reconstruction training method provided in this application embodiment is illustrated below with specific examples. An image reconstruction training method, for example:
[0179] Assuming the input image is I, using Fourier transform... Shift I to a higher frequency region, where q is the distance shifted in the frequency domain, i.e., the shifted frequency band.
[0180] The original image is shifted multiple times using different values of 'q', and the shifted results are then converted to the image domain for channel stitching. For example, after 10 shifts, the input features are... Become .
[0181] After this, the input will be compressed and downsampled three times by the encoder to obtain a dimension of The highest-dimensional latent space feature vector is obtained, and after channel iteration, the dimension is obtained step by step. , The latent space feature vectors are used to obtain the multidimensional latent space feature vector set as follows:
[0182] ;
[0183] Understandable, the initial result The feature vectors have the same dimensions as the VAEs required by DALL-E-3 and EMU, and the final result is... The feature vectors correspond to the feature dimensions of the VAE required by the original SD. The final feature vectors are then passed through three convolutional neural networks to gradually restore the dimension of the output features to the required dimensions. The image is then processed by a VAE decoder to output the reconstructed image.
[0184] The reconstruction loss is then kept consistent with the original VAE, while the KL divergence corresponds to the multi-level KL divergence function of the multidimensional encoding latent space. Therefore, the resulting loss function is:
[0185] ;
[0186] in, Indicates the losses incurred during reconstruction. This represents the multi-level KL divergence loss.
[0187] Figure 4 A schematic diagram of the image reconstruction training device provided in this application. Figure 4 As shown, the image reconstruction training device 400 provided in this embodiment includes:
[0188] The high-frequency injection module 401 is used to acquire the high-frequency components of the original image;
[0189] Encoder 402 is used to acquire the low-frequency components of the original image;
[0190] The encoder 402 is also used to acquire feature information of the original image, wherein the feature information includes merged low-frequency components and high-frequency components;
[0191] The encoder 402 is also used to determine whether the number of input channels is a preset value, wherein the number of input channels is an initial number of input channels or a target number of input channels, and the initial number of input channels is determined according to hardware resources;
[0192] If not, then obtain the output value, wherein the initial output value is obtained through feature information, the number of output values is the same as the number of input channels, and re-determine whether the target number of input channels is the preset value;
[0193] When the target number of input channels is the preset value, the feature information is sampled to obtain the latent space embedding component;
[0194] If so, the feature information is sampled to obtain the latent space embedding component;
[0195] Decoder 403 is used to continuously increase the dimension of the latent space embedding components to the initial number of input channels, and decode the increased latent space embedding components to output the reconstructed image.
[0196] In one possible implementation, the encoder 402 is further configured to sample the feature information to obtain latent space embedding components when the number of input channels is the initial number of input channels;
[0197] The decoder 403 is also used to decode the latent space embedding component to output a reconstructed image.
[0198] In one possible implementation, the encoder 402 is also used to extract low-frequency components from the original image;
[0199] The high-frequency injection module 401 is also used to extract high-frequency sub-components from the original image and up-frequency the high-frequency sub-components to obtain high-frequency components.
[0200] The encoder 402 is also used to merge the low-frequency component and the high-frequency component to obtain the feature information of the original image.
[0201] In one possible implementation, the high-frequency injection module 401 is further configured to sequentially merge the merged components in each of the same neural network layers and output them to the next layer, wherein the merged components include the merged components of the previous neural network layer and the high-frequency components of that layer, the low-frequency components are allocated to the first neural network layer, and the high-frequency components of each neural network layer are allocated according to a preset rule.
[0202] The encoder 402 is also used to obtain the merged components of the last neural network layer to obtain the feature information of the original image.
[0203] In one possible implementation, the output values include paired mean and variance;
[0204] The encoder 402 is also used to sort multiple output values according to the mean of each output value;
[0205] The encoder 402 is also used to sequentially and non-repeatedly add the mean and variance of the output values pairwise to obtain a new output value and a target number of input channels.
[0206] In one possible implementation, the encoder 402 is further configured to determine the feature vector of the latent space embedding component and construct the first loss function based on the feature vector, the first loss function being used to represent the reconstruction loss function;
[0207] The encoder 402 is further configured to construct a distribution loss function corresponding to each neural network layer based on the high-frequency layer components of each neural network layer;
[0208] The encoder 402 is further configured to determine KL divergence based on the distribution loss function, and to determine a second loss function based on the KL divergence, wherein the second loss function is used to represent a multi-level KL divergence function;
[0209] The encoder 402 is further configured to determine the loss function based on the first loss function and the second loss function;
[0210] The encoder 402 is also used to supervise the new output value through the loss function.
[0211] In one possible implementation, the encoder 402 is further configured to acquire several feature vectors of the input channel based on the feature information;
[0212] The encoder 402 is also used to determine the set of latent space embedding components based on several feature vectors of the input channel.
[0213] The image reconstruction training device provided in this embodiment can execute the method provided in the above method embodiment. Its implementation principle and technical effect are similar, and will not be described in detail here.
[0214] Figure 5 A schematic diagram of the image reconstruction training device provided in this application. Figure 5 As shown, the electronic device 500 provided in this embodiment includes at least one processor 501 and a memory 502. Optionally, the device 500 further includes a communication component 503. The processor 501, memory 502, and communication component 503 are connected via a bus 504.
[0215] In the specific implementation process, at least one processor 501 executes computer execution instructions stored in memory 502, causing at least one processor 501 to execute the above-described image reconstruction training method.
[0216] The specific implementation process of processor 501 can be found in the above-described image reconstruction training method embodiment, which has a similar implementation principle and technical effect, and will not be repeated here.
[0217] In the above embodiments, it should be understood that the processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), etc. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in this invention can be directly implemented by a hardware processor, or implemented by a combination of hardware and software modules within the processor.
[0218] The memory may include random access memory (RAM) and may also include non-volatile memory (NVM), such as at least one disk storage device.
[0219] The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of illustration, the buses shown in the accompanying drawings are not limited to a single bus or a single type of bus.
[0220] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the above-described image reconstruction training method.
[0221] This application also provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the above-described image reconstruction training method.
[0222] The aforementioned readable storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. The readable storage medium can be any available medium accessible to a general-purpose or special-purpose computer.
[0223] An exemplary readable storage medium is coupled to a processor, enabling the processor to read information from and write information to the readable storage medium. Of course, the readable storage medium can also be a component of the processor. The processor and the readable storage medium can reside in an Application Specific Integrated Circuit (ASIC). Alternatively, the processor and the readable storage medium can exist as discrete components in the device.
[0224] The division of units is merely a logical functional division; in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be indirect coupling or communication connection through some interfaces, devices, or units, and may be electrical, mechanical, or other forms.
[0225] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.
[0226] In addition, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.
[0227] If a function is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.
[0228] Those skilled in the art will understand that all or part of the steps of the above-described method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When executed, the program performs the steps of the above-described method embodiments; and the aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks.
[0229] Finally, it should be noted that other embodiments of the invention will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This invention is intended to cover any variations, uses, or adaptations of the invention that follow the general principles of the invention and include common knowledge or customary techniques in the art not disclosed herein, and is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of the invention is limited only by the appended claims.
Claims
1. An image reconstruction training method, characterized in that, The method includes: Obtain feature information from the original image, wherein the feature information includes merged low-frequency components and high-frequency components; Determine whether the number of input channels is a preset value, wherein the number of input channels is either an initial number of input channels or a target number of input channels, and the initial number of input channels is determined based on hardware resources; If not, then obtain the initial output value, wherein the initial output value is obtained through feature information, and the number of output values of the initial output value is the same as the number of initial input channels; The output values are sorted and summed pairwise to obtain new output values and the target number of input channels. The target number of input channels is then re-evaluated to determine whether it is the preset value. When the target number of input channels is the preset value, the feature information is sampled to obtain the latent space embedding component; The dimension of the latent space embedding component is continuously increased to the initial number of input channels, and the increased-dimensional latent space embedding component is decoded to output the reconstructed image.
2. The method according to claim 1, characterized in that, The method for determining whether the number of input channels is a preset value further includes: When the number of input channels is the same as the initial number of input channels, the feature information is sampled to obtain latent space embedding components, and the latent space embedding components are decoded to output a reconstructed image.
3. The method according to claim 1, characterized in that, The acquisition of feature information from the original image includes: Extract low-frequency components from the original image; Extract the high-frequency sub-components from the original image, and up-convert the high-frequency sub-components to obtain the high-frequency components; The low-frequency components and the high-frequency components are combined to obtain the feature information of the original image.
4. The method according to claim 3, characterized in that, The process of merging the low-frequency components and the high-frequency components to obtain the feature information of the original image includes: The components in each identical neural network layer are merged sequentially and output to the next layer. The merged components include the merged components of the previous neural network layer and the high-frequency components of that layer. The low-frequency components are assigned to the first neural network layer, and the high-frequency components of each neural network layer are assigned according to a preset rule. The merged components of the last neural network layer are obtained to obtain the feature information of the original image.
5. The method according to claim 1, characterized in that, The step of sorting the output values and adding them pairwise to obtain new output values and the target number of input channels includes: The output values include paired mean and variance; Sort multiple output values based on the mean of each output value; The mean and variance of the output values are added in pairs without repetition to obtain a new output value and the target number of input channels.
6. The method according to claim 5, characterized in that, After sequentially and non-repeatingly adding the mean and variance of the output values pairwise to obtain a new output value and the target number of input channels, the method further includes: The feature vectors of the latent space embedding components are determined, and the first loss function is constructed based on the feature vectors. The first loss function is used to represent the reconstruction loss function. Based on the high-frequency layer components of each neural network layer, construct the distribution loss function corresponding to each neural network layer; The KL divergence is determined based on the distribution loss function, and a second loss function is determined based on the KL divergence, wherein the second loss function is used to represent the multi-level KL divergence function; The loss function is determined based on the first loss function and the second loss function; The new output value is monitored through the loss function.
7. The method according to claim 1, characterized in that, The sampling of the feature information to obtain latent space embedding components includes: Based on the aforementioned feature information, several feature vectors for the input channel are obtained; The set of latent space embedding components is determined based on several feature vectors of the input channel.
8. An image reconstruction training device, characterized in that, The device includes: The high-frequency injection module is used to obtain the high-frequency components of the original image; An encoder is used to acquire the low-frequency components of the original image; The encoder is also used to acquire feature information of the original image, wherein the feature information includes merged low-frequency components and high-frequency components. The encoder is also used to determine whether the number of input channels is a preset value, wherein the number of input channels is an initial number of input channels or a target number of input channels, and the initial number of input channels is determined according to hardware resources; If not, then obtain the initial output value, wherein the initial output value is obtained through feature information, and the number of output values of the initial output value is the same as the number of initial input channels; The output values are sorted and summed pairwise to obtain new output values and the target number of input channels. The target number of input channels is then re-evaluated to determine whether it is the preset value. When the target number of input channels is the preset value, the feature information is sampled to obtain the latent space embedding component; The decoder is used to continuously increase the dimension of the latent space embedding components to the initial number of input channels, and decode the increased latent space embedding components to output the reconstructed image.
9. An image reconstruction training device, characterized in that, include: Memory, processor; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory, causing the processor to perform the image reconstruction training method as described in any one of claims 1-7.
10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the image reconstruction training method as described in any one of claims 1-7.