Method for training first model, and method and device for image reconstruction

Abstract

The invention relates to a method for training a first model, and a method and device for image reconstruction. The first model comprises an encoder, a potential diffusion model and a decoder, and the method for training the first model comprises the following steps: obtaining a training image set; training the encoder and the decoder based on the training image set; obtaining a first coding feature of each training image in the training image set by using the trained encoder, and obtaining a potential feature vector corresponding to each training image based on the first coding feature by using the potential diffusion model; obtaining a semantic feature vector corresponding to each training image in the training image set; and determining a first training loss based on the potential feature vector and the semantic feature vector, and training the potential diffusion model according to the first training loss.

Description

Technical Field

[0001] This disclosure relates to the field of image processing, and more particularly to a method for training a first model, a method for image reconstruction, an apparatus for training a first model, an apparatus for image reconstruction, an electronic device, a computer-readable storage medium, and a computer program product. Background Technology

[0002] Image super-resolution (ISR) reconstruction refers to the technique of recovering or reconstructing high-resolution, high-quality images from low-resolution or low-quality images. It is an important technique in the field of image restoration and reconstruction. With the development of deep learning technology, machine learning models can be used to generate reconstructed images with better visual quality. In recent years, with the development of diffusion models, image ISR reconstruction methods based on diffusion models have become a research hotspot. For example, image ISR reconstruction technology based on the Latent Diffusion Model (LDM) (also known as the "latent space diffusion model") first uses an encoder to compress the image into a latent space, and then performs diffusion modeling in the latent space to obtain the reconstructed image.

[0003] However, the same degraded image often corresponds to multiple reasonable high-resolution results, and key information, such as high-frequency details like texture and edges, as well as semantically relevant structural cues, is irreversibly lost during the degradation process. Therefore, the goal of ISR is not merely to "generate a decent-looking image," but to reconstruct as much information as possible that has been erased or distorted during degradation.

[0004] Although LDM-based image ISR reconstruction techniques proposed in recent years have been able to generate reconstructed images with high perceptual quality, they still suffer from low image fidelity. This is because the training objective in LDM-based image ISR reconstruction techniques employs per-sample Kullback-Leibler Divergence (KL) regularization, forcibly aligning the latent distribution of each sample to a fixed prior. This compresses the information content of the latent space, making it difficult for the latent representation to carry high-precision texture, detail, and semantic information, ultimately resulting in "information-poor" latent feature vectors. The decoder then needs to reconstruct the image from these latent feature vectors, which inevitably leads to problems such as texture blurring, missing details, and even semantic errors, limiting the improvement of image reconstruction fidelity. To address this, related techniques typically introduce external guidance information (e.g., input text prompts, image structure maps, etc.) to constrain the diffusion process and guide image reconstruction, thereby improving image fidelity. However, introducing external guidance information is costly, increases system complexity, and is difficult to obtain stably reliable external guidance information in some application scenarios. Therefore, there is a need for techniques to improve the fidelity of reconstructed images without relying on complex external guidance. Summary of the Invention

[0005] This disclosure provides a method for training a first model, a method for image reconstruction, an apparatus for training a first model, an apparatus for image reconstruction, an electronic device, a computer-readable storage medium, and a computer program product, to at least solve the problem of low fidelity in reconstructed images in related technologies. The technical solution of this disclosure is as follows: According to a first aspect of the present disclosure, a method for training a first model is provided, the first model including an encoder, a latent diffusion model, and a decoder. The method includes: acquiring a training image set; training the encoder and the decoder based on the training image set; obtaining a first encoding feature of each training image in the training image set using the trained encoder, and obtaining a latent feature vector corresponding to each training image based on the first encoding feature using the latent diffusion model; acquiring a semantic feature vector corresponding to each training image in the training image set; determining a first training loss based on the latent feature vector and the semantic feature vector, and training the latent diffusion model based on the first training loss.

[0006] Optionally, determining the first training loss based on the latent feature vector and the semantic feature vector includes: for each training image in the training image set, determining the similarity between the latent feature vector corresponding to the training image and the semantic feature vectors corresponding to all training images in the training image set; and determining the first training loss based on the similarity.

[0007] Optionally, determining the first training loss based on the similarity includes: normalizing the similarity for each training image in the training image set using a normalization function; and determining the first training loss based on the normalization result.

[0008] Optionally, determining the first training loss based on the similarity includes: obtaining the entropy value corresponding to each training image in the training image set; obtaining the first parameter value corresponding to each training image based on the entropy value; and determining the first training loss based on the first parameter value and the similarity.

[0009] Optionally, determining the first training loss based on the first parameter value and the similarity includes: for each training image in the training image set, adjusting the similarity based on the first parameter value to obtain an adjusted similarity; normalizing the adjusted similarity using a normalization function, and determining the first training loss based on the normalization result.

[0010] Optionally, adjusting the similarity based on the first parameter value to obtain the adjusted similarity includes: dividing the similarity by the first parameter value to obtain the adjusted similarity; normalizing the adjusted similarity using a normalization function and determining the first training loss based on the normalization result includes: normalizing the adjusted similarity using a normalization exponential function and determining the first training loss based on the logarithm of the normalization result.

[0011] Optionally, training the encoder and the decoder based on the training image set includes: obtaining a second encoding feature of each training image in the training image set using the encoder; obtaining a reconstructed image corresponding to each training image using the decoder based on the second encoding feature; determining a second training loss based on each training image, the reconstructed image corresponding to each training image, and the probability of whether the reconstructed image is identified as a real image; and training the encoder and the decoder according to the second training loss.

[0012] Optionally, determining the second training loss based on each training image, the reconstructed image corresponding to each training image, and the probability of whether the reconstructed image is identified as a real image includes: determining a third training loss based on the difference between each training image and the reconstructed image corresponding to each training image; determining the probability of whether the reconstructed image is identified as a real image, and determining a fourth training loss based on the probability; and determining the second training loss based on the third training loss and the fourth training loss.

[0013] Optionally, obtaining the latent feature vector corresponding to each training image based on the first encoded feature using the latent diffusion model includes: obtaining the latent features corresponding to each training image based on the first encoded feature using the latent diffusion model; performing a first feature mapping on the latent features corresponding to each training image to obtain the latent feature vector corresponding to each training image. Obtaining the semantic feature vector corresponding to each training image in the training image set includes: obtaining a pre-trained second model, wherein the second model is trained to obtain the semantic features of the image; using the second model to obtain the semantic features corresponding to each training image in the training image set; and performing a second feature mapping on the semantic features corresponding to each training image to obtain the semantic feature vector corresponding to each training image.

[0014] According to a second aspect of the present disclosure, a method for image reconstruction is provided, the method comprising: acquiring a first image; acquiring a second image by performing the following operations: obtaining encoded features corresponding to the first image based on the first image using an encoder included in a first model; obtaining a latent feature vector corresponding to the first image based on the encoded features using a latent diffusion model included in the first model; and obtaining the second image based on the latent feature vector using a decoder included in the first model, wherein the first model is trained according to the method described above for training the first model.

[0015] According to a third aspect of the present disclosure, an apparatus for training a first model is provided, the first model including an encoder, a latent diffusion model, and a decoder. The apparatus includes: a training image set acquisition unit configured to acquire a training image set; a first training unit configured to train the encoder and the decoder based on the training image set; and a second training unit configured to: obtain a first encoding feature of each training image in the training image set using the trained encoder; obtain a latent feature vector corresponding to each training image using the latent diffusion model based on the first encoding feature; acquire a semantic feature vector corresponding to each training image in the training image set; for each training image in the training image set, determine the similarity between the latent feature vector corresponding to the training image and the semantic feature vectors corresponding to all training images in the training image set; determine a first training loss based on the similarity; and train the latent diffusion model based on the first training loss.

[0016] Optionally, the second training unit is configured to: for each training image in the training image set, determine the similarity between the latent feature vector corresponding to the training image and the semantic feature vectors corresponding to all training images in the training image set; and determine a first training loss based on the similarity.

[0017] Optionally, the second training unit is configured to: normalize the similarity for each training image in the training image set using a normalization function; and determine the first training loss based on the normalization result.

[0018] Optionally, the second training unit is configured to: obtain the entropy value corresponding to each training image in the training image set; obtain the first parameter value corresponding to each training image based on the entropy value; and determine the first training loss based on the first parameter value and the similarity.

[0019] Optionally, the second training unit is configured to: for each training image in the training image set, adjust the similarity based on the first parameter value to obtain an adjusted similarity; normalize the adjusted similarity using a normalization function, and determine the first training loss based on the normalization result.

[0020] Optionally, the second training unit is configured to: divide the similarity by the first parameter value to obtain the adjusted similarity; normalize the adjusted similarity using a normalized exponential function; and determine the first training loss based on the logarithm of the normalization result.

[0021] Optionally, the first training unit is configured to: obtain second coding features of each training image in the training image set using the encoder; obtain reconstructed images corresponding to each training image using the decoder based on the second coding features; determine a second training loss based on each training image, the reconstructed images corresponding to each training image, and the probability of whether the reconstructed images are identified as real images; and train the encoder and the decoder according to the second training loss.

[0022] Optionally, the first training unit is configured to: determine a third training loss based on the differences between each training image and the reconstructed image corresponding to each training image; determine the probability of whether the reconstructed image is identified as a real image, and determine a fourth training loss based on the probability; and determine a second training loss based on the third training loss and the fourth training loss.

[0023] Optionally, the second training unit is configured to: obtain latent features corresponding to each training image based on the first encoded features using the latent diffusion model; perform a first feature mapping on the latent features corresponding to each training image to obtain a latent feature vector corresponding to each training image; obtain a pre-trained second model, wherein the second model is trained to obtain semantic features of an image; use the second model to obtain semantic features corresponding to each training image in the training image set; and perform a second feature mapping on the semantic features corresponding to each training image to obtain a semantic feature vector corresponding to each training image.

[0024] According to a fourth aspect of the present disclosure, an apparatus for image reconstruction is provided, the apparatus comprising: a first image acquisition unit configured to acquire a first image; and a second image acquisition unit configured to acquire a second image by performing the following operations: obtaining encoded features corresponding to the first image based on the first image using an encoder included in a first model; obtaining a latent feature vector corresponding to the first image based on the encoded features using a latent diffusion model included in the first model; and obtaining the second image based on the latent feature vector using a decoder included in the first model, wherein the first model is trained according to the method described above for training the first model.

[0025] According to a fifth aspect of the present disclosure, an electronic device is provided, comprising: a processor; and a memory for storing processor-executable instructions; wherein the processor is configured to execute the instructions to implement the method described above.

[0026] According to a sixth aspect of the present disclosure, a computer-readable storage medium is provided that, when instructions in the computer-readable storage medium are executed by a processor of an electronic device, enables the electronic device to perform the above-described method.

[0027] According to a seventh aspect of the present disclosure, a computer program product is provided, comprising a computer program / instructions, wherein the computer program / instructions, when executed by a processor, implement the above-described method.

[0028] The technical solutions provided by the embodiments of this disclosure have at least the following beneficial effects: According to the method for training a first model disclosed herein, after training the encoder and decoder included in the first model, the first encoded features of each training image in the training image set are obtained using the trained encoder. Based on the first encoded features, the latent feature vectors corresponding to each training image are obtained using the latent diffusion model included in the first model. The semantic feature vectors corresponding to each training image in the training image set are obtained. A first training loss is determined based on the latent feature vectors and the semantic feature vectors. The latent diffusion model is trained according to the first training loss, and finally the trained first model is obtained. Since the first training loss is determined based on the latent feature vectors and the semantic feature vectors and the latent diffusion model is trained according to the first training loss, the latent feature vectors corresponding to the training images can be guided to align with the semantics of the training images (promoting the consistency between the latent feature vectors and the image semantics). This allows the latent feature vectors to retain information that is truly useful for image reconstruction, thereby enabling the training of a first model that can improve the fidelity of the reconstructed image.

[0029] According to the image reconstruction method of this disclosure, since a second image is obtained based on a first image using a first model trained according to the method for training a first model described above, the fidelity of the obtained second image can be improved.

[0030] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0031] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure, and are not intended to unduly limit this disclosure.

[0032] Figure 1 An exemplary system architecture 100 is shown that can be applied to an exemplary embodiment of the present disclosure.

[0033] Figure 2 This is a flowchart illustrating a method for training a first model according to an exemplary embodiment.

[0034] Figure 3 This is a schematic diagram illustrating the two-stage training of a first model according to an exemplary embodiment.

[0035] Figure 4 This is a schematic diagram illustrating the determination of a first training loss according to an exemplary embodiment.

[0036] Figure 5 This is a flowchart illustrating a method for image reconstruction according to an exemplary embodiment.

[0037] Figure 6 This is a block diagram illustrating an apparatus for training a first model according to an exemplary embodiment.

[0038] Figure 7 This is a block diagram of an apparatus for image reconstruction according to an exemplary embodiment.

[0039] Figure 8 This is a block diagram of an electronic device illustrated according to an exemplary embodiment. Detailed Implementation

[0040] To enable those skilled in the art to better understand the technical solutions of this disclosure, the technical solutions in the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings.

[0041] It should be noted that the terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this disclosure are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this disclosure described herein can be implemented in orders other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this disclosure as detailed in the appended claims.

[0042] As mentioned in the background section of this disclosure, LDM-based image ISR reconstruction technology still suffers from low image fidelity. Related technologies often improve image fidelity by introducing external guidance information, which makes the improvement of image fidelity highly dependent on external guidance and increases system complexity and cost. In view of this, this disclosure proposes a technical solution to improve the fidelity of reconstructed images without relying on complex external guidance.

[0043] Figure 1 An exemplary system architecture 100 is shown that can be applied to an exemplary embodiment of the present disclosure.

[0044] like Figure 1 As shown, the system architecture 100 may include terminal devices 101, 102, and 103, a network 104, and a server 105. Network 104 serves as the medium for providing communication links between terminal devices 101, 102, and 103 and server 105. Network 104 may include various connection types, such as wired or wireless communication links or fiber optic cables. Users can use terminal devices 101, 102, and 103 to interact with server 105 through network 104 to receive or send messages (such as audio / video data upload requests or audio / video data retrieval requests). Various communication client applications may be installed on terminal devices 101, 102, and 103, such as video recording applications, audio playback applications, instant messaging tools, email clients, and social media platform software. Terminal devices 101, 102, and 103 may be hardware or software. When terminal devices 101, 102, and 103 are hardware, they can be various electronic devices with displays capable of audio and video playback and recording, including but not limited to smartphones, tablets, laptops, and desktop computers. When terminal devices 101, 102, and 103 are software, they can be installed in the aforementioned electronic devices. They can be implemented as multiple software programs or software modules (e.g., to provide distributed services) or as a single software program or software module. No specific limitations are imposed here.

[0045] Terminal devices 101, 102, and 103 may be equipped with image acquisition devices (e.g., cameras) to capture video data. In practice, the smallest visual unit that makes up video is a frame. Each frame is a static image. Combining temporally consecutive frame sequences creates a dynamic video. Furthermore, terminal devices 101, 102, and 103 may also be equipped with components for converting electrical signals into sound (e.g., speakers) to play sound, and may also be equipped with devices for converting analog audio signals into digital audio signals (e.g., microphones) to capture sound.

[0046] Terminal devices 101, 102, and 103 can acquire video data using the image acquisition device installed on them, and can play audio data using the audio processing component that supports audio playback installed on them. Furthermore, terminal devices 101, 102, and 103 can perform timestamp calculations and other processing on the acquired audio and video data, and can store the processing results.

[0047] Server 105 can be a server that provides various services, such as a backend server that supports video recording applications installed on terminal devices 101, 102, and 103. The backend server can parse, store, and process received audio and video data upload requests, and can also receive audio and video data acquisition requests sent by terminal devices 101, 102, and 103, and feed back the audio and video data indicated by the acquisition request to terminal devices 101, 102, and 103.

[0048] It should be noted that a server can be either hardware or software. When the server is hardware, it can be implemented as a distributed server cluster consisting of multiple servers, or as a single server. When the server is software, it can be implemented as multiple software programs or software modules (e.g., used to provide distributed services), or as a single software program or software module. No specific limitations are made here.

[0049] It should be noted that the method provided in this application embodiment can be executed by terminal devices 101, 102, and 103 or by server 105. Accordingly, the device provided in this application embodiment can be located in terminal devices 101, 102, and 103 or in server 105.

[0050] It should be understood that Figure 1 The number of terminal devices, networks, and servers shown is merely illustrative. Any number of terminal devices, networks, and servers can be used as needed, and this disclosure places no limitation on this.

[0051] Figure 2This is a flowchart illustrating a method for training a first model according to an exemplary embodiment.

[0052] According to an exemplary embodiment, the first model may include an encoder, a latent diffusion model, and a decoder. For example, the encoder may be an autoencoder, such as a variational autoencoder (VAE), but is not limited thereto. For example, the decoder may be a VAE decoder, but is not limited thereto. According to embodiments, both the encoder and decoder may be implemented based on neural networks, and this disclosure does not impose any limitations on the type of neural network, etc.

[0053] Reference Figure 2 In step S210, a training image set is obtained. For example, the training image set can be a collection of multiple training images for each batch. This disclosure does not impose any restrictions on the content of the training images in the training image set, the color format of the training images, or the method of obtaining the training image set. Training images of any form can be used, and the training image set can be obtained in any image acquisition method.

[0054] In step S220, the encoder and decoder are trained based on the training image set. According to an exemplary embodiment, a two-stage training approach can be used to train the first model. The first stage can train the encoder and decoder included in the first model, and the second stage can train the latent diffusion model included in the first model. Hereinafter, the training loss of the second stage is referred to as the first training loss, and the training loss of the first stage is referred to as the second training loss.

[0055] According to an exemplary embodiment, in the first stage of training, the encoder and decoder can be trained based on the training image set using any possible method of training the encoder and decoder, and this disclosure is not limited thereto.

[0056] For example, step S220 may include: obtaining second encoded features of each training image in the training image set using an encoder; obtaining reconstructed images corresponding to each training image using a decoder based on the second encoded features; determining a second training loss based on each training image, the reconstructed image corresponding to each training image, and the probability of whether the reconstructed image is identified as a real image; and training the encoder and decoder according to the second training loss. The second training loss is the training loss of the first stage of the first model, and may be denoted as [missing information] in the following text. .

[0057] Alternatively, for example, a third training loss (e.g., denoted below) can be determined first based on the differences between each training image and the corresponding reconstructed image. Then, based on each training image and the second encoded features of each training image, the KL regularization loss (hereinafter referred to as...) is determined. LKL The second training loss is determined based on the third training loss and the KL regularization loss. For example, the second training loss can be determined based on the following Equation 1. : = + L KL Equation 1 in, These are hyperparameters, which can be any constant, for example... It can be equal to 1. This is well known to those skilled in the art. and L KL The calculation method will not be elaborated here.

[0058] However, the method used to determine the training loss in the first stage through Equation 1 above employs KL regularization loss. The KL regularization term forces the posterior distribution of the encoded features of each sample to align with a simple, fixed prior distribution (e.g., a standard Gaussian distribution). This leads to compression of information in the feature space; for example, it compresses the mutual information between the input image and the latent representation in the feature space, potentially resulting in the loss of information relevant to the image reconstruction task. To address this, and to increase the information content in the feature space, thereby retaining more information relevant to the image reconstruction task, this disclosure further proposes to constrain the overall distribution of the generated results in the image space during the first stage of training, replacing the per-sample KL regularization loss with image space adversarial loss.

[0059] Figure 3 This is a schematic diagram illustrating the two-stage training of a first model according to an exemplary embodiment. Figure 3 As shown, in the first stage (Stage I) of model training, after obtaining the second encoded features of each training image in the training image set using an encoder (e.g., a VAE encoder), and obtaining the reconstructed image corresponding to each training image using a decoder (e.g., a VAE decoder) based on the second encoded features, the training loss of the first stage (i.e., the second training loss mentioned above) can be determined according to the exemplary implementation by the following operations. First, based on the differences between each training image and the corresponding reconstructed image, a third training loss is determined. Secondly, the probability of whether the reconstructed image is identified as a real image is determined, and a fourth training loss (also known as "image spatial adversarial loss", hereinafter referred to as) is determined based on the probability. Finally, the second training loss is determined based on the third and fourth training losses. For example, the above probabilities can be determined using a discriminator. For example, the second training loss can be determined using the following Equation 2. : Equation 2 in, These are hyperparameters, which can be any constant, for example... It can be equal to 1, but it is not limited to this.

[0060] According to an exemplary embodiment, It can be determined based on the following equation 3: Equation 3 in, These are training images. It is the reconstructed image corresponding to the training image. It is the output of the discriminator. The probability of whether it is judged as a real image. Indicates sampling from the training dataset For each calculate .

[0061] By determining the training loss in the first stage based on image spatial adversarial loss, the amount of information in the feature space can be increased, allowing the encoded features to retain more information related to the image reconstruction task, which is beneficial to improving the fidelity of the reconstructed image.

[0062] Return to reference Figure 2 After the encoder and decoder of the first model are trained in the first stage, in step S230, the first coding features of each training image in the training image set are obtained by using the trained encoder, and the latent feature vectors corresponding to each training image are obtained by using the latent diffusion model based on the first coding features.

[0063] As an example, latent features corresponding to each training image can be obtained using a latent diffusion model based on the first encoded features; a first feature mapping is then performed on the latent features corresponding to each training image to obtain the latent feature vector corresponding to each training image. For example, a learnable projection head can be used to map the latent features corresponding to each training image to the latent feature vector corresponding to each training image, denoted as ψ(z). This disclosure does not limit the implementation form of the learnable projection head; for example, the learnable projection head can be implemented using a neural network, but is not limited thereto.

[0064] In step S240, the semantic feature vector corresponding to each training image in the training image set is obtained. As an example, step S240 may include: obtaining a pre-trained second model, wherein the second model is trained to obtain the semantic features of the image; using the second model to obtain the semantic features corresponding to each training image in the training image set; and performing a second feature mapping on the semantic features corresponding to each training image to obtain the semantic feature vector corresponding to each training image.

[0065] For example, the second model can be a pre-trained visual base model, but is not limited to this. Any model capable of acquiring semantic features of training images can be pre-trained as the second model. This disclosure does not limit the method of acquiring semantic feature vectors. Furthermore, for example, the semantic features corresponding to each training image can be mapped to semantic feature vectors corresponding to each training image, denoted as φ(x), through another learnable projection head. This disclosure does not limit the implementation of the learnable projection head; for example, the learnable projection head can be implemented using a neural network, but is not limited to this.

[0066] By using the first feature mapping and the second feature mapping, latent features and semantic features can be mapped to the same feature vector space.

[0067] After obtaining the latent feature vector and semantic feature vector corresponding to each training image through steps S230 and S240 respectively, in step S250, a first training loss is determined based on the latent feature vector and semantic feature vector, and a latent diffusion model is trained according to the first training loss. Step S250 is to perform a second-stage training on the first model to train the latent diffusion model. In the following text, the first training loss is also referred to as the second-stage training loss, which can be labeled as... .

[0068] According to an exemplary embodiment, step S250 may include: for each training image in the training image set, determining the similarity between the latent feature vector corresponding to the training image and the semantic feature vectors corresponding to all training images in the training image set; and determining a first training loss based on the similarity. For each training image in the training image set, the similarity between the latent feature vector corresponding to the training image and the semantic feature vectors corresponding to all training images in the training image set includes both the similarity between the latent feature vector corresponding to the training image and the semantic feature vector corresponding to the training image, and the similarity between the latent feature vector corresponding to the training image and the semantic feature vectors corresponding to other training images in the training image set. In the following description, for convenience, for each training image, its latent feature vector ψ(z) i ) and its own semantic feature vector φ(x i ) is considered as a positive sample pair, ψ(z) i The semantic feature vectors φ(x) of other training images j (j≠i) are considered negative sample pairs. In step S250, for each training image in the training image set, the similarity of both positive and negative sample pairs is determined; in other words, for each ψ(z)... i ), calculate its relationship with all φ(x) jThe similarity in step S250 includes each ψ(z) i ) and all φ(x j The similarity between ).

[0069] like Figure 4 As shown, for a training image x, the latent feature vector Z corresponding to the training image is obtained using a VAE encoder and a latent diffusion model (including multiple DiT blocks). i =ψ(z i (i=1 to n), using a pre-trained visual base model to obtain the semantic feature vector F corresponding to the training image. j =φ(x j (j=1 to n), Z i F j Z represents i With F j The similarity, where Z is the similarity when i=j. i With F j To form a positive sample pair, when i is not equal to j, Z i With F j These constitute negative sample pairs. As an example, similarity can be the structural similarity between latent feature vectors and semantic feature vectors, but is not limited thereto. This disclosure does not restrict the type of similarity or the specific method of calculating similarity.

[0070] As an example, after determining the similarity, a normalization function can be used to normalize the similarity for each training image in the training image set, and a first training loss can be determined based on the normalization result. For example, the normalization function can be a normalization exponential function, such as the softmax function, but is not limited to this. According to the embodiment, the similarity can be normalized using a normalization function, the first training loss can be determined based on the normalization result, and a latent diffusion model can be trained based on the first training loss, such that the similarity probability of the positive sample pairs mentioned above is as high as possible, and the similarity probability of the negative sample pairs mentioned above is as low as possible.

[0071] According to an embodiment, optionally, determining the first training loss based on the normalization result may include: determining a third training loss (hereinafter, denoted as...) based on the normalization result. ), and based on the third training loss and the diffusion loss of the potential diffusion model (hereinafter, denoted as Determine the first training loss .

[0072] According to an exemplary embodiment, for example, the first training loss can be determined by the following Equation 4. : Equation 4 in, These are hyperparameters, which can be any constant, for example... It can be equal to 1, but it is not limited to this.

[0073] As an example, when the normalization function is the softmax function, it can be determined by the following equation 5. : Equation 5 In the above formula, for each training image Its latent eigenvector ψ(z) i ) and its own semantic feature vector φ(x i The similarity is , and ψ(z i The semantic feature vector φ(x) of all training images in the training image set. j The similarity of j to (where j is less than or equal to B and greater than or equal to 1, and B is the number of training images in the training image set) is: The result of normalizing these similarities using the softmax function is: It can be determined by taking the negative logarithm of the normalized result. .

[0074] The first training loss can be proven to correspond to a lower bound on the mutual information between the training image and the latent feature vectors obtained from the training image using the encoder and latent diffusion model. By minimizing the first training loss, this lower bound can be maximized, thereby improving the mutual information between the training image and the latent feature vectors.

[0075] For example, such as Figure 3 As shown, in the second stage of model training (Stage II), after obtaining the first encoded features of each training image in the training image set using the trained encoder (e.g., VAE encoder), and obtaining the latent feature vectors corresponding to each training image based on the first encoded features using a latent diffusion model (including multiple DiT blocks), the semantic feature vectors corresponding to each training image can be further obtained. Based on the latent feature vectors and semantic feature vectors, the model is then determined. and based on and diffusion loss Determine the training loss in the second phase .

[0076] The preceding text describes determining the first training loss based on the latent feature vectors and semantic feature vectors corresponding to each training image, and then training the latent diffusion model based on this first training loss. For example, for each training image in the training image set, the similarity between the latent feature vector corresponding to that training image and the semantic feature vectors corresponding to all training images in the set is determined, and the first training loss is determined based on this similarity. This approach forces the latent diffusion model to retain information aligned with semantic features in the latent space; that is, it allows the latent feature vectors to retain more semantically relevant and truly useful content for image reconstruction. This improves the mutual information between the latent feature vectors of the training images, facilitating the reconstruction of details in the images and enabling the training of a model that improves the fidelity of image reconstruction.

[0077] Optionally, considering the varying complexity and uncertainty of different training images, this disclosure further proposes imposing different constraints on different training images to give more attention to training images with high complexity or uncertainty. To this end, according to an exemplary implementation of this disclosure, the determination of the first training loss based on similarity mentioned above may include: obtaining the entropy value corresponding to each training image in the training image set; obtaining a first parameter value corresponding to each training image based on the entropy value; and determining the first training loss based on the first parameter value and similarity. For example, the first parameter value may be positively correlated with the entropy value, but is not limited thereto.

[0078] For example, for each training image, calculate the entropy of its posterior distribution. A higher entropy value indicates greater complexity or uncertainty in the training image. Map the entropy value to the first parameter value. The range of values This yields the first parameter value corresponding to the training image. For example, a linear mapping function can be used to map the entropy value to a range of values. , so that the first parameter value It changes smoothly with the entropy value. For example, the value of the first parameter can be obtained through the following equation 6. : Equation 6 in, It is the sigmoid function. Training images The entropy of the posterior distribution ,in, and These are the minimum and maximum values ​​of the first parameter, respectively. , and These are all hyperparameters and can be set as needed. For example, min can be 0.05 and max can be 0.2.

[0079] After determining the first parameter value for each training image, a first training loss can be determined based on the first parameter value and the similarity. For example, for each training image in the training image set, the similarity is adjusted based on the first parameter value to obtain an adjusted similarity; the adjusted similarity is normalized using a normalization function, and the first training loss is determined based on the normalization result. For example, the adjusted similarity can be obtained by dividing the similarity by the first parameter value. For example, the adjusted similarity can be normalized using a normalization exponential function, and the first training loss is determined based on the logarithm of the normalization result.

[0080] According to an exemplary embodiment, as mentioned above, the first training loss can be determined by the following equation. : , As an example, when the normalization function is the softmax function, it can be determined by the following equation 7. : Equation 7 In the above formula, for each training image Its latent eigenvector ψ(z) i ) and its own semantic feature vector φ(x i The similarity is , ψ(z i The semantic feature vector φ(x) of all training images in the training image set. j The similarity of j to (where j is less than or equal to B and greater than or equal to 1, and B is the number of training images in the training image set) is: By and Divide by respectively Obtain the adjusted similarity and The result of normalizing these adjusted similarities using the softmax function is: It can be determined by taking the negative logarithm of the normalized result. .

[0081] Using Equation 7 above, for each training image The softmax distribution (normalized result) uses As a control parameter, it is adjusted so that for high-entropy samples, Larger values ​​result in a smoother softmax distribution and gentler contrast constraints, allowing the model to explore its representation space more freely on complex samples, while being more lenient on low-entropy samples. Smaller softmax values ​​result in sharper softmax and stronger constraints, leading to a high degree of consistency between the latent representation and semantic features. In this way, the model can adaptively allocate representation power based on the complexity and uncertainty of the samples, giving more attention to complex or highly uncertain samples.

[0082] The above has been illustrated with examples and references. Figure 2 to Figure 4 A method for training a first model according to an exemplary embodiment of the present disclosure is described. According to this method, after training the encoder and decoder of the first model in a first stage of model training, in a second stage of model training, a first training loss is determined based on the latent feature vectors and semantic feature vectors corresponding to each training image, and a latent diffusion model is trained based on the first training loss. Therefore, the latent feature vectors corresponding to the training images can be guided to semantically align with the training images, thereby enabling the latent feature vectors to retain information truly useful for image reconstruction, and thus enabling the training of a first model that can improve the fidelity of the reconstructed image. Optionally, by determining the first-stage training loss in the image space in the first stage of model training, the information content of the feature space can be increased, allowing the encoded features obtained using the encoder to retain more information relevant to the image reconstruction task. Furthermore, optionally, in the second stage of model training, by further adaptively adjusting the value of the first parameter according to the complexity and uncertainty of the samples, samples with high complexity or uncertainty can receive more attention.

[0083] Figure 5 This is a flowchart illustrating a method for image reconstruction according to an exemplary embodiment.

[0084] After the first model is trained according to the method described above for training the first model, image reconstruction can be performed using the first model.

[0085] Reference Figure 5 In step S510, a first image is acquired. The first image may be a low-resolution image to be reconstructed, and this disclosure does not impose any restrictions on the form of the first image or the method of acquiring the first image. In step S520, a second image is acquired by performing the following operations: based on the first image, the encoded features corresponding to the first image are obtained using the encoder included in the first model; based on the encoded features, the latent feature vector corresponding to the first image is obtained using the latent diffusion model included in the first model; based on the latent feature vector, the second image is acquired using the decoder included in the first model.

[0086] according to Figure 5 The method shown improves the fidelity of the obtained second image by using a first model trained according to the method described above for training the first model, based on the first image.

[0087] Optionally, after acquiring the second image, various downstream image processing tasks can be performed using it. For example, downstream image processing tasks can include object detection, object recognition, etc., based on the second image. For instance, optical character recognition can be performed based on the second image. Because the fidelity of the second image is improved, the performance of downstream image processing tasks will also be correspondingly enhanced.

[0088] The method according to the embodiments of this disclosure has been described above. The following description, in conjunction with... Figure 6 and Figure 7 A brief description of the apparatus according to embodiments of the present disclosure is provided.

[0089] Figure 6 This is a block diagram illustrating an apparatus for training a first model according to an exemplary embodiment. According to the exemplary embodiment, the first model includes an encoder, a latent diffusion model, and a decoder.

[0090] Reference Figure 6 The apparatus 600 for training the first model may include a training image set acquisition unit 610, a first training unit 620, and a second training unit 630.

[0091] The training image set acquisition unit 610 can be configured to acquire a training image set. The first training unit 620 can be configured to train an encoder and a decoder based on the training image set. The second training unit 630 can be configured to: obtain the first encoded features of each training image in the training image set using the trained encoder; obtain the latent feature vectors corresponding to each training image using a latent diffusion model based on the first encoded features; acquire the semantic feature vectors corresponding to each training image in the training image set; for each training image in the training image set, determine the similarity between the latent feature vector corresponding to the training image and the semantic feature vectors corresponding to all training images in the training image set; determine a first training loss based on the similarity; and train the latent diffusion model based on the first training loss.

[0092] Figure 7 This is a block diagram illustrating an apparatus for image reconstruction according to an exemplary embodiment. (Refer to...) Figure 7The apparatus 700 for image reconstruction may include a first image acquisition unit 710 and a second image acquisition unit 720. According to an exemplary embodiment, the first image acquisition unit 710 may be configured to acquire a first image. The second image acquisition unit 720 may be configured to acquire a second image by performing the following operations: obtaining encoded features corresponding to the first image using an encoder included in a first model based on the first image; obtaining a latent feature vector corresponding to the first image using a latent diffusion model included in the first model based on the encoded features; and acquiring the second image using a decoder included in the first model based on the latent feature vector. According to an embodiment, the first model is trained according to the method for training the first model described above, and includes an encoder, a latent diffusion model, and a decoder.

[0093] Regarding the apparatus in the above embodiments, the specific manner in which each module performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated upon here.

[0094] Figure 8 This is a block diagram illustrating an electronic device 800 according to an exemplary embodiment. (Refer to...) Figure 8 The electronic device 800 may include a processor 810 and a memory 820 for storing executable instructions of the processor 810, wherein the processor 810 may be configured to execute the instructions to implement the methods described above for training a first model and / or for image reconstruction.

[0095] As an example, an electronic device can be a PC, tablet, personal digital assistant, smartphone, or other device capable of executing the aforementioned set of instructions. Here, the electronic device is not necessarily a single device; it can be any collection of devices or circuits capable of executing the aforementioned instructions (or instruction sets) individually or in combination. The electronic device can also be part of an integrated control system or system manager, or can be configured to interconnect locally or remotely (e.g., via wireless transmission) through an interface.

[0096] In electronic devices, processors may include central processing units (CPUs), graphics processing units (GPUs), programmable logic devices, dedicated processor systems, microcontrollers, or microprocessors. By way of example and not limitation, processors may also include analog processors, digital processors, microprocessors, multi-core processors, processor arrays, network processors, etc.

[0097] The processor can execute instructions or code stored in memory, which can also store data. Instructions and data can also be sent and received over a network via a network interface device, which can employ any known transport protocol.

[0098] Memory can be integrated with the processor; for example, RAM or flash memory can be housed within an integrated circuit microprocessor. Alternatively, memory can comprise a separate device, such as an external disk drive, storage array, or other storage device that can be used by any database system. Memory and processor can be operatively coupled, or can communicate with each other, for example, via I / O ports, network connections, etc., enabling the processor to read files stored in the memory.

[0099] In addition, electronic devices may include video displays (such as liquid crystal displays) and user interaction interfaces (such as keyboards, mice, touch input devices, etc.). All components of the electronic device may be interconnected via buses and / or networks.

[0100] In an exemplary embodiment, a computer-readable storage medium including instructions is also provided, such as a memory 820 including instructions, which, when executed by a processor 810 of an electronic device 800, enables the electronic device 800 to perform the methods described above for training a first model and / or for image reconstruction. Optionally, the computer-readable storage medium may be a ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, and optical data storage device, etc.

[0101] In an exemplary embodiment, a computer program product is also provided, including a computer program / instructions that, when executed by a processor, implement the methods described above for training a first model and / or for image reconstruction.

[0102] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the following claims.

[0103] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.

Claims

1. A method for training a first model, characterized in that, The first model includes an encoder, a latent diffusion model, and a decoder, and the method includes: Obtain the training image set; The encoder and the decoder are trained based on the training image set; The first coding features of each training image in the training image set are obtained using the trained encoder, and the latent feature vector corresponding to each training image is obtained using the latent diffusion model based on the first coding features. Obtain the semantic feature vector corresponding to each training image in the training image set; A first training loss is determined based on the latent feature vector and the semantic feature vector, and the latent diffusion model is trained based on the first training loss.

2. The method according to claim 1, characterized in that, Determining the first training loss based on the latent feature vector and the semantic feature vector includes: For each training image in the training image set, determine the similarity between the latent feature vector corresponding to the training image and the semantic feature vectors corresponding to all training images in the training image set. The first training loss is determined based on the similarity.

3. The method according to claim 2, characterized in that, Determining the first training loss based on the similarity includes: For each training image in the training image set, the similarity is normalized using a normalization function; The first training loss is determined based on the normalization result.

4. The method according to claim 2, characterized in that, Determining the first training loss based on the similarity includes: Obtain the entropy value corresponding to each training image in the training image set; The first parameter value corresponding to each training image is obtained based on the entropy value; The first training loss is determined based on the first parameter value and the similarity.

5. The method according to claim 4, characterized in that, Determining the first training loss based on the first parameter value and the similarity includes: For each training image in the training image set, the similarity is adjusted based on the first parameter value to obtain the adjusted similarity; The adjusted similarity is normalized using a normalization function, and the first training loss is determined based on the normalization result.

6. The method according to claim 5, characterized in that, The step of adjusting the similarity based on the first parameter value to obtain the adjusted similarity includes: dividing the similarity by the first parameter value to obtain the adjusted similarity; Normalizing the adjusted similarity using a normalization function and determining the first training loss based on the normalization result includes: normalizing the adjusted similarity using a normalization exponential function and determining the first training loss based on the logarithm of the normalization result.

7. The method according to any one of claims 1 to 6, characterized in that, Training the encoder and the decoder based on the training image set includes: The encoder is used to obtain the second encoded features of each training image in the training image set; Based on the second encoded feature, the decoder is used to obtain the reconstructed image corresponding to each training image; The second training loss is determined based on each training image, the reconstructed image corresponding to each training image, and the probability of whether the reconstructed image is identified as a real image. The encoder and the decoder are trained according to the second training loss.

8. The method according to claim 7, characterized in that, The determination of the second training loss based on each training image, the reconstructed image corresponding to each training image, and the probability of whether the reconstructed image is identified as a real image includes: A third training loss is determined based on the differences between each training image and the corresponding reconstructed image. Determine the probability of whether the reconstructed image is identified as a real image, and determine the fourth training loss based on the probability; The second training loss is determined based on the third training loss and the fourth training loss.

9. The method according to claim 1, characterized in that, The step of obtaining the latent feature vector corresponding to each training image based on the first encoded feature using the latent diffusion model includes: obtaining the latent features corresponding to each training image based on the first encoded feature using the latent diffusion model; and performing a first feature mapping on the latent features corresponding to each training image to obtain the latent feature vector corresponding to each training image. The step of obtaining the semantic feature vector corresponding to each training image in the training image set includes: Obtain a pre-trained second model, wherein the second model is trained to acquire semantic features of an image; The second model is used to obtain the semantic features corresponding to each training image in the training image set; Perform a second feature mapping on the semantic features corresponding to each training image to obtain the semantic feature vector corresponding to each training image.

10. A method for image reconstruction, characterized in that, include: Get the first image; Obtain the second image by performing the following operations: Based on the first image, the encoder included in the first model is used to obtain the encoded features corresponding to the first image; Based on the encoded features, the latent feature vector corresponding to the first image is obtained using the latent diffusion model included in the first model; Based on the latent feature vectors, the second image is obtained using the decoder included in the first model. The first model is trained using the method described in any one of claims 1 to 9.

11. An apparatus for training a first model, characterized in that, The first model includes an encoder, a latent diffusion model, and a decoder, and the device includes: The training image set acquisition unit is configured to acquire a training image set; The first training unit is configured to train the encoder and the decoder based on the training image set; The second training unit is configured to: obtain first encoded features of each training image in the training image set using the trained encoder; obtain latent feature vectors corresponding to each training image using the latent diffusion model based on the first encoded features; obtain semantic feature vectors corresponding to each training image in the training image set; for each training image in the training image set, determine the similarity between the latent feature vector corresponding to the training image and the semantic feature vectors corresponding to all training images in the training image set; determine a first training loss based on the similarity; and train the latent diffusion model according to the first training loss.

12. An apparatus for image reconstruction, characterized in that, include: The first image acquisition unit is configured to acquire a first image; The second image acquisition unit is configured to acquire a second image by performing the following operations: Based on the first image, the encoder included in the first model is used to obtain the encoded features corresponding to the first image; Based on the encoded features, the latent feature vector corresponding to the first image is obtained using the latent diffusion model included in the first model; Based on the latent feature vectors, the second image is obtained using the decoder included in the first model. The first model is trained using the method described in any one of claims 1 to 9.

13. An electronic device, characterized in that, include: processor; Memory used to store the processor's executable instructions; The processor is configured to execute the instructions to implement the method according to any one of claims 1 to 10.

14. A computer-readable storage medium, wherein instructions in the computer-readable storage medium, when executed by a processor of an electronic device, enable the electronic device to perform the method according to any one of claims 1 to 10.

15. A computer program product comprising a computer program / instructions, characterized in that, When the computer program / instructions are executed by the processor, they implement the method according to any one of claims 1 to 10.

Images