Model training method and image processing method

Abstract

The application discloses a model training method and an image processing method. It relates to the technical field of artificial intelligence, and comprises the following steps: performing forward coding processing on a sample image by an initial normalization flow model to obtain an initial hidden variable corresponding to the sample image; performing inverse change processing on the initial hidden variable based on the initial normalization flow model, and obtaining a first feature vector in the inverse change processing; and training the initial normalization flow model according to the initial hidden variable, the first feature vector and a second feature vector to obtain a target normalization flow model, wherein the second feature vector is obtained by performing feature extraction on the sample image by a visual encoder. The application solves the technical problem of low accuracy of normalization flow models in related technologies in processing images.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and more specifically, to a model training method and an image processing method. Background Technology

[0002] In the field of AI-generated content, Normalizing Flows (NFs) models have attracted attention for their accurate density estimation and efficient data sampling capabilities, particularly demonstrating unique advantages in image synthesis. Theoretically, NFs models can achieve high-quality image generation and deep representation learning by constructing an invertible mapping between the data distribution and a simple prior distribution. However, in practical applications, while theoretically sound, NFs models often neglect the semantic richness of intermediate representations when dealing with complex image details, leading to poor-quality generated images.

[0003] There is currently no effective solution to the above problems. Summary of the Invention

[0004] This application provides a model training method and an image processing method to at least solve the technical problem that the normalized flow model in related technologies has relatively low accuracy in image processing.

[0005] According to one aspect of the embodiments of this application, a model training method is provided, comprising: performing forward encoding processing on a sample image using an initial normalized flow model to obtain initial latent variables corresponding to the sample image; performing inverse transformation processing on the initial normalized flow model based on the initial latent variables and obtaining a first feature vector in the inverse transformation processing; training the initial normalized flow model based on the initial latent variables, the first feature vector, and a second feature vector to obtain a target normalized flow model, wherein the second feature vector is obtained by feature extraction of the sample image using a visual encoder.

[0006] Further, obtaining the first feature vector in the inverse transformation process includes: obtaining the intermediate feature vectors generated by each inverse transformation step in the process of performing inverse transformation based on the initial latent variables of the initial normalized flow model; and obtaining the first feature vector based on the intermediate feature vectors.

[0007] Furthermore, performing inverse transformation processing based on the initial latent variables through the initial normalized flow model includes: separating the computation graph in the initial latent variables to obtain the target latent variable, wherein the computation graph in the initial latent variables is generated during the forward encoding process of the sample image by the initial normalized flow model; and performing inverse transformation processing through the initial normalized flow model starting from the target latent variable.

[0008] Further, training the initial normalized flow model based on the initial latent variables, the first feature vector, and the second feature vector to obtain the target normalized flow model includes: calculating an alignment loss function based on the first feature vector and the second feature vector; calculating a negative log-likelihood loss based on the initial latent variables and the intermediate variables generated when the initial normalized flow model performs forward encoding processing on the sample image; and performing backpropagation training on the initial normalized flow model based on the alignment loss function and the negative log-likelihood loss to obtain the target normalized flow model.

[0009] Further, the alignment loss function is calculated based on the first feature vector and the second feature vector, including: projecting the first feature vector through a projection head to obtain a projected first feature vector, wherein the projected first feature vector and the second feature vector are in the same dimensional space; and calculating the alignment loss function based on the projected first feature vector and the second feature vector.

[0010] According to one aspect of the embodiments of this application, an image processing method is provided, comprising: acquiring a first target image to be classified; processing the first target image through a target normalized flow model to obtain a classification result, wherein the target normalized flow model is trained by the model training method described in any one of the above claims.

[0011] Further, processing the first target image using a target normalized flow model to obtain a classification result includes: obtaining the category embedding matrices corresponding to multiple categories learned by the target normalized flow model during training; calculating a soft category embedding matrix with weight information based on the category embedding matrices; calculating a score value under the soft category embedding matrix using the target normalized flow model on the soft category embedding matrix and the first target image; and obtaining the classification result based on the score value.

[0012] Further, the calculation based on the category embedding matrix to obtain a soft category embedding matrix with weight information includes: initializing the numerical values ​​corresponding to the multiple categories to target numerical values ​​to obtain an initial vector; calculating the initial vector using an activation function to obtain a weight vector; and calculating the weight vector and the category embedding matrix to obtain a soft category embedding matrix with weight information.

[0013] Further, obtaining the classification result based on the score value includes: performing gradient calculation on the score value to obtain the gradient value corresponding to the initial vector; and obtaining the classification result based on the gradient value.

[0014] According to one aspect of the embodiments of this application, an image processing method is provided, comprising: acquiring a target noise distribution and an image category of a second target image to be generated; sampling the target noise distribution using a target normalized flow model to obtain target noise, wherein the target normalized flow model is trained based on the model training method described in any one of the above claims; and processing the target noise based on the image category using the target normalized flow model to output a second target image.

[0015] According to one aspect of the embodiments of this application, a model training apparatus is provided, comprising: a first processing unit, configured to perform forward encoding processing on a sample image using an initial normalized flow model to obtain initial latent variables corresponding to the sample image; a second processing unit, configured to perform inverse transformation processing on the initial normalized flow model based on the initial latent variables, and obtain a first feature vector in the inverse transformation processing; and a training unit, configured to train the initial normalized flow model based on the initial latent variables, the first feature vector, and a second feature vector to obtain a target normalized flow model, wherein the second feature vector is obtained by feature extraction of the sample image using a visual encoder.

[0016] Furthermore, the second processing unit includes: a first acquisition module, used to acquire intermediate feature vectors generated by each inverse transformation during the inverse transformation process based on the initial latent variables of the initial normalized flow model; and a first processing module, used to obtain the first feature vector based on the intermediate feature vectors.

[0017] Furthermore, the second processing unit includes: a separation module, used to separate the computation graph in the initial latent variables to obtain the target latent variables, wherein the computation graph in the initial latent variables is generated during the forward encoding process of the sample image by the initial normalized flow model; and a second processing module, used to perform inverse transformation processing through the initial normalized flow model, starting from the target latent variables.

[0018] Further, the training unit includes: a first calculation module, used to calculate an alignment loss function based on the first feature vector and the second feature vector; a second calculation module, used to calculate a negative log-likelihood loss based on the initial latent variables and the intermediate variables generated when the initial normalized flow model performs forward encoding processing on the sample image; and a training module, used to perform backpropagation training on the initial normalized flow model based on the alignment loss function and the negative log-likelihood loss to obtain the target normalized flow model.

[0019] Further, the first calculation module includes: a projection submodule, used to project the first feature vector through a projection head to obtain a projected first feature vector, wherein the projected first feature vector and the second feature vector are in the same dimensional space; and a first calculation submodule, used to calculate the alignment loss function by calculating the projected first feature vector and the second feature vector.

[0020] According to one aspect of the embodiments of this application, an image processing apparatus is provided, comprising: a first acquisition unit for acquiring a first target image to be classified; and a third processing unit for processing the first target image through a target normalization flow model to obtain a classification result, wherein the target normalization flow model is trained based on the model training method described in any one of the above claims.

[0021] Further, the third processing unit includes: a second acquisition module, used to acquire the category embedding matrices corresponding to multiple categories learned by the target normalized flow model during training; a third calculation module, used to calculate based on the category embedding matrices to obtain a soft category embedding matrix with weight information; a fourth calculation module, used to calculate the soft category embedding matrix and the first target image through the target normalized flow model to obtain a score value under the soft category embedding matrix; and a third processing module, used to obtain the classification result based on the score value.

[0022] Furthermore, the third calculation module includes: a second calculation submodule, used to initialize the numerical values ​​corresponding to the multiple categories to target numerical values ​​to obtain an initial vector; a third calculation submodule, used to calculate the initial vector through an activation function to obtain a weight vector; and a fourth calculation submodule, used to calculate the weight vector and the category embedding matrix to obtain a soft category embedding matrix with weight information.

[0023] Furthermore, the third processing module includes: a fifth calculation submodule, used to perform gradient calculation on the score values ​​to obtain the gradient value corresponding to the initial vector; and a determination submodule, used to obtain the classification result based on the gradient value.

[0024] According to one aspect of the embodiments of this application, an image processing apparatus is provided, comprising: a second acquisition unit, configured to acquire a target noise distribution and an image category of a second target image to be generated; a sampling unit, configured to sample the target noise distribution using a target normalized flow model to obtain target noise, wherein the target normalized flow model is trained based on the model training method described in any one of the above embodiments; and a fourth processing unit, configured to process the target noise based on the image category using the target normalized flow model to output the second target image.

[0025] According to another aspect of the present invention, an electronic device is also provided, comprising: a memory storing an executable program; and a processor for running the program, wherein the program executes a model training method or an image processing method as described above during runtime.

[0026] According to another aspect of the present invention, a computer-readable storage medium is also provided, wherein the storage medium stores a program, wherein the program controls the device where the storage medium is located to execute the model training method or the image processing method described above during runtime.

[0027] According to another aspect of the present invention, a computer program product is also provided, including a computer program or instructions, which, when executed by a processor, implement the model training method or the image processing method described above.

[0028] In this embodiment, the following steps are employed: Forward encoding is performed on the sample image using an initial normalized flow model to obtain the initial latent variables corresponding to the sample image; inverse transformation processing is performed on the initial normalized flow model based on the initial latent variables, and the first feature vector in the inverse transformation processing is obtained; the initial normalized flow model is trained based on the initial latent variables, the first feature vector, and the second feature vector to obtain the target normalized flow model. The second feature vector is obtained by feature extraction from the sample image using a visual encoder, thus solving the technical problem of low accuracy in image processing of normalized flow models in related technologies.

[0029] In this scheme, the sample image is encoded using an initial normalized flow model, transforming it into an initial latent variable representation that is easily processed by the model. Inverse transformation processing is then performed based on these initial latent variables, i.e., proceeding backward along the generation path to gradually recover image features. During the inverse transformation process, a first feature vector is extracted, reflecting the image feature description at a certain stage of the inverse transformation. Then, a second feature vector, obtained by feature extraction from the sample image using a visual encoder, is introduced. Through the initial latent variables, the first feature vector, and the second feature vector, it is ensured that the normalized flow model can generate samples that more closely resemble the semantic features of the original image during inverse image generation, thereby improving the accuracy of image processing. Attached Figure Description

[0030] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0031] Figure 1 This is a hardware structure block diagram of a computer terminal provided according to Embodiment 1 of this application;

[0032] Figure 2 This is a flowchart of the model training method provided in Embodiment 1 of this application;

[0033] Figure 3 This is a schematic diagram of the model training method provided according to Embodiment 1 of this application;

[0034] Figure 4 This is a flowchart of the image processing method provided according to Embodiment 2 of this application;

[0035] Figure 5 This is a flowchart of the image processing method provided according to Embodiment 3 of this application;

[0036] Figure 6 This is a schematic diagram of the model training apparatus provided according to Embodiment 4 of this application;

[0037] Figure 7 This is a schematic diagram of an image processing apparatus provided according to Embodiment 5 of this application;

[0038] Figure 8 This is a schematic diagram of an image processing apparatus provided according to Embodiment Six of this application;

[0039] Figure 9 This is a structural block diagram of an electronic device provided according to Embodiment 7 of this application. Detailed Implementation

[0040] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.

[0041] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0042] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of the relevant data must comply with the relevant laws, regulations and standards of the relevant regions, and corresponding operation entry points are provided for users to choose to authorize or refuse.

[0043] Example 1

[0044] According to an embodiment of this application, a model training method is also provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0045] The method embodiment provided in Embodiment 1 of this application can be executed on a mobile terminal, computer terminal, or similar computing device. Figure 1 A hardware block diagram of a computer terminal (or mobile device) for implementing a model training method is shown. Figure 1 As shown, the computer terminal (or mobile device) 10 may include a processor set 102 (the processor set 102 may include, but is not limited to, a processing device such as a microprocessor MCU or a programmable logic device FPGA, and the processor set 102 may include a processor set, Figure 1 The data is illustrated using 102a, 102b, ..., 102n. A memory 104 is used for storing data, and a transmission module 106 is used for communication functions. In addition, it may include: a display, an input / output interface (I / O interface), a Universal Serial Bus (USB) port (which may be included as one of the ports of a BUS bus), a network interface, a power supply, and / or a camera. Those skilled in the art will understand that... Figure 1 The structure shown is for illustrative purposes only and does not limit the structure of the aforementioned electronic device. For example, computer terminal 10 may also include... Figure 1 The more or fewer components shown, or having the same Figure 1 The different configurations shown.

[0046] It should be noted that the aforementioned one or more processors 102 and / or other data processing circuits are generally referred to herein as "data processing circuits". These data processing circuits may be embodied, in whole or in part, in software, hardware, firmware, or any other combination thereof. Furthermore, the data processing circuits may be a single, independent processing module, or may be integrated, in whole or in part, into any other element within the computer terminal 10 (or mobile device). As involved in the embodiments of this application, the data processing circuits serve as a processor control mechanism (e.g., selection of a variable resistor termination path connected to an interface).

[0047] The memory 104 can be used to store software programs and modules of application software, such as the program instructions / data storage device corresponding to the model training method in this embodiment. The processor 102 executes various functional applications and data processing by running the software programs and modules stored in the memory 104, thereby realizing the above-mentioned model training method. The memory 104 may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include memory remotely located relative to the processor 102, and these remote memories can be connected to the computer terminal 10 via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0048] The transmission device 106 is used to receive or send data via a network. Specific examples of the network described above may include a wireless network provided by the communication provider of the computer terminal 10. In one example, the transmission device 106 includes a Network Interface Controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In another example, the transmission device 106 may be a Radio Frequency (RF) module, used for wireless communication with the Internet.

[0049] The display may be a touchscreen LCD display that allows the user to interact with the user interface of the computer terminal 10 (or mobile device).

[0050] Under the aforementioned operating environment, this application provides the following: Figure 2 The model training method shown. Figure 2 This is a flowchart of a model training method according to Embodiment 1 of this application. The model training method includes:

[0051] Step S201: Perform forward encoding processing on the sample image using the initial normalized flow model to obtain the initial latent variables corresponding to the sample image.

[0052] Alternatively, Normalizing Flows (NFs) are an advanced machine learning technique used in probability density modeling and generative models. The core of NFs is constructing a series of continuous, invertible functions chained together to form a complex transformation chain. This chain transforms the data space into a latent variable space that is easily probabilistically computational, while maintaining the invertibility of the transformation. This means the model can reverse-engineer data points to generate new samples.

[0053] In the model training method provided in Embodiment 1 of this application, an initial normalized flow model to be trained is first obtained. Then, forward encoding processing is performed on the sample image x using the initial normalized flow model. Through an invertible transformation, the image is mapped from its original space to a latent space, resulting in a latent variable (or latent variable) corresponding to the original image. This yields the initial latent variable corresponding to the sample image mentioned above. For example, the initial latent variable is... .

[0054] Step S202: Perform inverse transformation processing based on the initial latent variables using the initial normalized flow model, and obtain the first feature vector in the inverse transformation processing.

[0055] Optionally, a unique feature of the normalized flow model is its reversibility, meaning it can be reversed from the latent variable space back to the data space by applying all the transformations in the encoding process. In step S202, the initial normalized flow model starts from the obtained initial latent variables and reverses through multiple reversible transformations to gradually recover the features of the image, thereby obtaining the sample image recovered from the latent variable space.

[0056] During the inverse transformation process, the normalized flow model goes through multiple intermediate layers, which generate different feature vectors. The first feature vector is obtained from the different feature vectors generated by the intermediate layers.

[0057] Step S203: Train the initial normalized flow model based on the initial latent variables, the first feature vector, and the second feature vector to obtain the target normalized flow model. The second feature vector is obtained by extracting features from the sample image using a visual encoder.

[0058] Optionally, in step S201, the sample image is converted into initial latent variables through forward encoding of the initial normalized flow model, and in step S202, the initial latent variables are used to perform inverse transformation processing to recover the features of the image, and the first feature vector is extracted during the inverse transformation process.

[0059] The second feature vector is obtained by extracting features from the sample image using a pre-trained visual encoder. The visual encoder is a specialized model for processing image features, capable of extracting high-level semantic features. Based on the initial latent variables, the first feature vector, and the second feature vector, the initial normalized flow model gradually evolves until it can effectively generate high-quality sample images with rich semantic information in its internal representation. The final target normalized flow model is not only visually highly similar to the training data but also semantically closely related to it, thus achieving a dual improvement in performance and quality.

[0060] In summary, by encoding the sample image using an initial normalized flow model, the sample image is transformed into an initial latent variable representation that is easily processed by the model. Inverse transformation processing is then performed based on the initial latent variables, i.e., proceeding backward along the generation path to gradually recover image features. During the inverse transformation process, a first feature vector is extracted, reflecting the image feature description at a certain stage of the inverse transformation process. Then, a second feature vector, obtained by feature extraction from the sample image using a visual encoder, is introduced. Through the initial latent variables, the first feature vector, and the second feature vector, it is ensured that the normalized flow model can generate samples that more closely resemble the semantic features of the original image when generating images in reverse, thereby improving the accuracy of image processing.

[0061] To improve the training effect of the ground model, in the model training method provided in Embodiment 1 of this application, obtaining the first feature vector in the inverse transformation process includes: obtaining the intermediate feature vector generated by each step of the inverse transformation of the initial normalized flow model in the process of performing inverse transformation based on the initial latent variables; and obtaining the first feature vector based on the intermediate feature vector.

[0062] Optionally, in the inverse transformation process, the initial normalized flow model gradually recovers from the latent variable space to the image space. Each inverse transformation step corresponds to the processing result of a specific layer inside the model. That is, during the inverse transformation process based on the initial latent variables, the initial normalized flow model generates multiple intermediate feature variables. The intermediate feature vectors reflect the image feature state of the model at different stages on the generation path.

[0063] Having obtained all intermediate feature vectors, the first feature vector can be determined using specific strategies or methods. For example, the initial normalized flow model can be selected through human experience. Intermediate features of specific layers l and blocks t As the first feature vector mentioned above, the first feature vector can also be automatically selected by calculating the contribution or importance of feature vectors from different layers or blocks to the final generated image. For example, sensitivity analysis or gradient-based importance scoring can be used to determine which feature vectors have a significant impact on the quality of the generated image due to changes during the generation process; multiple intermediate feature vectors can also be fused to form a comprehensive first feature vector; or an intermediate feature vector can be randomly selected as the final first feature vector.

[0064] By selecting suitable options from the intermediate feature vectors as the first feature vector, the semantic alignment of the generated image is effectively guided in the inverse transformation process of the normalized flow model, thereby improving the quality and semantic consistency of the generated image.

[0065] To avoid mutual interference between the forward process and the inverse transformation during model parameter updates, the model training method provided in Embodiment 1 of this application includes performing inverse transformation processing based on the initial latent variables through the initial normalized flow model: separating the computation graph in the initial latent variables to obtain the target latent variables, wherein the computation graph in the initial latent variables is generated during the forward encoding process of the initial normalized flow model on the sample image; and performing inverse transformation processing through the initial normalized flow model starting from the target latent variables.

[0066] Optionally, when the initial normalized flow model When performing forward encoding on the sample image x, a computation graph is generated. This graph records the operations and dependencies during the forward propagation of the initial normalized flow model. To prevent gradients from propagating back from the initial latent variables, the computation graph within the initial latent variables is separated. For example, this can be achieved by calling the `detach()` function, which separates the computation graph from the initial latent variables to obtain the target latent variables. Finally, starting with the target latent variable, an inverse transformation is performed using the initial normalized flow model.

[0067] In a normalized flow model, the forward and backward paths share the same set of parameters θ. In traditional training methods, optimizing one path may unintentionally affect the performance of the other. For example, if z is directly used for inverse transformation and gradient calculation, these gradients will simultaneously affect both the forward encoding and backward generation processes, potentially leading to model instability. In the model training method provided in Embodiment 1 of this application, by separating the computation graph before the inverse transformation, an independent generation path is constructed. This ensures that the gradients calculated during the inverse transformation process only affect the model's generation capability, effectively avoiding mutual interference between the forward and backward processes during parameter updates and improving the effectiveness of model training.

[0068] To further improve the effectiveness of normalized flow model training, the model training method provided in Embodiment 1 of this application trains the initial normalized flow model based on the initial latent variables, the first feature vector, and the second feature vector to obtain the target normalized flow model. This includes: calculating the alignment loss function based on the first feature vector and the second feature vector; calculating the negative log-likelihood loss based on the initial latent variables and the intermediate variables generated when the initial normalized flow model performs forward encoding processing on the sample image; and performing backpropagation training on the initial normalized flow model based on the alignment loss function and the negative log-likelihood loss to obtain the target normalized flow model.

[0069] Optionally, to further improve the generation quality and semantic representation capability of the normalized flow model, a comprehensive training strategy combining the alignment loss function and the negative log-likelihood loss is introduced. First, the first feature vector obtained during the inverse transformation process is compared with the second feature vector obtained through a pre-trained visual encoder to measure the similarity or difference between the two. The alignment loss function is calculated to quantify the distance between the intermediate features of the model-generated image and the semantic features of the real image.

[0070] In an alternative embodiment, the alignment loss function can be obtained by calculating the Euclidean distance or KL divergence between the first feature vector and the second feature vector obtained through a pre-trained visual encoder. Using the alignment loss function, the model can learn a generated representation that is more consistent with the semantics of the real image.

[0071] Then, the negative log-likelihood loss function is calculated using the initial latent variable *z* and the intermediate variables generated when the model performs forward encoding on the sample image *x*. The negative log-likelihood loss function consists of the prior probability distribution of the initial latent variable *z* and the Jacobian determinant (i.e., the intermediate variables generated when the initial normalized flow model performs forward encoding on the sample image). In the normalized flow model, the latent variable *z* is typically assumed to follow a simple prior probability distribution, such as a standard Gaussian distribution, or other easily sampled distribution. That is, the model makes explicit assumptions about the distribution of data points in the latent space, and this information determines the probability density of *z*. The Jacobian determinant reflects the spatial distortion effect of the model's forward transformation on the input data.

[0072] Finally, the initial normalized flow model is trained by backpropagation based on the alignment loss function and the negative log-likelihood loss to obtain the target normalized flow model.

[0073] In an alternative embodiment, the alignment loss function can be used. and negative log-likelihood loss The total loss function is obtained by weighted summation. The initial normalized flow model is trained by backpropagation using the total loss function to obtain the target normalized flow model. For example, ,in, Learnable parameters.

[0074] Combining alignment loss and negative log-likelihood loss aims to optimize the visual quality and semantic accuracy of generated images while preserving the statistical properties of the model's generated samples. This comprehensive optimization strategy avoids the trade-off between generation quality and density estimation ability, thereby improving the overall performance of the model.

[0075] The method for calculating the alignment loss function is crucial. In the model training method provided in Embodiment 1 of this application, the alignment loss function is calculated based on the first feature vector and the second feature vector, including: projecting the first feature vector through a projection head to obtain the projected first feature vector, wherein the projected first feature vector and the second feature vector are in the same dimensional space; and calculating the alignment loss function based on the projected first feature vector and the second feature vector.

[0076] Alternatively, to improve the accuracy of calculating the alignment loss function, a lightweight, learnable projection head can be used. The first feature vector h generated in the normalized flow model generation path is subjected to a projection transformation, so that the projected first feature vector and the second feature vector are in the same dimensional space. The projection head can be a simple multilayer perceptron or a complex neural network structure. The dimension and format of the feature vector are adjusted by the projection head so that it is in the same dimensional space as the second feature vector y, which facilitates subsequent comparison and alignment.

[0077] Then, the first feature vector after projection... Second eigenvector The alignment loss function is calculated. For example, the alignment loss function is obtained by calculating the patch-by-patch similarity (such as cosine similarity) of the first and second feature vectors after projection.

[0078] In an optional embodiment, the alignment loss function ,in, This indicates taking the average.

[0079] By calculating the alignment loss function, the normalized flow model can be guided to maintain a high degree of consistency between its internal features and the semantic features of real images when generating images. This strategy greatly enhances the generation effect of the model, especially in the field of image generation, making the generated images not only statistically similar to the training set, but also closer to reality in content and meaning.

[0080] In an optional embodiment, a normalized flow model is initialized, and for a training sample (x), forward encoding is performed through the normalized flow model to map the sample from the image space to the latent variable space (z). This process generates intermediate features (h), which are the model's internal representations along the generation path.

[0081] A lightweight, learnable projector is used to transform the intermediate feature (h) to the same dimension as the encoder's feature space, generating the first projected feature vector. For each sample (x), the encoder extracts its second feature vector (y). Then, the similarity (e.g., cosine similarity) between the first feature vector and (y) is calculated, and the similarity loss across all samples is averaged to obtain the alignment loss. This loss aims to minimize the distance between the model-generated features and the real image features, enhancing the model's semantic awareness. Simultaneously, a negative log-likelihood loss is calculated based on the properties of the normalized flow model to ensure the model can accurately estimate the distribution of training samples, maintaining the model's generative diversity and fidelity.

[0082] Finally, the alignment loss and negative log-likelihood loss are combined to form a comprehensive loss function, which is then used to further normalize the flow model parameters and projection head parameters through the backpropagation algorithm.

[0083] Through iterative execution of the above steps, the normalized flow model not only generates high-quality images but also significantly enhances the semantic information of its internal features. Ultimately, a model with significant improvements in both generated image quality and training efficiency is obtained, while also possessing stronger internal representation capabilities.

[0084] In an alternative embodiment, such as Figure 3 The schematic diagram shows that the normalized flow model consists of multiple normalization blocks. Forward encoding of the training sample (x) is performed using these normalization blocks to obtain the latent variable (z). The latent variable is then detached from the current computation graph, preventing gradient backpropagation. .by Starting from this point, the inverse transformation (generation) of the normalized flow is performed. During the generation process, intermediate features are extracted, and the alignment loss is calculated based on these intermediate features. Backpropagation Since z has been separated, the gradient will only propagate backward along the generation path, updating only the relevant model parameters on the generation path, thus obtaining the final normalized flow model.

[0085] In the model training method provided in Embodiment 1 of this application, the sample image is forward encoded using an initial normalized flow model to obtain the initial latent variables corresponding to the sample image; the initial normalized flow model is then used to perform inverse transformation processing based on the initial latent variables to obtain the first feature vector in the inverse transformation processing; the initial normalized flow model is trained based on the initial latent variables, the first feature vector, and the second feature vector to obtain the target normalized flow model. The second feature vector is obtained by feature extraction of the sample image using a visual encoder, which solves the technical problem of low accuracy of normalized flow models in image processing in related technologies.

[0086] In this scheme, the sample image is encoded using an initial normalized flow model, transforming it into an initial latent variable representation that is easily processed by the model. Inverse transformation processing is then performed based on these initial latent variables, i.e., proceeding backward along the generation path to gradually recover image features. During the inverse transformation process, a first feature vector is extracted, reflecting the image feature description at a certain stage of the inverse transformation. Then, a second feature vector, obtained by feature extraction from the sample image using a visual encoder, is introduced. Through the initial latent variables, the first feature vector, and the second feature vector, it is ensured that the normalized flow model can generate samples that more closely resemble the semantic features of the original image during inverse image generation, thereby improving the accuracy of image processing.

[0087] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this application. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential to this application.

[0088] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods of the various embodiments of this application.

[0089] Example 2

[0090] This application provides, as follows: Figure 4The image processing method shown. Figure 4 This is a flowchart of an image processing method according to Embodiment 2 of this application. The image processing method includes:

[0091] Step S401: Obtain the first target image to be classified;

[0092] Step S402: The first target image is processed by the target normalization flow model to obtain the classification result, wherein the target normalization flow model is trained by any of the above-mentioned model training methods.

[0093] Optionally, a target normalized flow model can be trained using the model training method provided in Embodiment 1 of this application, which can be used to directly achieve zero-shot classification. A first target image to be classified is obtained, and the classification result of the first target image is obtained by processing the first target image using the target normalized flow model.

[0094] To improve classification accuracy, the image processing method provided in Embodiment 2 of this application processes the first target image using a target normalized flow model to obtain classification results, including: obtaining the category embedding matrices corresponding to multiple categories learned by the target normalized flow model during training; calculating a soft category embedding matrix with weight information based on the category embedding matrices; calculating the soft category embedding matrix and the first target image using the target normalized flow model to obtain the score value under the soft category embedding matrix; and obtaining the classification result based on the score value.

[0095] Optionally, during the training phase, the normalized flow model not only learns to generate and reconstruct images, but also learns a set of class embedding matrices (E). The class embedding matrices contain high-level semantic features of all classes in the dataset and can be viewed as feature representations of each class.

[0096] When a new image needs to be classified, a soft class embedding matrix with weight information can be obtained by calculating the class embedding matrix.

[0097] In an optional embodiment, the calculation of a soft class embedding matrix with weight information based on the class embedding matrix includes: initializing the numerical values ​​corresponding to multiple classes to target numerical values ​​to obtain an initial vector; calculating the initial vector using an activation function to obtain a weight vector; and calculating the weight vector and the class embedding matrix to obtain a soft class embedding matrix with weight information.

[0098] First, initialize a logits vector, which is initialized to a zero vector (i.e., the values ​​corresponding to multiple categories are initialized to the target values ​​to obtain the initial vector). Then, the logits vector is calculated using the softmax function (i.e., the activation function mentioned above) to obtain a probability distribution vector, which can be regarded as the weight of the selected category, i.e., the weight vector mentioned above. Finally, the soft category embedding matrix with weight information is calculated based on the weight vector and the category embedding matrix.

[0099] In an optional embodiment, the soft-class embedding matrix is: Here, a weight vector p is obtained by softmax(μ), μ is the logits vector, and E is the class embedding matrix of all classes.

[0100] After obtaining the soft-class embedding matrix, the first target image (x) and the soft-class embedding matrix are input into the target normalized flow model to calculate the conditional log-likelihood score under the given soft-class embedding matrix, i.e., the score under the soft-class embedding matrix mentioned above. For example, the log-likelihood score is... .

[0101] Finally, by analyzing the obtained score vectors, the category with the larger score is identified, and this category is used as the classification result.

[0102] In an optional embodiment, obtaining the classification result based on the score value includes: performing gradient calculation on the score value to obtain the gradient value corresponding to the initial vector; and obtaining the classification result based on the gradient value.

[0103] After obtaining the log-likelihood score, a gradient vector (i.e., the gradient value mentioned above) can be calculated from the log-likelihood score. The gradient vector represents the impact of fine-tuning the logits value on the classification result, that is, which category's logits value can be adjusted to improve the image's likelihood the fastest. Therefore, the category of the first target image can be determined directly based on the direction of the gradient value, and thus the final classification result can be obtained.

[0104] In an optional embodiment, calculation gradient with respect to the logits vector This gradient intuitively represents which class's weight adjustment will more quickly improve the image's likelihood. We can choose the class corresponding to the component with the larger gradient value as the prediction result, i.e. The entire process requires only one forward and one backward propagation, without any additional training.

[0105] Through the above process, accurate classification can be achieved by constructing a soft class embedding matrix and calculating gradients, even without direct training data for class classification.

[0106] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this application. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential to this application.

[0107] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods of the various embodiments of this application.

[0108] Example 3

[0109] This application provides, as follows: Figure 5 The image processing method shown. Figure 5 This is a flowchart of an image processing method according to Embodiment 3 of this application. The image processing method includes:

[0110] Step S501: Obtain the target noise distribution and the image category of the second target image to be generated;

[0111] Step S502: Sample the target noise distribution using the target normalized flow model to obtain the target noise, wherein the target normalized flow model is trained based on any of the above model training methods;

[0112] Step S503: The target noise is processed based on the image category using the target normalized flow model, and the second target image is output.

[0113] Optionally, at the start of the image generation task, a target noise distribution needs to be determined first. For example, a standard Gaussian distribution. This target noise distribution serves as the starting point for the generation process, providing the model with a basic random input for generating the initial form of the image. In addition to the target noise distribution, the category of the image to be generated also needs to be specified, such as "cat" or "dog." Category information provides direction for the generation process, guiding the model to generate images related to the features of that category.

[0114] A target normalized flow model is trained using the model training method provided in Embodiment 1 of this application. This model is then used to extract samples from the target noise distribution. The normalized flow model has the ability to map high-dimensional data to low-dimensional random variables, and conversely, it can map low-dimensional random variables back to the high-dimensional data space, i.e., the image pixel space. Through this characteristic of the model, samples extracted from a simple noise distribution can be transformed into potential image information. The target normalized flow model transforms the target noise into an image with specific category attributes, i.e., the second target image mentioned above, through conditional transformation.

[0115] Under the model training strategy provided in Embodiment 1 of this application, the normalized flow model can not only significantly improve the quality of generated images, but also achieve a good balance between training efficiency and model generalization ability. Through the execution of steps S501 to S503, the normalized flow model can stably generate high-quality images with specific category attributes, while maintaining a high degree of diversity in the generated results, thus meeting the dual requirements of innovation and accuracy in the field of image generation.

[0116] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this application. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential to this application.

[0117] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods of the various embodiments of this application.

[0118] Example 4

[0119] According to embodiments of this application, a model training apparatus for implementing the above-described model training method is also provided, such as... Figure 6 As shown, the device includes: a first processing unit 601, a second processing unit 602, and a training unit 603.

[0120] The first processing unit 601 is used to perform forward encoding processing on the sample image through the initial normalized flow model to obtain the initial latent variables corresponding to the sample image.

[0121] The second processing unit 602 is used to perform inverse transformation processing based on the initial latent variables through the initial normalized flow model, and to obtain the first feature vector in the process of performing inverse transformation processing;

[0122] Training unit 603 is used to train the initial normalized flow model based on the initial latent variables, the first feature vector, and the second feature vector to obtain the target normalized flow model. The second feature vector is obtained by extracting features from the sample image through a visual encoder.

[0123] In the model training apparatus provided in Embodiment 4 of this application, the first processing unit 601 performs forward encoding processing on the sample image using an initial normalized flow model to obtain the initial latent variables corresponding to the sample image; the second processing unit 602 performs inverse transformation processing based on the initial latent variables using the initial normalized flow model and obtains the first feature vector in the inverse transformation processing; the training unit 603 trains the initial normalized flow model based on the initial latent variables, the first feature vector, and the second feature vector to obtain the target normalized flow model. The second feature vector is obtained by feature extraction of the sample image using a visual encoder, which solves the technical problem of low accuracy of normalized flow models in image processing in related technologies.

[0124] In this scheme, the sample image is encoded using an initial normalized flow model, transforming it into an initial latent variable representation that is easily processed by the model. Inverse transformation processing is then performed based on these initial latent variables, i.e., proceeding backward along the generation path to gradually recover image features. During the inverse transformation process, a first feature vector is extracted, reflecting the image feature description at a certain stage of the inverse transformation. Then, a second feature vector, obtained by feature extraction from the sample image using a visual encoder, is introduced. Through the initial latent variables, the first feature vector, and the second feature vector, it is ensured that the normalized flow model can generate samples that more closely resemble the semantic features of the original image during inverse image generation, thereby improving the accuracy of image processing.

[0125] Optionally, in the model training apparatus provided in Embodiment 4 of this application, the second processing unit includes: a first acquisition module, used to acquire the intermediate feature vectors generated by each inverse transformation during the inverse transformation process based on the initial latent variables of the initial normalized flow model; and a first processing module, used to obtain a first feature vector based on the intermediate feature vectors.

[0126] Optionally, in the model training apparatus provided in Embodiment 4 of this application, the second processing unit includes: a separation module, used to separate the computation graph in the initial latent variables to obtain the target latent variables, wherein the computation graph in the initial latent variables is generated during the process of the initial normalized flow model performing forward encoding processing on the sample image; and a second processing module, used to perform inverse transformation processing with the target latent variables as the starting point through the initial normalized flow model.

[0127] Optionally, in the model training apparatus provided in Embodiment 4 of this application, the training unit includes: a first calculation module, used to calculate based on the first feature vector and the second feature vector to obtain an alignment loss function; a second calculation module, used to calculate based on the initial latent variables and the intermediate variables generated when the initial normalized flow model performs forward encoding processing on the sample image to obtain a negative log-likelihood loss; and a training module, used to perform backpropagation training on the initial normalized flow model based on the alignment loss function and the negative log-likelihood loss to obtain a target normalized flow model.

[0128] Optionally, in the model training apparatus provided in Embodiment 4 of this application, the first calculation module includes: a projection submodule, used to project the first feature vector through a projection head to obtain the projected first feature vector, wherein the projected first feature vector and the second feature vector are in the same dimensional space; and a first calculation submodule, used to calculate the projected first feature vector and the second feature vector to obtain an alignment loss function.

[0129] It should be noted that the first processing unit 601, the second processing unit 602, and the training unit 603 mentioned above correspond to steps S2101 to S203 in Embodiment 1. The three units and their corresponding steps implement the same instances and application scenarios, but are not limited to the content disclosed in Embodiment 1. It should also be noted that the above modules, as part of the device, can run on the computer terminal 10 provided in Embodiment 1.

[0130] It should be noted that the preferred implementation schemes involved in the above embodiments of this application are the same as the schemes, application scenarios and implementation processes provided in Embodiment 1, but are not limited to the schemes provided in Embodiment 1.

[0131] Example 5

[0132] According to embodiments of this application, an image processing apparatus for implementing the above-described image processing method is also provided, such as... Figure 7 As shown, the device includes a first acquisition unit 701 and a third processing unit 702.

[0133] The first acquisition unit 701 is used to acquire the first target image to be classified;

[0134] The third processing unit 702 is used to process the first target image through the target normalization flow model to obtain the classification result, wherein the target normalization flow model is trained based on any of the above-mentioned model training methods.

[0135] Optionally, in the image processing apparatus provided in Embodiment 5 of this application, the third processing unit includes: a second acquisition module, used to acquire the category embedding matrices corresponding to multiple categories learned by the target normalized flow model during training; a third calculation module, used to calculate based on the category embedding matrices to obtain a soft category embedding matrix with weight information; a fourth calculation module, used to calculate the soft category embedding matrix and the first target image through the target normalized flow model to obtain the score value under the soft category embedding matrix; and a third processing module, used to obtain the classification result based on the score value.

[0136] Optionally, in the image processing apparatus provided in Embodiment 5 of this application, the third calculation module includes: a second calculation submodule, used to initialize the numerical values ​​corresponding to multiple categories to target numerical values ​​to obtain an initial vector; a third calculation submodule, used to calculate the initial vector through an activation function to obtain a weight vector; and a fourth calculation submodule, used to calculate the weight vector and the category embedding matrix to obtain a soft category embedding matrix with weight information.

[0137] Optionally, in the image processing apparatus provided in Embodiment 5 of this application, the third processing module includes: a fifth calculation submodule, used to perform gradient calculation on the score values ​​to obtain the gradient value corresponding to the initial vector; and a determination submodule, used to obtain the classification result based on the gradient value.

[0138] It should be noted here that the first acquisition unit 701 and the third processing unit 702 mentioned above correspond to steps S401 to S403 in Embodiment 2. The two units and the corresponding steps implement the same instances and application scenarios, but are not limited to the content disclosed in Embodiment 2.

[0139] Example 6

[0140] According to embodiments of this application, an image processing apparatus for implementing the above-described image processing method is also provided, such as... Figure 8 As shown, the device includes: a second acquisition unit 801, a sampling unit 802, and a fourth processing unit 803.

[0141] The second acquisition unit 801 is used to acquire the target noise distribution and the image category of the second target image to be generated;

[0142] The sampling unit 802 is used to sample the target noise distribution through the target normalized flow model to obtain the target noise, wherein the target normalized flow model is trained based on the model training method of any one of the above.

[0143] The fourth processing unit 803 is used to process the target noise based on the image category using a target normalized flow model and output the second target image.

[0144] It should be noted here that the second acquisition unit 801, the sampling unit 802 and the fourth processing unit 803 mentioned above correspond to steps S501 to S503 in Embodiment 3. The three units and the corresponding steps implement the same instances and application scenarios, but are not limited to the content disclosed in Embodiment 3.

[0145] Example 7

[0146] Embodiments of this application may provide an electronic device, which may be any one of a group of electronic device terminals. Optionally, in this embodiment, the aforementioned electronic device may also be replaced by a terminal device such as a mobile terminal.

[0147] Optionally, in this embodiment, the aforementioned electronic device may be located in at least one of a plurality of network devices in a computer network.

[0148] In this embodiment, the above-mentioned electronic device can execute the program code of the following steps in the model training method and image processing method: performing forward encoding processing on the sample image through the initial normalized flow model to obtain the initial latent variables corresponding to the sample image; performing inverse transformation processing based on the initial latent variables through the initial normalized flow model and obtaining the first feature vector in the inverse transformation processing; training the initial normalized flow model according to the initial latent variables, the first feature vector and the second feature vector to obtain the target normalized flow model, wherein the second feature vector is obtained by feature extraction of the sample image through a visual encoder.

[0149] The aforementioned electronic device can execute the following steps in the model training method and image processing method: obtaining the first feature vector in the inverse transformation process includes: obtaining the intermediate feature vectors generated by each inverse transformation step in the process of performing inverse transformation based on the initial latent variables of the initial normalized flow model; and obtaining the first feature vector based on the intermediate feature vectors.

[0150] The aforementioned electronic device can execute the following steps in the model training method and image processing method: performing inverse transformation processing based on the initial latent variables through the initial normalized flow model, including: separating the computation graph in the initial latent variables to obtain the target latent variables, wherein the computation graph in the initial latent variables is generated during the forward encoding process of the initial normalized flow model on the sample image; and performing inverse transformation processing through the initial normalized flow model starting from the target latent variables.

[0151] The aforementioned electronic device can execute the following steps in the model training method and image processing method: training the initial normalized flow model based on the initial latent variables, the first feature vector, and the second feature vector to obtain the target normalized flow model includes: calculating the alignment loss function based on the first feature vector and the second feature vector; calculating the negative log-likelihood loss based on the initial latent variables and the intermediate variables generated when the initial normalized flow model performs forward encoding processing on the sample image; and performing backpropagation training on the initial normalized flow model based on the alignment loss function and the negative log-likelihood loss to obtain the target normalized flow model.

[0152] The aforementioned electronic device can execute the following steps in the model training method and image processing method: calculating the alignment loss function based on the first feature vector and the second feature vector, including: projecting the first feature vector through the projection head to obtain the projected first feature vector, wherein the projected first feature vector and the second feature vector are in the same dimensional space; calculating the alignment loss function based on the projected first feature vector and the second feature vector.

[0153] The aforementioned electronic device can execute the following steps in the model training method and image processing method: acquiring a first target image to be classified; processing the first target image through a target normalization flow model to obtain a classification result, wherein the target normalization flow model is trained by any of the above-mentioned model training methods.

[0154] The aforementioned electronic device can execute the following steps in the model training method and image processing method: processing the first target image through the target normalized flow model to obtain the classification result includes: obtaining the category embedding matrix corresponding to multiple categories learned by the target normalized flow model during training; calculating based on the category embedding matrix to obtain the soft category embedding matrix with weight information; calculating the soft category embedding matrix and the first target image through the target normalized flow model to obtain the score value under the soft category embedding matrix; and obtaining the classification result based on the score value.

[0155] The aforementioned electronic device can execute the following steps in the model training method and image processing method: calculating a soft class embedding matrix with weight information based on the class embedding matrix, including: initializing the numerical values ​​corresponding to multiple classes to target numerical values ​​to obtain an initial vector; calculating the initial vector through an activation function to obtain a weight vector; and calculating the weight vector and the class embedding matrix to obtain a soft class embedding matrix with weight information.

[0156] The aforementioned electronic device can execute the following steps in the model training method and image processing method: obtaining the classification result based on the score value includes: performing gradient calculation on the score value to obtain the gradient value corresponding to the initial vector; and obtaining the classification result based on the gradient value.

[0157] The aforementioned electronic device can execute the following steps in the model training method and image processing method: obtaining the target noise distribution and the image category of the second target image to be generated; sampling the target noise distribution through the target normalized flow model to obtain the target noise, wherein the target normalized flow model is trained based on any of the above model training methods; processing the target noise based on the image category through the target normalized flow model to output the second target image.

[0158] Optionally, Figure 9 This is a structural block diagram of an electronic device according to an embodiment of this application. Figure 9 As shown, the electronic device 90 may include: one or more ( Figure 9 (Only one is shown in the image) Processor 902 and memory 904. The electronic device 90 may also include a memory controller to control and manage the memory 904; the electronic device 90 may also include a peripheral interface to connect to a radio frequency module, an audio module, and a display screen, etc.

[0159] The memory can be used to store software programs and modules, such as the program instructions / modules corresponding to the model training method and image processing method and apparatus in the embodiments of this application. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory, thereby realizing the aforementioned model training method and image processing method. The memory may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory may further include memory remotely located relative to the processor, and these remote memories can be connected to the electronic device 90 via a network. Examples of the aforementioned networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0160] The processor can access the information and application programs stored in the memory via the transmission device to perform the following steps: performing forward encoding on the sample image using an initial normalized flow model to obtain the initial latent variables corresponding to the sample image; performing inverse transformation processing based on the initial latent variables using the initial normalized flow model and obtaining the first feature vector in the inverse transformation processing; training the initial normalized flow model based on the initial latent variables, the first feature vector, and the second feature vector to obtain the target normalized flow model, wherein the second feature vector is obtained by feature extraction of the sample image using a visual encoder.

[0161] The processor can call the information and application program stored in the memory through the transmission device to perform the following steps: obtaining the first feature vector in the inverse transformation process includes: obtaining the intermediate feature vectors generated by each step of the inverse transformation of the initial normalized flow model in the process of performing the inverse transformation based on the initial latent variables; and obtaining the first feature vector based on the intermediate feature vectors.

[0162] The processor can access information and applications stored in memory via a transmission device to perform the following steps: performing inverse transformation processing based on initial latent variables through an initial normalized flow model, including: separating the computation graph in the initial latent variables to obtain the target latent variable, wherein the computation graph in the initial latent variables is generated during the forward encoding process of the initial normalized flow model on the sample image; and performing inverse transformation processing starting from the target latent variable through the initial normalized flow model.

[0163] The processor can access information and applications stored in memory via a transmission device to perform the following steps: training an initial normalized flow model based on initial latent variables, a first feature vector, and a second feature vector to obtain a target normalized flow model, including: calculating an alignment loss function based on the first and second feature vectors; calculating a negative log-likelihood loss based on the initial latent variables and intermediate variables generated during forward encoding of sample images using the initial normalized flow model; and performing backpropagation training on the initial normalized flow model based on the alignment loss function and the negative log-likelihood loss to obtain the target normalized flow model.

[0164] The processor can call the information and application program stored in the memory through the transmission device to perform the following steps: calculating the alignment loss function based on the first feature vector and the second feature vector, including: projecting the first feature vector through the projection head to obtain the projected first feature vector, wherein the projected first feature vector and the second feature vector are in the same dimensional space; calculating the alignment loss function based on the projected first feature vector and the second feature vector.

[0165] The processor can call the information and application program stored in the memory through the transmission device to perform the following steps: acquire the first target image to be classified; process the first target image through the target normalization flow model to obtain the classification result, wherein the target normalization flow model is trained by any of the above-mentioned model training methods.

[0166] The processor can access the information and application programs stored in the memory via the transmission device to perform the following steps: processing the first target image using a target normalized flow model to obtain classification results, including: obtaining the category embedding matrices corresponding to multiple categories learned by the target normalized flow model during training; calculating a soft category embedding matrix with weight information based on the category embedding matrices; calculating the soft category embedding matrix and the first target image using the target normalized flow model to obtain the score value under the soft category embedding matrix; and obtaining the classification result based on the score value.

[0167] The processor can access the information and application program stored in the memory via the transmission device to perform the following steps: calculating a soft class embedding matrix with weight information based on the class embedding matrix, including: initializing the values ​​corresponding to multiple classes to the target values ​​to obtain an initial vector; calculating the initial vector using an activation function to obtain a weight vector; and calculating the weight vector and the class embedding matrix to obtain a soft class embedding matrix with weight information.

[0168] The processor can access the information and application program stored in the memory via the transmission device to perform the following steps: obtaining the classification result based on the score values, including: calculating the gradient of the score values ​​to obtain the gradient value corresponding to the initial vector; and obtaining the classification result based on the gradient value.

[0169] The processor can access the information and application program stored in the memory via the transmission device to perform the following steps: obtaining the target noise distribution and the image category of the second target image to be generated; sampling the target noise distribution using a target normalized flow model to obtain the target noise, wherein the target normalized flow model is trained based on any of the above model training methods; processing the target noise based on the image category using the target normalized flow model to output the second target image.

[0170] Those skilled in the art will understand that Figure 9 The structure shown is for illustrative purposes only. Electronic device 90 can also be a smartphone, tablet, handheld computer, mobile internet device (MID), PAD and other terminal devices. Figure 9 This does not limit the structure of the aforementioned electronic device. For example, the electronic device 90 may also include components that are more... Figure 9 The more or fewer components shown (such as network interfaces, display devices, etc.), or having the same Figure 9 The different configurations shown.

[0171] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be implemented by a program instructing the hardware related to the terminal device. The program can be stored in a computer-readable storage medium, which may include: flash drive, read-only memory (ROM), random access memory (RAM), disk or optical disk, etc.

[0172] Example 4

[0173] Embodiments of this application also provide a computer program product. Optionally, in this embodiment, the computer program product can be used to store the program code executed by the model training method and image processing method provided in Embodiment 1.

[0174] Optionally, in this embodiment, the computer program product may be located in any computer terminal in a group of computer terminals in a computer network, or in any mobile terminal in a group of mobile terminals.

[0175] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0176] In the above embodiments of this application, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0177] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual coupling, direct coupling, or communication connection may be through some interfaces; the indirect coupling or communication connection between units or modules may be electrical or other forms.

[0178] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0179] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0180] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard drive, magnetic disk, or optical disk.

[0181] The above description is only a preferred embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. A model training method, characterized in that, include: The initial latent variables corresponding to the sample image are obtained by performing forward encoding processing on the sample image through the initial normalized flow model; The inverse transformation process is performed on the initial normalized flow model based on the initial latent variables, and the first feature vector in the inverse transformation process is obtained. The initial normalized flow model is trained based on the initial latent variables, the first feature vector, and the second feature vector to obtain the target normalized flow model. The second feature vector is obtained by extracting features from the sample image using a visual encoder. The initial normalized flow model is trained based on the initial latent variables, the first feature vector, and the second feature vector to obtain the target normalized flow model, which includes: The alignment loss function is calculated based on the first feature vector and the second feature vector. The negative log-likelihood loss is calculated based on the initial latent variables and the intermediate variables generated when performing forward encoding on the sample images in the initial normalized flow model. The initial normalized flow model is trained by backpropagation based on the alignment loss function and the negative log-likelihood loss to obtain the target normalized flow model.

2. The method according to claim 1, characterized in that, Obtaining the first feature vector during the inverse transformation process includes: Obtain the intermediate feature vectors generated by each inverse transformation step during the inverse transformation process based on the initial latent variables of the initial normalized flow model; The first feature vector is obtained based on the intermediate feature vector.

3. The method according to claim 1, characterized in that, The inverse transformation process performed on the initial normalized flow model based on the initial latent variables includes: The computation graph in the initial latent variables is separated to obtain the target latent variables, wherein the computation graph in the initial latent variables is generated during the forward encoding process of the sample image by the initial normalized flow model; Starting with the target latent variable, the inverse transformation process is performed through the initial normalized flow model.

4. The method according to claim 1, characterized in that, The alignment loss function is calculated based on the first feature vector and the second feature vector, including: The first feature vector is projected by the projection head to obtain the projected first feature vector, wherein the projected first feature vector and the second feature vector are in the same dimensional space. The alignment loss function is obtained by calculating the first feature vector and the second feature vector after projection.

5. An image processing method, characterized in that, include: Obtain the first target image to be classified; The first target image is processed by a target normalized flow model to obtain a classification result, wherein the target normalized flow model is trained based on the model training method described in any one of claims 1 to 4.

6. The method according to claim 5, characterized in that, The first target image is processed using a target normalized flow model to obtain the following classification results: Obtain the category embedding matrices corresponding to multiple categories learned by the target normalized flow model during training; Based on the category embedding matrix, a soft category embedding matrix with weight information is obtained; The soft-class embedding matrix and the first target image are calculated using the target normalized flow model to obtain the score value under the soft-class embedding matrix; The classification result is obtained based on the score.

7. The method according to claim 6, characterized in that, The soft class embedding matrix with weight information is calculated based on the class embedding matrix, including: The numerical values ​​corresponding to the multiple categories are initialized to the target numerical values ​​to obtain an initial vector; The initial vector is calculated using an activation function to obtain the weight vector; The weight vector and the category embedding matrix are calculated to obtain a soft category embedding matrix with weight information.

8. The method according to claim 7, characterized in that, Based on the scores, the classification results include: The gradient of the fraction is calculated to obtain the gradient value corresponding to the initial vector; The classification result is obtained based on the gradient value.

9. An image processing method, characterized in that, include: Obtain the target noise distribution and the image category of the second target image to be generated; The target noise distribution is sampled by a target normalized flow model to obtain the target noise, wherein the target normalized flow model is trained based on the model training method described in any one of claims 1 to 4; The target noise is processed based on the image category using the target normalized flow model to output a second target image.

10. A model training device, characterized in that, include: The first processing unit is used to perform forward encoding processing on the sample image through an initial normalized flow model to obtain the initial latent variables corresponding to the sample image. The second processing unit is used to perform inverse transformation processing based on the initial latent variables through the initial normalized flow model, and to obtain the first feature vector in the process of performing inverse transformation processing; The training unit is used to train the initial normalized flow model based on the initial latent variables, the first feature vector, and the second feature vector to obtain the target normalized flow model, wherein the second feature vector is obtained by feature extraction of the sample image through a visual encoder; The training unit includes: a first calculation module for calculating an alignment loss function based on the first feature vector and the second feature vector; a second calculation module for calculating a negative log-likelihood loss based on the initial latent variables and intermediate variables generated when the initial normalized flow model performs forward encoding on the sample image; and a training module for backpropagating the initial normalized flow model based on the alignment loss function and the negative log-likelihood loss to obtain the target normalized flow model.

11. An image processing apparatus, characterized in that, include: The first acquisition unit is used to acquire the first target image to be classified. The third processing unit is used to process the first target image through a target normalized flow model to obtain a classification result, wherein the target normalized flow model is trained based on the model training method described in any one of claims 1 to 4.

12. An image processing apparatus, characterized in that, include: The second acquisition unit is used to acquire the target noise distribution and the image category of the second target image to be generated; A sampling unit is used to sample the target noise distribution through a target normalized flow model to obtain target noise, wherein the target normalized flow model is trained based on the model training method described in any one of claims 1 to 4; The fourth processing unit is used to process the target noise based on the image category using the target normalized flow model, and output the second target image.

13. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a stored program, wherein, when the program is executed, it controls the device on which the storage medium is located to perform the model training method according to any one of claims 1 to 4, or the image processing method according to any one of claims 5 to 9.

14. An electronic device, characterized in that, include: Memory, which stores executable programs; A processor for running the program, wherein the program, when running, performs the model training method according to any one of claims 1 to 4, or the image processing method according to any one of claims 5 to 9.

15. A computer program product, characterized in that, It includes a computer program or instructions that, when executed by a processor, implement the model training method according to any one of claims 1 to 4, or the image processing method according to any one of claims 5 to 9.

Images