Target domain image generation method and device based on zero-shot adaptation, electronic equipment, storage medium and program product

Abstract

This disclosure provides a method, apparatus, electronic device, storage medium, and program product for generating target domain images based on zero-shot adaptation. The method includes: processing a first noise vector using a target domain mapping network to obtain a first latent vector; processing the first latent vector using a pre-trained synthesis network to obtain a first generated image; wherein the target domain mapping network is trained through the following steps: processing a second noise vector using a mapping network to be trained to obtain a second latent vector, wherein the initial parameters of the mapping network to be trained are equal to the parameters of the pre-trained mapping network corresponding to the pre-trained synthesis network; processing the second latent vector using the pre-trained synthesis network to obtain a second generated image; determining a priori guidance loss based on target domain prior guidance information and the second generated image; determining a total loss based on the prior guidance loss; and adjusting the parameters of the mapping network to be trained based on the total loss to obtain the target domain mapping network.

Description

Technical Field

[0001] This disclosure generally relates to the field of computer vision technology, and more specifically, to a method, apparatus, electronic device, storage medium, and program product for generating target domain images based on zero-shot adaptation. Background Technology

[0002] With the continuous development of deep generative model technology, Generative Adversarial Networks (GANs) have been widely used in various visual tasks such as image generation, image editing, image enhancement, and data augmentation due to their advantages in high-quality image synthesis. By training on large-scale datasets, GANs can effectively learn complex image distributions and generate image results with high realism and detail.

[0003] In practical applications, it is often necessary to transfer or adapt generative adversarial networks (GANs) already trained on source domain data to a new target domain to meet image generation requirements under specific scenarios or semantics. Existing GAN domain adaptation methods typically rely on image samples in the target domain, guiding the generative model to converge towards the target domain distribution through fine-tuning with a small number of samples or conditional constraints. However, in many real-world scenarios, target domain image samples are difficult to obtain or costly to acquire, making the aforementioned methods ineffective when target domain samples are unavailable or extremely limited. Summary of the Invention

[0004] This disclosure provides a target domain image generation method, apparatus, electronic device, storage medium, and program product based on zero-sample adaptation, for solving at least one of the above-mentioned problems.

[0005] According to a first aspect of the present disclosure, a target domain image generation method based on zero-shot adaptation is provided, comprising: processing a first noise vector using a target domain mapping network to obtain a first latent vector, wherein the first noise vector is sampled from a prior noise distribution; processing the first latent vector using a pre-trained synthesis network to obtain a first generated image; wherein the target domain mapping network is trained by: processing a second noise vector using a mapping network to be trained to obtain a second latent vector, wherein the initial parameters of the mapping network to be trained are equal to the parameters of a pre-trained mapping network corresponding to the pre-trained synthesis network, and the second noise vector is sampled from the prior noise distribution; processing the second latent vector using the pre-trained synthesis network to obtain a second generated image; determining a prior guidance loss based on target domain prior guidance information and the second generated image; determining a total loss based on the prior guidance loss; and adjusting the parameters of the mapping network to be trained based on the total loss to obtain the target domain mapping network.

[0006] Optionally, determining the prior guidance loss based on the target domain prior guidance information and the second generated image includes: processing the target domain prior guidance information and the second generated image using a guidance network to obtain a prior guidance signal, wherein the prior guidance signal is used to represent the degree of consistency between the target domain prior guidance information and the second generated image; and determining the prior guidance loss based on the prior guidance signal.

[0007] Optionally, the number of the second latent vectors is multiple, wherein determining the total loss based on the prior guidance loss includes: determining a perceptual clustering loss based on the multiple second latent vectors, wherein the perceptual clustering loss is used to increase the difference between different second latent vectors; and determining the total loss based on the prior guidance loss and the perceptual clustering loss.

[0008] Optionally, determining the perceptual clustering loss based on a plurality of second latent vectors includes: performing random augmentation on each second latent vector to obtain an augmentation vector corresponding to the second latent vector; constructing a plurality of positive sample pairs based on each second latent vector and its corresponding augmentation vector, and determining the difference between each positive sample pair as a positive pair difference; constructing a plurality of negative sample pairs based on different second latent vectors and the augmentation vectors corresponding to each second latent vector, and determining the difference between each negative sample pair as a negative pair difference, wherein the negative samples in each negative sample pair originate from different second latent vectors; and determining the perceptual clustering loss based on the positive pair difference and the negative pair difference, wherein the perceptual clustering loss is positively correlated with the positive pair difference and negatively correlated with the negative pair difference.

[0009] Optionally, in determining the perceptual clustering loss, each positive difference has a positive weight greater than 0, and each positive weight is positively correlated with the corresponding positive difference; each negative difference has a negative weight greater than 0, and each negative weight is negatively correlated with the corresponding negative difference.

[0010] Optionally, determining the total loss based on the prior guidance loss includes: processing the second noise vector using the pre-trained mapping network to obtain a reference latent vector; determining the offset constraint loss based on the reference latent vector and the second latent vector; and determining the total loss based on the offset constraint loss and the prior guidance loss.

[0011] According to a second aspect of the present disclosure, a target domain image generation apparatus based on zero-shot adaptation is provided, comprising: a mapping unit configured to process a first noise vector using a target domain mapping network to obtain a first latent vector, wherein the first noise vector is sampled from a prior noise distribution; and a synthesis unit configured to process the first latent vector using a pre-trained synthesis network to obtain a first generated image; wherein the target domain mapping network is trained by the following steps: processing a second noise vector using a mapping network to be trained to obtain a second latent vector, wherein the initial parameters of the mapping network to be trained are equal to the parameters of a pre-trained mapping network corresponding to the pre-trained synthesis network, and the second noise vector is sampled from the prior noise distribution; processing the second latent vector using the pre-trained synthesis network to obtain a second generated image; determining a prior guidance loss based on target domain prior guidance information and the second generated image; determining a total loss based on the prior guidance loss; and adjusting the parameters of the mapping network to be trained based on the total loss to obtain the target domain mapping network.

[0012] Optionally, determining the prior guidance loss based on the target domain prior guidance information and the second generated image includes: processing the target domain prior guidance information and the second generated image using a guidance network to obtain a prior guidance signal, wherein the prior guidance signal is used to represent the degree of consistency between the target domain prior guidance information and the second generated image; and determining the prior guidance loss based on the prior guidance signal.

[0013] Optionally, the number of the second latent vectors is multiple, wherein determining the total loss based on the prior guidance loss includes: determining a perceptual clustering loss based on the multiple second latent vectors, wherein the perceptual clustering loss is used to increase the difference between different second latent vectors; and determining the total loss based on the prior guidance loss and the perceptual clustering loss.

[0014] Optionally, determining the perceptual clustering loss based on a plurality of second latent vectors includes: performing random augmentation on each second latent vector to obtain an augmentation vector corresponding to the second latent vector; constructing a plurality of positive sample pairs based on each second latent vector and its corresponding augmentation vector, and determining the difference between each positive sample pair as a positive pair difference; constructing a plurality of negative sample pairs based on different second latent vectors and the augmentation vectors corresponding to each second latent vector, and determining the difference between each negative sample pair as a negative pair difference, wherein the negative samples in each negative sample pair originate from different second latent vectors; and determining the perceptual clustering loss based on the positive pair difference and the negative pair difference, wherein the perceptual clustering loss is positively correlated with the positive pair difference and negatively correlated with the negative pair difference.

[0015] Optionally, in determining the perceptual clustering loss, each positive difference has a positive weight greater than 0, and each positive weight is positively correlated with the corresponding positive difference; each negative difference has a negative weight greater than 0, and each negative weight is negatively correlated with the corresponding negative difference.

[0016] Optionally, determining the total loss based on the prior guidance loss includes: processing the second noise vector using the pre-trained mapping network to obtain a reference latent vector; determining the offset constraint loss based on the reference latent vector and the second latent vector; and determining the total loss based on the offset constraint loss and the prior guidance loss.

[0017] According to a third aspect of the present disclosure, an electronic device is provided, comprising: at least one processor; and at least one memory storing computer-executable instructions, wherein the computer-executable instructions, when executed by the at least one processor, cause the at least one processor to perform a zero-sample adaptation-based target domain image generation method according to an exemplary embodiment of the present disclosure.

[0018] According to a fourth aspect of the present disclosure, a computer-readable storage medium is provided, wherein instructions in the computer-readable storage medium, when executed by at least one processor, cause the at least one processor to perform a zero-sample adaptation-based target domain image generation method according to an exemplary embodiment of the present disclosure.

[0019] According to a fifth aspect of the present disclosure, a computer program product is provided, including computer instructions that, when executed by at least one processor, cause at least one processor to perform a zero-sample adaptation-based target domain image generation method according to an exemplary embodiment of the present disclosure.

[0020] The technical solutions provided by the embodiments of this disclosure bring at least the following beneficial effects: According to the target domain image generation method and apparatus, electronic device, and storage medium based on zero-shot adaptation of this disclosure, based on a generative adversarial network (including a mapping network and a synthesis network) pre-trained on source domain image data, prior guidance information of the target domain is introduced in the target domain adaptation stage, and a prior guidance loss is constructed by combining the image generated in the adaptation stage (i.e., the second generated image). This guides the generation distribution to align with the target domain, enabling target domain adaptation (i.e., zero-shot adaptation) without using real image samples of the target domain. This allows the adapted network to be used for high-quality target domain image generation, improving the applicability and deployment efficiency of generative adversarial networks in cross-domain adaptation tasks. Simultaneously, by keeping the parameters of the pre-trained synthesis network fixed and only optimizing the parameters of the pre-trained mapping network, the adaptation process can be restricted to the latent space mapping stage. This allows the latent vectors obtained by the mapping network from the prior noise to gradually shift their distribution without destroying the source domain generation capability. Thus, while ensuring generation stability, a gradual shift in the target domain distribution is achieved, realizing target domain adaptation.

[0021] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0022] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure, and are not intended to unduly limit this disclosure.

[0023] Figure 1 This is a flowchart of a target domain image generation method based on zero-sample adaptation according to an exemplary embodiment of the present disclosure.

[0024] Figure 2 This is a flowchart of a training method for a target domain mapping network according to an exemplary embodiment of the present disclosure.

[0025] Figure 3 This is a schematic diagram of the architecture of a training method for a target domain mapping network according to a specific embodiment of the present disclosure.

[0026] Figure 4 This is a schematic diagram of an image generation architecture of a pre-trained generative adversarial network according to an exemplary embodiment of the present disclosure.

[0027] Figure 5 This is a block diagram of a target domain image generation apparatus based on zero-sample adaptation according to exemplary embodiments of the present disclosure.

[0028] Figure 6 This is a block diagram of an electronic device according to exemplary embodiments of the present disclosure. Detailed Implementation

[0029] To enable those skilled in the art to better understand the technical solutions of this disclosure, the technical solutions in the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings.

[0030] It should be noted that the terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this disclosure are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this disclosure described herein can be implemented in orders other than those illustrated or described herein. The embodiments described in the following examples do not represent all embodiments consistent with this disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this disclosure as detailed in the appended claims.

[0031] It should be noted that the phrase "at least one of several items" in this disclosure refers to three parallel cases: "any one of the several items", "a combination of any number of the several items", and "all of the several items". For example, "including at least one of A and B" includes the following three parallel cases: (1) including A; (2) including B; (3) including A and B. Another example is "performing at least one of step one and step two", which means the following three parallel cases: (1) performing step one; (2) performing step two; (3) performing both step one and step two.

[0032] Hereinafter, with reference to the accompanying drawings, a method and apparatus for generating target domain images based on zero-sample adaptation, an electronic device, and a storage medium according to exemplary embodiments of the present disclosure will be described in detail.

[0033] Figure 1This is a flowchart of a target domain image generation method based on zero-shot adaptation according to exemplary embodiments of the present disclosure. The method can be executed on an electronic device with sufficient computing power and can specifically use generative adversarial networks, such as, but not limited to, StyleGAN (Style-Based Generator Architecture for GANs). Exemplary embodiments of the present disclosure may, for example, first obtain a pre-trained generative adversarial network (including a mapping network and a synthesis network) on source domain image data. This network has been trained on the source domain image data and is therefore capable of generating high-quality images. Based on this network, by performing target domain adaptation, the network can be transferred to the target domain to generate images in the target domain. It should be understood that the target domain and the source domain are semantically related, specifically meaning that the target domain and the source domain maintain consistency in basic visual features. For example, if the source domain is a large face dataset such as FFHQ (Flickr-Faces-High-Quality), then the target domain should conform to face data standards and undergo appropriate semantic transformations or attribute shifts based on the distribution of the source domain, such as being a face dataset with specific features, and not cartoon animal images. That is, the two should maintain basic consistency in image structure, with only stylistic differences.

[0034] Reference Figure 1 In step S101, the first noise vector is processed using a target domain mapping network to obtain the first potential vector.

[0035] The first noise vector is sampled from the prior noise distribution. The prior noise distribution is a statistical model used to describe the distribution of noise; common prior noise distributions include uniform and normal (Gaussian) distributions. The noise vector sampled from this distribution can serve as the starting point for a generative adversarial network (GAN), enabling it to generate a wide variety of images. It should be understood that the first noise vector here refers to the noise vector input during the image generation stage, while the second noise vector (described later) refers to the noise vector input during the target domain adaptation stage. This naming convention is only used to distinguish the application stage of the noise vector and does not represent a fixed noise vector value. In other words, during the image generation stage, multiple different first noise vectors can be sampled from the prior noise distribution to generate multiple different images, and the same applies to the second noise vector.

[0036] Generative Adversarial Networks (GANs) consist of a mapping network and a synthesis network. The mapping network maps noise vectors to a latent space, obtaining corresponding latent vectors. Its main function is to decouple the latent space, allowing for finer-grained control of the latent vectors, such as enabling layered control over styles like facial features and hair color. The synthesis network transforms the latent vectors into generated images that resemble realistic images. After pre-training on source domain image data, a pre-trained GAN is obtained, which includes a pre-trained mapping network and a pre-trained synthesis network. The target domain mapping network in this exemplary embodiment is obtained by adapting the pre-trained mapping network to the target domain. The following sections will combine... Figure 2 This disclosure describes how target domain adaptation is achieved.

[0037] In step S102, a pre-trained synthetic network is used to process the first latent vector to obtain the first generated image.

[0038] This step corresponds to the operation of the synthesis network. Considering that directly fine-tuning the synthesis network can easily destroy its learned generative ability in the source domain, leading to a significant decrease in generation quality, the exemplary embodiment of this disclosure uses a pre-trained synthesis network in this step. That is, in the target domain adaptation stage, the parameters of the synthesis network remain unchanged, which can restrict the adaptation process to the latent space mapping stage. This allows the latent vectors obtained by the mapping network from the prior noise to gradually undergo distribution shift without destroying the generative ability of the source domain. Thus, while ensuring the stability of generation, a gradual shift in the distribution of the target domain is achieved, thereby realizing target domain adaptation.

[0039] As an example, a lightweight mapping network can be used, and the optimization process only requires adjusting the parameters of the mapping network, which greatly reduces the computational overhead and enables the rapid generation of images that conform to the characteristics of the target domain during testing.

[0040] As an example, a multi-scale feature extraction strategy can be adopted, which can effectively utilize multi-scale information and fully consider the local and global features of the target domain during the image generation process, thereby significantly improving the visual quality and diversity of the generated images and enhancing the adaptability of the synthesis network to different target domains.

[0041] Figure 2 This is a flowchart of a training method for a target domain mapping network according to an exemplary embodiment of the present disclosure.

[0042] Reference Figure 2 In step S201, the second noise vector is processed using the mapping network to be trained to obtain the second potential vector.

[0043] The initial parameters of the mapping network to be trained are equal to the parameters of the pre-trained mapping network corresponding to the pre-trained synthetic network, and the second noise vector is sampled from the prior noise distribution.

[0044] In step S202, a pre-trained synthetic network is used to process the second latent vector to obtain the second generated image.

[0045] Steps S201 and S202 correspond to steps S101 and S102 respectively, and will not be described again.

[0046] In step S203, the prior guidance loss is determined based on the target domain prior guidance information and the second generated image.

[0047] This step uses high-level prior knowledge (such as text descriptions) in the target domain as prior guiding information for the target domain, which can establish a distribution model of the target domain without relying on image samples of the target domain, thus helping to ensure the quality of the generated images.

[0048] As an example, textual descriptions from the target domain can be converted into corresponding features in the latent space, which will guide the synthetic network to generate images of the target domain.

[0049] Optionally, step S203 includes: processing the target domain prior guidance information and the second generated image using a guidance network to obtain a prior guidance signal, wherein the prior guidance signal is used to represent the degree of consistency between the target domain prior guidance information and the second generated image; and determining a prior guidance loss based on the prior guidance signal. In zero-shot target domain adaptation, the prior guidance loss is used to provide target domain consistency constraints for the generated result. The target domain prior guidance information may be, for example, a prior text description of the target domain. By using a guidance network, semantic alignment can be performed between the target domain prior guidance information and the second generated image, for example, including but not limited to mapping the target domain prior guidance information and the second generated image to the same feature space in the latent space, thereby achieving cross-modal prior guidance, enabling the trained target domain mapping network to combine the text description from the target domain with visual features to generate cross-modal features of the target domain, and use them to guide the image generation process. This embodiment enhances the adaptability of generative adversarial networks (GANs) in target domain image generation. It not only ensures high visual quality and semantic consistency of the generated images within the target domain but also captures fine-grained features of the target domain, achieving more accurate image generation. This significantly improves the matching degree between the generated images and the target domain, further enhancing zero-shot adaptation capabilities and ensuring that the generated samples reflect the diversity of the target domain. It should be understood that the prior guidance employed in the exemplary embodiments of this disclosure does not rely on real samples from the target domain. Instead, a pre-trained external guidance network evaluates the generation results, thereby generating a computable prior guidance signal. Since the guidance network and the GAN are independent in terms of training data and parameters, their output only participates in the adaptation process as an evaluation function and does not introduce target domain sample information, thus not violating the zero-shot hypothesis. In this way, it is possible to gradually approach the semantic space of the target domain while maintaining the source domain generation capability.

[0050] As an example, the prior guidance form related to the target domain can be determined first based on the task type of the target domain. Then, the prior guidance signal is calculated on the second generated image based on the guidance network, and the prior guidance signal is then used to construct the prior guidance loss.

[0051] As examples, guidance networks can take, but are not limited to, CLIP-based guidance and VGG-based guidance. CLIP-based guidance can be achieved by optimizing the classification score or related scoring function of CLIP, and is used to provide prior guidance in tasks such as out-of-domain synthesis. VGG-based guidance can be achieved by a VGG-based classifier, which can be trained on datasets such as ImageNet and CelebA, and is used to predict the category or attribute of the generated results, thereby generating prior guidance signals.

[0052] Prior guidance loss can be instantiated as cross-entropy in classification guidance tasks:

[0053] in, Total number of categories; For prediction of the first The probability of a class serves as a priori guiding signal; This is a one-hot tag vector.

[0054] As an example, the guiding network and the backbone synthetic network can be trained on different datasets, such as the backbone synthetic network being pre-trained on FFHQ while the guiding network is trained on CelebA, to test the zero-shot adaptation capability under cross-dataset knowledge transfer conditions.

[0055] As an example, the Adam optimizer can be used with a learning rate and decay strategy set, and the hyperparameters related to the clustering loss weights can be chosen as follows. The batch size can be 32, and the weights of different loss terms can be determined by gradient normalization.

[0056] In step S204, the total loss is determined based on the prior guiding loss.

[0057] This step can directly use the prior guidance loss as the total loss, or it can introduce other loss terms, such as perceptual clustering loss and offset constraint loss, which are introduced later, to achieve joint optimization. This disclosure does not impose any restrictions on this.

[0058] In step S205, the parameters of the mapping network to be trained are adjusted according to the total loss to obtain the target domain mapping network.

[0059] Some generative adversarial network (GAN) adaptation methods in related technologies still require additional optimization processes during the inference or testing phases, such as repeatedly adjusting latent variables or network parameters for each generated sample. This approach not only incurs high computational costs and low inference efficiency but is also highly sensitive to optimization strategies and hyperparameter settings, limiting its deployment and application in practical engineering scenarios. This disclosure, however, addresses this issue by enabling... Figure 1 The inference stage shown generates an image that conforms to the characteristics of the target domain (i.e., the first generated image) through a forward propagation.

[0060] The following section will further describe step S204.

[0061] Optionally, there are multiple second latent vectors. Step S204 includes: determining a perceptual clustering loss based on the multiple second latent vectors, wherein the perceptual clustering loss is used to increase the difference between different second latent vectors; and determining the total loss based on the prior guidance loss and the perceptual clustering loss. Considering that relying solely on the prior guidance loss to constrain the generated results still carries potential risks, under zero-shot conditions, the guiding network can only evaluate the generated results at the semantic or attribute level and cannot explicitly distinguish the relationship between the generated images corresponding to different latent vectors. In extreme cases, multiple different latent vectors may be mapped to highly similar or even identical output results, i.e., mode collapse occurs. To address this problem, the exemplary embodiment of this disclosure further introduces a perceptual clustering loss into the overall training objective. By modeling the relationship between generated images in the perceptual embedding space, it emphasizes that different latent representations should correspond to different generated results. Compared to directly calculating the distance between samples in the pixel space, the perceptual embedding space can more stably characterize the high-level semantic structure, making the clustering constraints focus more on the semantic differences between generated results rather than local texture or noise variations. This is particularly important in zero-shot scenarios, because in the absence of real target domain samples, semantic-level distinctions are more generalizable than pixel-level differences. This embodiment encourages the synthetic network to maintain multiple semantic patterns during the generation process and generate diverse image outputs, reducing the risk of pattern collapse and thus enhancing the diversity and quality of generated images, structurally constraining the diversity of generated results.

[0062] Figure 3 This is a schematic diagram of the architecture of a training method for a target domain mapping network according to a specific embodiment of the present disclosure. In this specific embodiment, both the prior guidance loss described above and the perceptual clustering loss described herein are used. These two losses are functionally complementary: the former is responsible for constraining the consistency of the target domain in the generated results, while the latter is responsible for maintaining the diversity among the generated images. The joint optimization of these two losses enables the exemplary embodiment of the present disclosure to achieve high-quality and diverse target domain image generation without requiring optimization during testing. Specifically, Figure 3 In The noise vector representing the input mapping network (i.e., the second noise vector, which can follow a preset noise distribution) ), This represents the obtained latent vector (i.e., the second latent vector). The input projection module can obtain the embedding vector. ),in, The projection module is used to map latent vectors to an embedding space to which a perceptual clustering loss is applied, and the parameters of the projection module can be adjusted along with the parameters of the mapping network to be trained. The next column of squares can represent the style vector in StyleGAN, which is used to modulate the convolutional layers in the synthesis network, including controlling different visual attributes of the generated image, such as facial structure, hair color, skin color, etc.

[0063] Optionally, determining the perceptual clustering loss based on multiple second latent vectors in step S204 includes: performing random augmentation on each second latent vector to obtain an augmentation vector corresponding to the second latent vector; constructing multiple positive sample pairs based on each second latent vector and its corresponding augmentation vector, and determining the difference between each positive sample pair as a positive pair difference; constructing multiple negative sample pairs based on different second latent vectors and the augmentation vectors corresponding to each second latent vector, and determining the difference between each negative sample pair as a negative pair difference, wherein the negative samples in each negative sample pair originate from different second latent vectors; and determining the perceptual clustering loss based on the positive pair difference and the negative pair difference, wherein the perceptual clustering loss is positively correlated with the positive pair difference and negatively correlated with the negative pair difference. "Sample" refers to the generated image output after inputting the latent vector or augmentation vector into the synthesis network. By performing random augmentation on the latent vectors, the obtained augmentation vectors can be processed by the synthesis network to obtain different views of the same sample, such that different views of the same sample constitute positive sample pairs, and different samples constitute negative sample pairs, thereby constructing a "positive sample set" and a "negative sample pair set". The difference between each sample pair (the same applies to positive and negative sample pairs, hence they are uniformly referred to as sample pairs here) can be, for example, the vector distance between the vectors corresponding to the samples in the sample pair (which may be latent vectors or augmentation vectors), such as including but not limited to cosine distance, also known as the perceptual distance of the sample pair. Perceptual clustering loss calculates the perceptual distance between generated samples, compares positive and negative samples in the generated samples, making similar samples closer and dissimilar samples further apart. It can simultaneously impose the dual constraints of "positive pair alignment" and "negative pair separation" in the embedding space, so that the generative model can move closer to the target domain while still maintaining the ability to change driven by prior noise, thereby avoiding the degradation and compression of the latent space, reducing the risk of overly homogeneous generated results, and helping to ensure the diversity of generated samples.

[0064] As an example, random augmentation can employ a dimensional replacement random masking mechanism, defined as:

[0065] In the above formula, This represents the embedding vector obtained by mapping the second latent vector. Describes the augmentation vector of the second potential vector. For Bernoulli mask, For mask sampling probability, For independent noise Another embedding vector is generated. For each training batch... Can be obtained from a uniform distribution Sampling, in which Hyperparameters to enhance the strength range.

[0066] As an example, the perceptual clustering loss can be expressed as:

[0067] in, This refers to the batch size. Indicates embedding pairs The distance; the first term corresponds to the distance constraint for positive samples, and the second term corresponds to the distance separation constraint for negative samples.

[0068] In actual training, the number of negative sample pairs increases exponentially with the increase of batch size. At this point, many negative sample pairs have significant differences, contributing little to model training. Applying the same weight to all sample pairs not only increases invalid computation but also weakens the constraint effect on key sample pairs. To address this issue, optionally, in determining the perceptual clustering loss, each positive pair difference has a positive weight greater than 0, and each positive weight is positively correlated with its corresponding positive difference; similarly, each negative pair difference has a negative weight greater than 0, and each negative weight is negatively correlated with its corresponding negative difference. In this case, the perceptual clustering loss can be expressed as:

[0069] In the above formula, The weights for each sample pair are assigned based on their positive or negative attributes and varying weights. This provides a dynamic weight allocation mechanism that adjusts the weights according to the similarity between generated samples. This prioritizes similar, easily confused negative sample pairs, effectively differentiating the generated results from different latent vectors. It also prioritizes poorly aligned positive sample pairs, helping the model better eliminate inconsistencies caused by random augmentation and improving the stability of clustering constraints. This embodiment prioritizes samples that contribute significantly to model learning, enabling the generated images to more accurately reflect the features of the target domain, thus accelerating the training process and improving generation quality. It not only improves training efficiency but also significantly enhances the ability to suppress pattern collapse without adding additional model structure.

[0070] As an example, the weights are positive in positive sample pairs ( In this scenario, it is defined as:

[0071] in, For hyperparameters; Distance between positive sample pairs; This is a probability parameter (a control variable consistent with the enhancement strength) related to the Bernoulli masking mechanism, used to incorporate the enhancement strength into the weight design. It should be noted that... The calculation method depends on the specific reinforcement strength implementation method. Taking scalar reinforcement as an example, let's assume the reinforcement strength is... ,but: .

[0072] That is, the greater the enhancement, the lower the probability of retaining the original value, and the more obvious the masking effect.

[0073] Negative pair weights in negative sample pairs ( In this scenario, it consists of both distance and sorting terms:

[0074] in, For hyperparameters; Indicates anchor point sample The set of negative sample pairs centered on the sample; Used to highlight more similar negative sample pairs; Sample pairs that are ranked higher in similarity among negative pairs are used to further emphasize their separation during training, thereby reducing the tendency for different latent codes to map to similar results.

[0075] Under the dynamic weight allocation mechanism, the perceptual clustering loss can be written in a directly optimizable form:

[0076] in, This refers to the batch size. For the first Distance between positive sample pairs; Indicates the anchor point Summing the negative samples of the set.

[0077] It should be noted that the perceptual clustering loss formula is not obtained by directly substituting parameters into the previous related formulas. Instead, it constructs an optimizable objective by writing the weights as the gradient of a differentiable function, making it effective for clustering. and The gradient and weights are consistent with the positive and negative weights. For positive sample pairs, the gradient and weights are... After writing it in exponential function form, construct the objective term so that this objective term, during optimization, has a high degree of influence on... The function of maintaining and Maintain a consistent exponential function form. The numerator does not contain... This is because it's encapsulated within an exponential function. For negative sample pairs, the log-sum-exp is used to directly construct the separating term for optimizable negative sample pairs; there's no separate... It's because the distance between negative sample pairs has entered... Combining with log, in an aggregation manner rather than The linear approach is presented. In summary, this perceptual clustering loss formula constructs a smoothing objective to ensure its optimization behavior (equivalent weight allocation) is consistent with both positive and negative weights; therefore, the linear product form no longer appears in this formula. and .

[0078] Optionally, step S204 may further include: processing the second noise vector using a pre-trained mapping network to obtain a reference latent vector; determining a offset constraint loss based on the reference latent vector and the second latent vector; and determining the total loss based on the offset constraint loss and the prior guidance loss. By introducing an offset constraint loss, the difference between the optimized second latent vector and the reference latent vector generated by the pre-trained mapping network can be penalized to limit the variation of the latent vector, thereby maintaining the source domain generation capability and improving the stability of the training process while adapting to the target domain. As an example, Figure 4 This is a schematic diagram of an image generation architecture of a pre-trained generative adversarial network according to an exemplary embodiment of the present disclosure. Figure 4 In this context, h' can represent the reference latent vector. The offset constraint loss can be defined, for example, as:

[0079] In the above formula, It is the squared L2 norm.

[0080] Overall, the exemplary embodiments of this disclosure achieve high-quality, highly diverse zero-sample target domain image generation through lightweight mapping network optimization and diversity constraint mechanisms, thereby improving the applicability and deployment efficiency of generative adversarial networks in cross-domain adaptation tasks.

[0081] Figure 5 This is a block diagram of a target domain image generation apparatus based on zero-sample adaptation according to an exemplary embodiment of the present disclosure. (Refer to...) Figure 5 The target domain image generation device 300 based on zero-sample adaptation includes a mapping unit 501 and a synthesis unit 502.

[0082] The mapping unit 501 can use the target domain mapping network to process the first noise vector to obtain the first potential vector, wherein the first noise vector is sampled from the prior noise distribution.

[0083] Synthesis unit 502 can use a pre-trained synthesis network to process the first latent vector to obtain the first generated image.

[0084] The target domain mapping network is trained through the following steps: The second noise vector is processed using the mapping network to be trained to obtain a second latent vector, where the initial parameters of the mapping network to be trained are equal to the parameters of the pre-trained mapping network corresponding to the pre-trained synthetic network, and the second noise vector is sampled from the prior noise distribution; the second latent vector is processed using the pre-trained synthetic network to obtain a second generated image; a prior guidance loss is determined based on the target domain prior guidance information and the second generated image; the total loss is determined based on the prior guidance loss; and the parameters of the mapping network to be trained are adjusted based on the total loss to obtain the target domain mapping network.

[0085] Optionally, determining the prior guidance loss based on the target domain prior guidance information and the second generated image includes: processing the target domain prior guidance information and the second generated image using a guidance network to obtain a prior guidance signal, wherein the prior guidance signal is used to represent the degree of consistency between the target domain prior guidance information and the second generated image; and determining the prior guidance loss based on the prior guidance signal.

[0086] Optionally, there are multiple second latent vectors, wherein determining the total loss based on the prior guidance loss includes: determining a perceptual clustering loss based on the multiple second latent vectors, wherein the perceptual clustering loss is used to increase the difference between different second latent vectors; and determining the total loss based on the prior guidance loss and the perceptual clustering loss.

[0087] Optionally, the perceptual clustering loss is determined based on multiple second latent vectors, including: performing random augmentation on each second latent vector to obtain an augmentation vector corresponding to the second latent vector; constructing multiple positive sample pairs based on each second latent vector and its corresponding augmentation vector, and determining the difference between each positive sample pair as a positive pair difference; constructing multiple negative sample pairs based on different second latent vectors and the augmentation vectors corresponding to each second latent vector, and determining the difference between each negative sample pair as a negative pair difference, wherein the negative samples in each negative sample pair originate from different second latent vectors; and determining the perceptual clustering loss based on the positive pair difference and the negative pair difference, wherein the perceptual clustering loss is positively correlated with the positive pair difference and negatively correlated with the negative pair difference.

[0088] Optionally, in determining the perceptual clustering loss, each positive pair difference has a positive pair weight greater than 0, and each positive pair weight is positively correlated with the corresponding positive pair difference; each negative pair difference has a negative pair weight greater than 0, and each negative pair weight is negatively correlated with the corresponding negative pair difference.

[0089] Optionally, the total loss is determined based on the prior guidance loss, including: processing the second noise vector using a pre-trained mapping network to obtain a reference latent vector; determining the offset constraint loss based on the reference latent vector and the second latent vector; and determining the total loss based on the offset constraint loss and the prior guidance loss.

[0090] Regarding the apparatus in the above embodiments, the specific manner in which each unit performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated upon here.

[0091] Figure 6 A structural block diagram of an electronic device according to an exemplary embodiment of the present disclosure is shown.

[0092] Reference Figure 6 The electronic device 600 includes at least one memory 601 and at least one processor 602. The at least one memory 601 stores computer-executable instructions that, when executed by the at least one processor 602, cause the at least one processor to perform the target correspondence method as described in the exemplary embodiments above.

[0093] As an example, electronic device 600 may be a PC, tablet, personal digital assistant, smartphone, or other device capable of executing the aforementioned set of instructions. Here, electronic device 600 is not necessarily a single electronic device 600, but may be any collection of devices or circuits capable of executing the aforementioned instructions (or instruction sets) individually or in combination. Electronic device 600 may also be part of an integrated control system or system manager, or may be configured to interconnect with a portable electronic device 600 locally or remotely (e.g., via wireless transmission) through an interface.

[0094] In electronic device 600, processor 602 may include a central processing unit (CPU), a graphics processing unit (GPU), a programmable logic device, a dedicated processor system, a microcontroller, or a microprocessor. By way of example and not limitation, processor 602 may also include analog processors, digital processors, microprocessors, multi-core processors, processor arrays, network processors, etc.

[0095] The processor 602 can execute instructions or code stored in the memory 601, which can also store data. Instructions and data can also be sent and received via a network through a network interface device, which can employ any known transmission protocol.

[0096] The memory 601 may be integrated with the processor 602, for example, by placing RAM or flash memory within an integrated circuit microprocessor. Alternatively, the memory 601 may include a separate device, such as an external disk drive, a storage array, or other storage device that can be used by any database system. The memory 601 and the processor 602 may be operatively coupled, or may communicate with each other, for example, via I / O ports, network connections, etc., enabling the processor 602 to read files stored in the memory.

[0097] In addition, the electronic device 600 may also include a video display (such as a liquid crystal display) and a user interaction interface (such as a keyboard, mouse, touch input device, etc.). All components of the electronic device 600 can be interconnected via a bus and / or network.

[0098] According to exemplary embodiments of the present disclosure, a computer-readable storage medium storing instructions may also be provided, wherein the instructions, when executed by at least one processor, cause at least one processor to perform the target-correspondence method as described in the exemplary embodiments above. Examples of computer-readable storage media herein include: read-only memory (ROM), random access programmable read-only memory (PROM), electrically erasable programmable read-only memory (EEPROM), random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), flash memory, non-volatile memory, CD-ROM, CD-R, CD+R, CD-RW, CD+RW, DVD-ROM, DVD-R, DVD+R, DVD-RW, DVD+RW, DVD-RAM, BD-ROM, BD-R, BD-R LTH, BD-RE, Blu-ray or optical disc storage, hard disk drive (HDD), solid-state drive (SSD), card storage (such as multimedia cards, secure digital (SD) cards, or ultra-fast digital (XD) cards), magnetic tape, floppy disk, magneto-optical data storage device, optical data storage device, hard disk, solid-state drive, and any other device configured to store a computer program and any associated data, data files, and data structures in a non-transitory manner and to provide the computer program and any associated data, data files, and data structures to a processor or computer so that the processor or computer can execute the computer program. The computer program in the aforementioned computer-readable storage medium can run in an environment deployed in computer devices such as clients, hosts, agent devices, servers, etc. Furthermore, in one example, the computer program and any associated data, data files, and data structures are distributed across a networked computer system, such that the computer program and any associated data, data files, and data structures are stored, accessed, and executed in a distributed manner through one or more processors or computers.

[0099] According to exemplary embodiments of the present disclosure, a computer program product may also be provided, including computer instructions that, when executed by at least one processor, perform the target correspondence method as described in the exemplary embodiments above.

[0100] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This disclosure is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the appended claims.

[0101] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.

Claims

1. A target domain image generation method based on zero-shot adaptation, characterized in that, include: The first noise vector is processed using a target domain mapping network to obtain a first latent vector, wherein the first noise vector is sampled from a prior noise distribution; The first latent vector is processed using a pre-trained synthetic network to obtain the first generated image; The target domain mapping network is trained through the following steps: The second noise vector is processed by the mapping network to be trained to obtain the second potential vector, wherein the initial parameters of the mapping network to be trained are equal to the parameters of the pre-trained mapping network corresponding to the pre-trained synthesis network, and the second noise vector is sampled from the prior noise distribution; The second latent vector is processed using the pre-trained synthesis network to obtain the second generated image; Based on the prior guidance information of the target domain and the second generated image, the prior guidance loss is determined; The total loss is determined based on the aforementioned prior guiding loss; The parameters of the mapping network to be trained are adjusted based on the total loss to obtain the target domain mapping network.

2. The target domain image generation method as described in claim 1, characterized in that, The step of determining the prior guidance loss based on the target domain prior guidance information and the second generated image includes: The target domain prior guidance information and the second generated image are processed using a guidance network to obtain a prior guidance signal, wherein the prior guidance signal is used to represent the degree of consistency between the target domain prior guidance information and the second generated image; The prior guidance loss is determined based on the prior guidance signal.

3. The target domain image generation method as described in claim 1, characterized in that, The number of the second potential vectors is multiple, wherein determining the total loss based on the prior guiding loss includes: Based on a plurality of second latent vectors, a perceptual clustering loss is determined, wherein the perceptual clustering loss is used to increase the difference between different second latent vectors; The total loss is determined based on the prior guidance loss and the perceptual clustering loss.

4. The target domain image generation method as described in claim 3, characterized in that, The step of determining the perceptual clustering loss based on multiple second latent vectors includes: For each of the second potential vectors, a random augmentation process is performed to obtain an augmented vector corresponding to the second potential vector; Based on each second potential vector and its corresponding enhancement vector, multiple positive sample pairs are constructed, and the difference between each positive sample pair is determined as the positive pair difference. Based on different second latent vectors and the enhancement vectors corresponding to each second latent vector, multiple negative sample pairs are constructed, and the difference between each negative sample pair is determined as the negative pair difference, wherein the negative samples in each negative sample pair originate from different second latent vectors; The perceptual clustering loss is determined based on the positive and negative differences, wherein the perceptual clustering loss is positively correlated with the positive differences and negatively correlated with the negative differences.

5. The target domain image generation method as described in claim 4, characterized in that, In determining the perceptual clustering loss, each positive difference has a positive weight greater than 0, and each positive weight is positively correlated with the corresponding positive difference; each negative difference has a negative weight greater than 0, and each negative weight is negatively correlated with the corresponding negative difference.

6. The target domain image generation method as described in claim 1 or 2, characterized in that, The determination of the total loss based on the prior guided loss includes: The second noise vector is processed using the pre-trained mapping network to obtain a reference potential vector; The offset constraint loss is determined based on the reference potential vector and the second potential vector; The total loss is determined based on the offset constraint loss and the prior guidance loss.

7. A target domain image generation device based on zero-shot adaptation, characterized in that, include: The mapping unit is configured to process a first noise vector using a target domain mapping network to obtain a first latent vector, wherein the first noise vector is sampled from a prior noise distribution; A synthesis unit is configured to process the first latent vector using a pre-trained synthesis network to obtain a first generated image; The target domain mapping network is trained through the following steps: The second noise vector is processed by the mapping network to be trained to obtain the second potential vector, wherein the initial parameters of the mapping network to be trained are equal to the parameters of the pre-trained mapping network corresponding to the pre-trained synthesis network, and the second noise vector is sampled from the prior noise distribution; The second latent vector is processed using the pre-trained synthesis network to obtain the second generated image; Based on the prior guidance information of the target domain and the second generated image, the prior guidance loss is determined; The total loss is determined based on the aforementioned prior guiding loss; The parameters of the mapping network to be trained are adjusted based on the total loss to obtain the target domain mapping network.

8. An electronic device, characterized in that, include: At least one processor; At least one memory that stores computer-executable instructions. The computer-executable instructions, when executed by the at least one processor, cause the at least one processor to perform the target domain image generation method based on zero-sample adaptation as described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, When the instructions in the computer-readable storage medium are executed by at least one processor, the at least one processor causes the at least one processor to perform the target domain image generation method based on zero-sample adaptation as described in any one of claims 1 to 6.

10. A computer program product comprising computer instructions, characterized in that, When the computer instructions are executed by at least one processor, the at least one processor causes the processor to perform the target domain image generation method based on zero-sample adaptation as described in any one of claims 1 to 6.

Images