Semantic guidance-based portrait generation method and device, and storage medium

Abstract

The invention relates to the technical field of artificial intelligence, and provides a semantic guidance-based portrait generation method, which comprises the following steps of: dynamically analyzing a description text input by a user through a text semantic understanding model, establishing a feature association node matched with text semantics obtained by analysis in a potential space of an image generation network, and generating a semantic guidance vector; decomposing the semantic guide vector into a generation control parameter group with interpretability; and performing hierarchical cascading fusion on the generation control parameter group and the random noise vector, dynamically adjusting the feature response intensity of each layer of the generation network through a hierarchical cascading fusion result, and generating a portrait image consistent with text semantics based on the adjusted feature response intensity. According to the technical scheme, the dynamic portrait can be accurately and controllably generated.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a semantically guided human image generation method, device and storage medium. Background Technology

[0002] Currently, generative artificial intelligence has made significant progress in the field of image synthesis. Among them, text-guided portrait generation technology has attracted much attention due to its application value in scenarios such as virtual character design and personalized content creation. Existing technologies mainly establish feature associations between text and images through joint embedding models and utilize generative adversarial networks to achieve semantic-to-visual mapping. For example, based on static semantic encoding, text descriptions are mapped into fixed-dimensional latent space vectors and then directly drive the generative network; another approach is to adopt an end-to-end training strategy, dynamically adjusting the generation process through reinforcement learning. The aforementioned existing technologies suffer from core drawbacks such as a lack of dynamic adaptability in the association mechanism between text semantics and image latent space features, reliance on empirical trial and error in the debugging process, and susceptibility to style distortion or loss of detail. Summary of the Invention

[0003] This application provides a semantically guided portrait generation method, device, and storage medium that can accurately and controllably generate dynamic portraits.

[0004] On the one hand, this application provides a semantically guided image generation method, the method comprising: The text semantic understanding model dynamically parses the descriptive text input by the user, establishes feature association nodes that match the semantics of the parsed text in the latent space of the image generation network, and generates semantic guidance vectors. The semantic guidance vector is decomposed into a set of interpretable generation control parameters; The generation control parameter set is cascaded and fused with a random noise vector in a hierarchical manner. The feature response intensity of each layer of the generation network is dynamically adjusted based on the result of the hierarchical cascade fusion, and a portrait image consistent with the text semantics is generated based on the adjusted feature response intensity.

[0005] On the other hand, this application provides a semantically guided portrait generation device, the device comprising: The association module is used to dynamically parse the descriptive text input by the user through the text semantic understanding model, establish feature association nodes that match the semantics of the parsed text in the latent space of the image generation network, and generate semantic guidance vectors. A generation module is used to decompose the semantic guidance vector into a set of interpretable generation control parameters; The fusion module is used to perform hierarchical cascade fusion of the generated control parameter group and random noise vector, dynamically adjust the feature response intensity of each layer of the generation network based on the result of the hierarchical cascade fusion, and generate a portrait image consistent with the text semantics based on the adjusted feature response intensity.

[0006] Thirdly, this application provides an electronic device, the device including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the above-described semantically guided portrait generation method.

[0007] Fourthly, this application provides a storage medium storing a computer program, which, when executed by a processor, implements the steps of the above-described semantically guided portrait generation method.

[0008] As can be seen from the technical solution provided in this application, on the one hand, by constructing feature association nodes matching the text description in the latent space of the image generation network, it is possible not only to adaptively parse the implicit global style, main structure, and local detail semantics in the text, but also to effectively improve the semantic consistency between the text description and the generated image. On the other hand, by decomposing the semantic guidance vector into an interpretable parameter set, the style transfer, content layout, and detail optimization in the generation process form an independent and controllable adjustment dimension, providing a clear operational interface for locating and correcting generation deviations. Thirdly, by adopting a hierarchical cascade fusion strategy combined with the dynamic adjustment of feature response intensity, the semantic guidance vector and random noise vector are synergistically optimized during the adversarial generation process, which can effectively suppress semantic conflicts and feature distortions in the generated image and ensure the visual rationality of the generated result. In summary, the technical solution of this application can accurately and controllably generate dynamic portraits. Attached Figure Description

[0009] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0010] Figure 1 This is a flowchart of the semantically guided portrait generation method provided in the embodiments of this application; Figure 2 This is a schematic diagram of the structure of the semantically guided human image generation device provided in the embodiments of this application; Figure 3 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0011] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0012] In this specification, adjectives such as "first" and "second" are used only to distinguish one element or action from another, without necessarily requiring or implying any actual such relationship or order. Where circumstances permit, reference to an element or component or step (etc.) should not be construed as being limited to only one of the elements, components, or steps, but may be one or more of the elements, components, or steps, etc.

[0013] For ease of description, the dimensions of the various parts shown in the accompanying drawings are not drawn to actual scale.

[0014] Currently, generative artificial intelligence has made significant progress in the field of image synthesis. Among them, text-guided portrait generation technology has attracted much attention due to its application value in scenarios such as virtual character design and personalized content creation. Existing technologies mainly establish feature associations between text and images through joint embedding models and utilize generative adversarial networks to achieve semantic-to-visual mapping. Current mainstream methods can be divided into two categories: one is based on static semantic encoding, which maps text descriptions into fixed-dimensional latent space vectors and then directly drives the generative network. However, this type of method is prone to feature confusion when parsing complex semantics, resulting in insufficient matching between the generated image and text details. The other adopts an end-to-end training strategy, which dynamically adjusts the generation process through reinforcement learning. However, the parameter adjustment lacks interpretability and it is difficult to locate the source of semantic bias. The above-mentioned existing technologies have the following core defects: (1) The association mechanism between text semantics and image latent space features lacks dynamic adaptability. When the text description contains multi-level semantics (such as global style, main structure, and local details), it is difficult to establish fine-grained feature mapping; (2) The generation control parameters are mostly black box vectors, which cannot be decomposed into interpretable independent control dimensions, resulting in the debugging process relying on empirical trial and error; (3) The fixed weight injection strategy is adopted in the fusion generation stage, which makes it difficult to dynamically adjust the feature response intensity according to semantic consistency, and is prone to style distortion or lack of details.

[0015] To address the aforementioned problems in existing technologies, this application proposes a semantically guided image generation method, the flowchart of which is attached. Figure 1 As shown, the main steps include S101 to S103, which are detailed below: Step S101: Dynamically parse the descriptive text input by the user through the text semantic understanding model, establish feature association nodes that match the semantics of the parsed text in the latent space of the image generation network, and generate semantic guidance vectors.

[0016] In this embodiment, the text semantic understanding model is a deep learning model pre-trained on a large scale of text-image pairs. Its function is to map natural language descriptions into structured semantic representations. The descriptive text is natural language text input by the user, which typically includes global style features of the human image (e.g., "oil painting style"), main structural features (e.g., "standing posture"), and local detail features (e.g., "blue eyes"). The image generation network is a deep learning model based on a generative adversarial network architecture. Its function is to convert semantic vectors into images. The latent space is a low-dimensional vector space in the image generation network used to represent image features. Text semantics refers to the visual concepts and attributes expressed by the descriptive text. Feature association nodes are feature dimensions in the latent space that establish a mapping relationship with specific text semantics. Considering that traditional static encoding cannot handle complex semantic scenarios such as polysemous words and overlapping modifiers, for example, when the user-input descriptive text contains "soft retro style + sharp eye details," static encoding may lead to feature confusion. Dynamic parsing, on the other hand, can separate semantic nodes representing global style from local details. Therefore, to avoid the loss of semantic information during generation, this application can establish feature association nodes that match the text semantics in the latent space of the image generation network, so as to provide structured semantic guidance for subsequent parameter mapping. Specifically, as an embodiment of this application, the user-input descriptive text is dynamically parsed through a text semantic understanding model, and feature association nodes that match the text semantics are established in the latent space of the image generation network. The generation of semantic guidance vectors can be achieved through steps S1011 to S1013, as detailed below: Step S1011: Construct a text-image joint embedding space based on the pre-trained joint embedding model, and map the user-input descriptive text into a first semantic vector based on the text-image joint embedding space.

[0017] Using only a text autoencoder not only risks losing details during encoding (e.g., the hair texture details of "long blonde hair" might be lost in a pure text autoencoder), but also fails to establish a direct connection with the latent space of the image generation network, leading to a mismatch between the semantic guidance vector and the feature dimensions of the generation network. To unify the semantic representation of text and images and avoid feature mapping bias caused by modal differences, a text-image joint embedding space can be constructed based on a pre-trained joint embedding model, mapping the user-input descriptive text to a first semantic vector. Furthermore, considering that the joint embedding of a Contrastive Language-Image Pretraining (CLIP) model can preserve visual association features, in this embodiment, the pre-trained joint embedding model can be a CLIP model.

[0018] In the above embodiments, the pre-trained joint embedding model uses a large-scale dataset containing hundreds of millions of text-image pairs during the pre-training stage, including publicly available multimodal datasets (e.g., COCO, Visual Genome) and proprietary portrait datasets, covering portrait samples with different styles, poses, and detail dimensions. Specifically, the text-image joint embedding space is constructed based on the pre-trained joint embedding model, and the user-input descriptive text is mapped to a first semantic vector through the following steps S10111 to S10115: Step S10111: Use the CLIP model's text encoder to segment the user-input descriptive text and generate a word embedding sequence.

[0019] Step S10111 specifically involves using the CLIP model's byte-pair encoder to segment the descriptive text into sub-word units, then querying the embedding table to map each sub-word into a 512-dimensional word vector, forming a word embedding sequence. .

[0020] Step S10112: Perform contextual semantic modeling on the word embedding sequence using a self-attention mechanism, and output the text feature vector. ,in, This represents the text encoding dimension of the CLIP model.

[0021] The self-attention mechanism described in the above embodiments is a core component of the CLIP text encoder. It achieves context-aware semantic representation by calculating the attention weights between each position and all positions in the word embedding sequence. This process employs the standard Transformer encoder architecture and is existing technology. Specifically, for the word embedding sequence... By querying the matrix Key matrix Sum matrix Calculate attention output:

[0022] Finally, the sequence output is processed by global average pooling to obtain the text feature vector. t .

[0023] Step S10113: Use the CLIP model's image encoder to extract features from the training image dataset, generate image feature vectors, and normalize them.

[0024] In this embodiment, the training image dataset contains millions of portrait images, covering various ethnicities, ages, and lighting conditions, including frontal, side, and half-body portraits from multiple angles. Each image is accompanied by descriptive text annotations. Image feature vectors ,in, The image encoding dimension of the CLIP model can be expressed using the formula... Normalize the image feature vector v to a normalized image feature vector. .

[0025] Step S10114: Combine the text feature vector t with the normalized image feature vector Mapping to the same semantic space, cross-modal associations are established through similarity calculation, as follows:

[0026] in, yes The transpose of , This represents the semantic matching score of the text-image pair. This is used to verify whether the outputs of the CLIP text encoder and image encoder are in the same semantic space. When the score is higher than a preset threshold (e.g., 0.8), they are considered to be in the same semantic space; when it is lower than the threshold, semantic deviation is considered to exist.

[0027] Step S10115: Perform dimension alignment and normalization on the above text feature vector t to obtain the first semantic vector. :

[0028] in, , d For the latent spatial dimensions of image generation networks, Let LayerNorm be the learnable projection matrix, and denote the layer normalization operation. As mentioned earlier, This is used to verify whether the outputs of the CLIP text encoder and the image encoder are in the same semantic space. If the calculated score is too low, the projection matrix can be readjusted using the gradient descent method. The parameters are adjusted until the semantic matching score meets the requirements, then the adjusted parameters are used. Calculate the first semantic vector Through the steps S10111 to S10114 above, the text-image joint embedding space constructed based on the CLIP model corresponds to the text feature vector t and the normalized image feature vector t. The shared semantic space in which it resides.

[0029] Step S1012: Extract a second semantic vector that matches the dimension of the first semantic vector from the latent space of the image generation network.

[0030] Specifically, extracting a second semantic vector that matches the dimension of the first semantic vector from the latent space of the image generation network can be achieved through the following steps S10121 to S10124: Step S10121: Sample the fundamental noise vector from the latent space of the image generation network. The basic noise vector z is input into the shallow feature extraction module of the generator network, which outputs multi-level latent features. .

[0031] In the above embodiments, the basic noise vector Let be a random vector sampled from a Gaussian distribution, whose dimension is _____. d Consistent with the latent space dimension of the generative network, it provides the basis for the stochasticity of the generation process. The aforementioned shallow feature extraction module can be composed of 4-8 residual blocks concatenated, with the base noise vector... The feature transformation is performed sequentially through each residual block, and each residual block is transformed through skip connections, outputting the latent features of that layer. .

[0032] Step S10122: Perform cross-layer fusion on the multi-level latent features as follows to obtain aggregated features. :

[0033] in, The dynamic weight coefficients for each layer's features can be calculated using an attention mechanism. It should be noted that in the above embodiments, the "layer" in cross-layer fusion refers to the residual block level in the image generation network, aggregating features... In the calculation formula, L This represents the total number of layers (i.e., the number of residual blocks). By calculating the features of each layer Attention scores are dynamically determined in relation to the global context vector.

[0034] Step S10123: Aggregate features Project onto the first semantic vector in the following manner In the same dimensional space, the second semantic vector is obtained. :

[0035] in, For a learnable projection matrix, This is the aggregated feature dimension.

[0036] Step S10124: Calculate the first semantic vector With the second semantic vector If the cosine similarity is lower than a preset threshold, Then readjust the projection matrix. The parameters, up to the first semantic vector. With the second semantic vector The cosine similarity is not lower than a preset threshold. At that time, determine the second semantic vector. This is the second semantic vector obtained at the end of this step; if it is not lower than the preset threshold. Then the second semantic vector at this time is directly determined. This is the second semantic vector obtained at the end of this step.

[0037] Step S1013: Dynamically calculate the cross-modal association strength between the first semantic vector and the second semantic vector through a trainable attention adaptation layer to generate a semantic guidance vector.

[0038] In the above embodiments, the trainable attention adaptation layer is a learnable neural network module, whose weights are dynamically optimized through backpropagation of the generation task to adapt to the text-image modal differences. Specifically, the cross-modal association strength between the first semantic vector and the second semantic vector is dynamically calculated through the trainable attention adaptation layer, and the semantic guidance vector is generated through the following steps S10131 to S10134: Step S10131: Generate attention query vector - key-value vector.

[0039] Specifically, step S10131 can be implemented by converting the first semantic vector... Input linear transformation layer generates query vector Q , the second semantic vector Input linear transformation layer generates key vectors K Sum value vector V This forms an attention query-key value vector group.

[0040] Step S10132: Calculate the attention weights of the query vector and the key-value vector.

[0041] Specifically, step S10132 can calculate the attention weight matrix A using the scaled dot product attention formula: ,in, is the dimension of the key vector.

[0042] Step S10133: Perform weighted fusion of the second semantic vector according to the attention weight.

[0043] Specifically, step S10133 may involve adjusting the attention weight matrix. A AND value vector V Multiplying them together yields the weighted fused feature vector: .

[0044] Step S10134: Connect the fusion result with the residual of the first semantic vector to generate the final semantic guidance vector.

[0045] Specifically, firstly, an initial semantic guidance vector is generated. Then, during the generation process, the mapping relationship between text terms and feature nodes is monitored in real time. If multiple terms (e.g., "gold" and "long hair") are mapped to the same feature node through attention weights, the attention weight corresponding to that node is multiplied by the enhancement coefficient. β >1, the final output is the enhanced semantic guidance vector. g .

[0046] In another embodiment of this application, the generation of semantic guidance vectors by dynamically calculating the cross-modal association strength between the first semantic vector and the second semantic vector through a trainable attention adaptation layer can also be achieved through the following steps S'10131 to S'10133: Step S'10131: Generate an independent attention query vector for each term in the descriptive text.

[0047] Specifically, step S'10131 can be implemented by: mapping each term to a vector using a word embedding layer, and then generating independent query vectors through a learnable linear transformation. ,in This serves as an index for the term.

[0048] Step S'10132: Use the latent spatial feature nodes as attention key vectors to perform real-time matching with the attention query vectors.

[0049] Specifically, step S'10132 may involve setting the feature node set in the latent space. Transform into a key vector through a linear transformation. Calculate each query vector With all key vectors Similarity score; if the score exceeds a preset threshold Then determine the entry With feature nodes match.

[0050] Step S'10133: Based on the real-time matching results, the matching relationship between terms and feature nodes is aggregated into the dimensions of the first semantic vector and the second semantic vector. When the attention query vectors of multiple terms are detected to point to the attention key vector of the same feature node, the association strength coefficient of the feature node is automatically enhanced. The association strength coefficient is quantified by weighted summation to determine the cross-modal association strength between the first semantic vector and the second semantic vector.

[0051] The trainable attention adaptation layer monitors the matching results. When the query vectors of multiple terms point to the key vector of the same feature node, it multiplies the coefficient of that node in the attention weight by a dynamic enhancement factor. This factor is positively correlated with the term frequency and the number of node activations. It should be noted that the enhancement magnitude of the aforementioned association strength coefficient is dynamically related to factors such as the frequency of term occurrence in the text description, the effective activation count of the feature node in the historical generation process, and the priority setting of specific semantics in the current generation stage. The quantification of the association strength coefficient is specifically achieved as follows: for each feature node... Its correlation strength coefficient It is obtained by weighted summation of the attention weights of the term query vectors matched to this node:

[0052] in, To match nodes The set of term indexes, For the entry Attention weights are then assigned. Then, the association strength coefficients of all feature nodes are organized into an association strength vector. A Mapped to the first semantic vector through matrix operations. With the second semantic vector Cross-modal correlation strength score between: ,in, W For a learnable projection matrix, λ The regularization coefficient is used to directly quantify the degree of correlation between two semantic vectors in cross-modal space.

[0053] The embodiment of step S101 above establishes a dynamic cross-modal semantic association mechanism, constructs feature association nodes that match the descriptive text in the latent space of the image generation network, and adaptively parses the implicit global style, main structure and local detail semantics in the text, thereby effectively improving the semantic consistency between the text description and the generated image.

[0054] Step S102: Decompose the semantic guidance vector into a set of interpretable generation control parameters.

[0055] In this embodiment, style refers to the overall visual representation of the portrait, content refers to the main structure and posture of the portrait, and detail refers to local features. On the one hand, when generating a portrait, style (e.g., oil painting / realism), content (e.g., pose / gender), and detail (e.g., hair / texture) need to be controlled independently. If they are merged into a single vector, gradient conflicts will occur during back-optimization. On the other hand, if detail correction parameters are omitted and only style and content parameters are relied upon, the local detail matching rate of the generated image will decrease significantly. If the style weight distribution is removed, style transfer will rely on post-processing filters, which will also lead to a significant reduction in generation efficiency. Therefore, in this embodiment, the interpretable generation control parameter set includes a style weight vector that controls the global style, a content constraint vector that constrains the main content, and detail correction parameters that adjust the generation of details, etc. The engineering value of the interpretability of the above generation control parameter set lies in the fact that the separated parameters allow for manual intervention, such as manually enhancing detail parameters; in contrast, black-box vectors cannot support this operation. Furthermore, after decomposing the semantic guidance vector into an interpretable set of generation control parameters, the process includes: inputting the style weight vector into the inverse mapping network to generate a corresponding visual style heatmap; decoding the content constraint vector into readable attribute description text; comparing the semantic differences between the attribute description text and the user-input description text, and comparing the visual consistency between the style heatmap and the user's expected style; if both the semantic difference and visual deviation are below a preset threshold, the verification is considered successful; otherwise, a parameter adjustment mechanism is triggered, and the user-input description text is re-parsed until the verification is successful. This visual style weight distribution helps developers locate style transfer deviations (e.g., "retro style" not correctly mapped to the texture generation layer), while the text difference comparison optimizes the semantic understanding model, improving the semantic alignment between the content constraint vector and the original description text.

[0056] It should be noted that in the above embodiments, the back-mapping network is a pre-trained convolutional neural network that decodes the style weight vector into a heatmap, where highlighted areas correspond to spatial locations where style features are significant. The user's expected style is constructed based on the user's historical selection data or explicit feedback (e.g., ratings). The style heatmap and the user's expected style are compared for visual consistency, specifically by calculating the structural similarity index (SSIM) between the style heatmap and the user's expected style template. The deviation between the style heatmap and the user's expected style is the difference between the SSIM score and the ideal value of 1.0, and the deviation data is stored in scalar form. Simultaneously, the detail correction parameters are validated: their input to the detail decoder generates a magnified local detail image, which is then compared with the user's input detail description at the pixel level for consistency.

[0057] Furthermore, the aforementioned parameter adjustment mechanism includes readjusting the attention weight distribution and style transfer matrix parameters of the semantic understanding model, re-parses the descriptive text based on the adjusted model, and generates new semantic guidance vectors, rather than generating the control parameter set from scratch. Specifically, the implementation of the parameter adjustment mechanism includes the following steps: a) Bias analysis stage: Semantic difference analysis: calculating the keyword recall and semantic similarity between the attribute description text and the original description text; Visual bias analysis: calculating the structural similarity index (SSIM) between the style heatmap and the user's expected style template; b) Parameter adjustment decision: when the semantic difference... or visual deviation When this occurs, an adjustment mechanism is triggered, in which... and c) Targeted parameter adjustment: For semantic understanding models: Enhance the attention weights of incorrectly mapped keywords. The formula is adjusted as follows:

[0058] in, For learning rate, Keywords The correct mapping number in the historical generation, for the style transfer matrix: update the matrix parameters in reverse based on the style heatmap and the gradient direction of the expected style; d) Iterative optimization: re-execute step S101 with the adjusted model, generate new semantic guidance vectors, and re-perform interpretability verification until the verification conditions are met or the maximum number of iterations is reached.

[0059] As an embodiment of this application, decomposing the semantic guidance vector into a set of interpretable generation control parameters can be achieved through steps S1021 to S1024, as detailed below: Step S1021: Decouple the semantic guidance vector by separating the style feature subspace, content feature subspace and detail feature subspace.

[0060] Specifically, the semantic guidance vector is decoupled using an orthogonally constrained variational autoencoder: three independent encoders are used to extract style, content, and detail features respectively, and an orthogonal loss function ensures subspace independence, where each subspace is a low-dimensional manifold of the latent space. If style (e.g., color distribution), content (e.g., human pose), and detail (e.g., texture) are coupled in the original vector, direct use will lead to generation conflicts, such as adjusting style parameters unintentionally changing human proportions, while undecoupled parameter sets will cause gradient conflicts during backpropagation, significantly reducing training convergence speed. Therefore, in this embodiment, the semantic guidance vector can be feature decoupled to separate the style feature subspace, content feature subspace, and detail feature subspace.

[0061] Furthermore, when subspace features overlap, for example, "retro style" might affect hair details, orthogonal constraints can force the separation of feature dimensions; if only ordinary decoupling is used, residual correlations will still cause mutual interference in parameter tuning. Therefore, the feature decoupling of semantic guidance vectors in the above embodiments can adopt a dynamic orthogonalization strategy to feature the semantic guidance vectors. That is, when overlapping features are detected in different subspaces, orthogonal constraints are simultaneously applied to the projection matrices of the three subspaces: style feature subspace, content feature subspace, and detail feature subspace. The constraint strength is proportional to the feature overlap.

[0062] Step S1022: Within the style feature subspace, based on the style semantic components in the semantic guidance vector, the semantics of the descriptive text are mapped to a style weight vector through the style transfer matrix.

[0063] In this embodiment, the style semantic component is the projection of the semantic guidance vector onto the style feature subspace, carrying information about global visual features such as artistic style and color matching in the descriptive text. The style transfer matrix aligns the style semantic component with the style keywords in the descriptive text to generate corresponding style weight vectors.

[0064] Step S1023: Within the content feature subspace, construct a content constraint vector based on the content semantic components in the semantic guidance vector using a conditional generative adversarial network.

[0065] A Conditional Generative Adversarial Network (GAN) consists of a generator and a discriminator. The generator produces content constraint vectors conditioned on a semantically guided vector, while the discriminator ensures the consistency between the vectors and the text content. The content semantic component is the projection of the semantically guided vector onto the content feature subspace, encoding key structural information in the text such as human posture, gender, and age. The GAN, conditioned on the content semantic component, generates content constraint vectors consistent with the text description through adversarial training.

[0066] Step S1024: Within the detail feature subspace, detail correction parameters are generated using a multi-scale residual network based on the detail semantic components in the semantic guidance vector.

[0067] The multi-scale residual network is pre-trained through multi-task learning, and its weights are dynamically adjusted according to the complexity of details: the network is trained using a high-resolution portrait dataset, enabling it to output detail correction parameters at different scales. The detail semantic component is the projection of the semantic guidance vector onto the detail feature subspace, extracting local feature descriptions from the descriptive text (e.g., "blue eyes," "curly hair," etc.). The multi-scale residual network generates corresponding detail correction parameters based on the importance weights of different scales within the detail semantic component.

[0068] The weights of the aforementioned multi-scale residual network can be dynamically adjusted based on the detail complexity of the user-input descriptive text. Specifically, generating detail correction parameters through the multi-scale residual network can involve: converting detail keywords in the descriptive text into local attention masks; dynamically allocating residual computation resources at different scales based on the spatial distribution of the areas covered by the local attention masks; and automatically activating the high-precision residual module and increasing the feature resolution to a preset multiple of the original data when generating eye or hand details.

[0069] As can be seen from step S102 of the above embodiment, by decomposing the semantic guidance vector into interpretable parameter groups such as style weights that control the global style, content constraint vectors that constrain the main content, and detail correction parameters that adjust the generation of details, the style transfer, content layout, and detail optimization of the generation process can form an independent and controllable adjustment dimension, providing a clear operation interface for locating and correcting generation deviations.

[0070] Step S103: The generated control parameter group and random noise vector are fused in a hierarchical manner. The feature response intensity of each layer of the generation network is dynamically adjusted based on the result of the hierarchical fusion, and a portrait image consistent with the text semantics is generated based on the adjusted feature response intensity.

[0071] Since style parameters can be fused in shallow networks to control the overall tone and texture, and detail parameters can be injected into deep networks to avoid premature solidification of details in low-resolution layers, fixed-weight fusion can lead to style and detail conflicts in complex scenes, such as "cartoon style + realistic skin". Therefore, in order to combine the advantages of each layer of the generative network and optimize the fusion ratio in real time to improve the semantic consistency score, in this embodiment, the generation control parameter group and random noise vector can be fused in a hierarchical cascade. By dynamically adjusting the feature response intensity of each layer of the generative network, a portrait image consistent with the text semantics can be generated.

[0072] As one embodiment of this application, the generation of control parameter sets and random noise vectors are fused in a hierarchical cascade manner. The feature response intensity of each layer of the generation network is dynamically adjusted based on the result of the hierarchical cascade fusion. The generation of a portrait image consistent with the semantics of the text is based on the adjusted feature response intensity. This can be achieved through steps S1031 to S1033, which are described in detail below: Step S1031: In the low-resolution layer of the generative network, the style weight vector and the random noise vector are linearly superimposed to generate the basic feature map.

[0073] Step S1032: In the high-resolution layer of the generative network, the content constraint vector and detail correction parameters are injected into the basic feature map through a dynamic gating mechanism to generate a detail-enhanced feature map.

[0074] When the dynamic gating mechanism injects the content constraint vector and detail correction parameters into the base feature map, it first calculates the cosine similarity between the content constraint vector and the currently generated feature. Then, when the similarity is lower than the dynamic threshold, it triggers the incremental injection of detail correction parameters (specifically, incremental injection refers to injecting the detail correction parameters at a learnable step size Δ). d The features are successively superimposed onto the current feature map (the injected objects are the feature activation values ​​of the high-resolution layer). When the discriminant network detects style deviation, the gradient direction of the style weight vector is reversed. The initial value of the dynamic threshold is set according to the complexity of the user-input descriptive text and decays exponentially with the network layer depth during generation. Specifically, it can be set based on the number of keywords and semantic density in the user-input descriptive text. Based on the current layer number of the generated network n According to the formula For dynamic thresholds Attenuation is performed, where, λ These are learnable parameters; when user historical preference data is detected, a compensation term is applied to the threshold. ,in, For compensation coefficient, p The preference matching score can be obtained as follows: extract high-frequency style tags and detailed features from the user's historical preference data; perform cross-batch cosine similarity matching between the currently generated parameters and the historical preference data; when the preference matching score exceeds a preset standard, the compensation coefficient will be adjusted. Increase to the preset multiple of the original value.

[0075] Step S1033: Based on the discriminative network, the semantic consistency of the detail enhancement feature map is scored, and the fusion ratio of the style weight vector and the detail correction parameter is adjusted in real time based on the score result to generate a portrait image that is consistent with the semantics of the text.

[0076] Step S103 of the above embodiment adopts a hierarchical cascade fusion strategy and combines it with dynamic adjustment of feature response intensity to achieve synergistic optimization of semantic guidance vector and random noise vector in the adversarial generation process, effectively suppressing semantic conflicts and feature distortions in the generated image and ensuring the visual rationality of the generation result.

[0077] From the above appendix Figure 1As illustrated by the semantically guided portrait generation method in the example, on the one hand, by constructing feature association nodes matching the text description in the latent space of the image generation network, it can not only adaptively parse the implicit global style, main structure, and local detail semantics in the text, but also effectively improve the semantic consistency between the text description and the generated image. On the other hand, by decomposing the semantic guidance vector into an interpretable parameter set, the style transfer, content layout, and detail optimization in the generation process become independent and controllable adjustment dimensions, providing a clear operational interface for locating and correcting generation deviations. Thirdly, by adopting a hierarchical cascade fusion strategy combined with dynamic adjustment of feature response intensity, the semantic guidance vector and random noise vector are synergistically optimized during adversarial generation, which can effectively suppress semantic conflicts and feature distortions in the generated image and ensure the visual rationality of the generated result. In summary, the technical solution of this application can accurately and controllably generate dynamic portraits.

[0078] Please see the appendix Figure 2 This application provides a semantically guided portrait generation device, which may include an association module 201, a generation module 202, and a fusion module 203, as detailed below: The association module 201 is used to dynamically parse the descriptive text input by the user through the text semantic understanding model, establish feature association nodes that match the semantics of the parsed text in the latent space of the image generation network, and generate semantic guidance vectors. Generation module 202 is used to decompose the semantic guidance vector into a set of interpretable generation control parameters; The fusion module 203 is used to perform hierarchical cascade fusion of the generated control parameter group and the random noise vector, dynamically adjust the feature response intensity of each layer of the generation network through the result of hierarchical cascade fusion, and generate a portrait image consistent with the text semantics based on the adjusted feature response intensity.

[0079] From the above appendix Figure 2 As illustrated by the semantically guided portrait generation device, on the one hand, by constructing feature association nodes matching the text description in the latent space of the image generation network, it can not only adaptively parse the implicit global style, main structure, and local detail semantics in the text, but also effectively improve the semantic consistency between the text description and the generated image. On the other hand, by decomposing the semantic guidance vector into an interpretable parameter set, the style transfer, content layout, and detail optimization in the generation process become independent and controllable adjustment dimensions, providing a clear operational interface for locating and correcting generation deviations. Thirdly, by adopting a hierarchical cascade fusion strategy combined with dynamic adjustment of feature response intensity, the semantic guidance vector and random noise vector are synergistically optimized during adversarial generation, which can effectively suppress semantic conflicts and feature distortions in the generated image and ensure the visual rationality of the generated result. In summary, the technical solution of this application can accurately and controllably generate dynamic portraits.

[0080] Figure 3 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. For example... Figure 3 As shown, the electronic device 3 in this embodiment mainly includes: a processor 30, a memory 31, and a computer program 32 stored in the memory 31 and executable on the processor 30, such as a program for a semantically guided portrait generation method. When the processor 30 executes the computer program 32, it implements the steps described in the semantically guided portrait generation method embodiment, for example... Figure 1 The steps S101 to S103 are shown. Alternatively, when the processor 30 executes the computer program 32, it implements the functions of each module / unit in the above-described device embodiments, for example... Figure 2 The functions of the associated module 201, the generation module 202, and the fusion module 203 are shown.

[0081] For example, the computer program 32 based on the semantically guided portrait generation method mainly includes: dynamically parsing the descriptive text input by the user through a text semantic understanding model; establishing feature association nodes in the latent space of the image generation network that match the semantics of the parsed text; generating a semantically guided vector; decomposing the semantically guided vector into a set of interpretable generation control parameters; performing hierarchical cascaded fusion of the generation control parameter set with a random noise vector; dynamically adjusting the feature response intensity of each layer of the generation network based on the result of the hierarchical cascaded fusion; and generating a portrait image consistent with the semantics of the text based on the adjusted feature response intensity. The computer program 32 can be divided into one or more modules / units, one or more of which are stored in the memory 31 and executed by the processor 30 to complete this application. One or more modules / units can be a series of computer program instruction segments capable of performing specific functions, which describe the execution process of the computer program 32 in the electronic device 3. For example, computer program 32 can be divided into the functions of association module 201, generation module 202, and fusion module 203 (modules in the virtual device). The specific functions of each module are as follows: Association module 201 is used to dynamically parse the descriptive text input by the user through a text semantic understanding model, establish feature association nodes that match the semantics of the parsed text in the latent space of the image generation network, and generate semantic guidance vectors; Generation module 202 is used to decompose the semantic guidance vectors into a set of interpretable generation control parameters; Fusion module 203 is used to perform hierarchical cascade fusion of the generation control parameter set with random noise vectors, dynamically adjust the feature response intensity of each layer of the generation network through the result of hierarchical cascade fusion, and generate a portrait image consistent with the text semantics based on the adjusted feature response intensity.

[0082] Electronic device 3 may include, but is not limited to, processor 30 and memory 31. Those skilled in the art will understand that... Figure 3 This is merely an example of electronic device 3 and does not constitute a limitation on electronic device 3. It may include more or fewer components than shown, or combine certain components, or different components. For example, electronic devices may also include input / output devices, network access devices, buses, etc.

[0083] The processor 30 may be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor.

[0084] The memory 31 can be an internal storage unit of the electronic device 3, such as a hard disk or RAM. The memory 31 can also be an external storage device of the electronic device 3, such as a plug-in hard disk, Smart Media Card (SMC), Secure Digital (SD) card, or Flash Card. Furthermore, the memory 31 can include both internal and external storage units of the electronic device 3. The memory 31 is used to store computer programs and other programs and data required by the electronic device. The memory 31 can also be used to temporarily store data that has been output or will be output.

[0085] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed. That is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the units and modules in the above-described device can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0086] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0087] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0088] In the embodiments provided in this application, it should be understood that the disclosed apparatus / device and method can be implemented in other ways. For example, the apparatus / device embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another device, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0089] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0090] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0091] If integrated modules / units are implemented as software functional units and sold or used as independent products, they can be stored in a storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments can also be implemented by a computer program instructing related hardware. The computer program for the semantically guided portrait generation method can be stored in a storage medium. When the computer program is executed by a processor, it can implement the steps of the above-described method embodiments, namely, dynamically parsing the descriptive text input by the user through a text semantic understanding model, establishing feature association nodes that match the semantics of the parsed text in the latent space of the image generation network, and generating a semantic guidance vector; decomposing the semantic guidance vector into a set of interpretable generation control parameters; performing hierarchical cascade fusion of the generation control parameter set and a random noise vector, dynamically adjusting the feature response intensity of each layer of the generation network based on the result of the hierarchical cascade fusion, and generating a portrait image consistent with the text semantics based on the adjusted feature response intensity. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or some intermediate forms. Storage media can include: any entity or device capable of carrying computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the contents of storage media can be appropriately added to or removed according to the requirements of legislation and patent practice in a jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, storage media may not include electrical carrier signals and telecommunication signals.

[0092] The above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit it. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. These modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application. The specific embodiments described above further illustrate the purpose, technical solutions, and beneficial effects of this application. It should be understood that the above descriptions are merely specific embodiments of this application and are not intended to limit the protection scope of this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this invention.

Claims

1. A semantically guided portrait generation method, characterized in that, The method includes: The text semantic understanding model dynamically parses the descriptive text input by the user, establishes feature association nodes that match the semantics of the parsed text in the latent space of the image generation network, and generates semantic guidance vectors. The semantic guidance vector is decomposed into a set of interpretable generation control parameters; The generation control parameter set is cascaded and fused with a random noise vector in a hierarchical manner. The feature response intensity of each layer of the generation network is dynamically adjusted based on the result of the hierarchical cascade fusion, and a portrait image consistent with the text semantics is generated based on the adjusted feature response intensity.

2. The semantically guided portrait generation method according to claim 1, characterized in that, The process of dynamically parsing the user-input descriptive text using a text semantic understanding model, establishing feature association nodes that match the text semantics in the latent space of the image generation network, and generating semantic guidance vectors includes: A text-image joint embedding space is constructed based on a pre-trained joint embedding model, and the descriptive text is mapped to a first semantic vector based on the text-image joint embedding space; Extract a second semantic vector that matches the dimension of the first semantic vector from the latent space of the image generation network; The cross-modal association strength between the first semantic vector and the second semantic vector is dynamically calculated through a trainable attention adaptation layer, and a semantic guidance vector is generated based on the result of calculating the cross-modal association strength.

3. The semantically guided portrait generation method according to claim 2, characterized in that, The dynamic calculation of the cross-modal association strength between the first semantic vector and the second semantic vector through a trainable attention adaptation layer includes: Generate an independent attention query vector for each term in the descriptive text; The latent spatial feature nodes are used as attention key vectors and matched with the attention query vector in real time. Based on the results of the real-time matching, the matching relationship between terms and feature nodes is aggregated into the dimensions of the first semantic vector and the second semantic vector. When the attention query vectors of multiple terms are detected to point to the attention key vector of the same feature node, the association strength coefficient of the feature node is automatically enhanced. The association strength coefficient is quantified by weighted summation to determine the cross-modal association strength between the first semantic vector and the second semantic vector.

4. The semantically guided portrait generation method according to claim 1, characterized in that, The step of decomposing the semantic guidance vector into a set of interpretable generative control parameters includes: The semantic guidance vector is decoupled to separate the style feature subspace, content feature subspace, and detail feature subspace; Within the style feature subspace, based on the style semantic components in the semantic guidance vector, the textual semantics of the descriptive text are mapped to a style weight vector through a style transfer matrix; Within the content feature subspace, a content constraint vector is constructed using a conditional generative adversarial network based on the content semantic components in the semantic guidance vector. Within the detailed feature subspace, detailed correction parameters are generated through a multi-scale residual network based on the detailed semantic components in the semantic guidance vector. Determine the generation control parameter set that includes the style weight vector, content constraint vector, and detail correction parameters.

5. The semantically guided portrait generation method according to claim 1, characterized in that, The step of hierarchically cascading and fusing the generated control parameter set with a random noise vector, dynamically adjusting the feature response intensity of each layer of the generation network based on the result of the hierarchical cascading and fusing, and generating a portrait image consistent with the text semantics based on the adjusted feature response intensity includes: In the low-resolution layer of the image generation network, the style weight vector and the random noise vector are linearly superimposed to generate a basic feature map. In the high-resolution layer of the image generation network, a dynamic gating mechanism is used to inject the content constraint vector and detail correction parameters into the basic feature map to generate a detail-enhanced feature map. Based on the semantic consistency score of the detail enhancement feature map by the discriminative network, the fusion ratio of style weight vector and detail correction parameter is adjusted in real time based on the score result to generate a portrait image consistent with the semantics of the text.

6. The semantically guided portrait generation method according to claim 5, characterized in that, The dynamic operation of the dynamic gating mechanism includes: Calculate the cosine similarity between the content constraint vector and the currently generated feature map. When the similarity is lower than the dynamic threshold, trigger the incremental injection of detail correction parameters. The initial value of the dynamic threshold is set according to the complexity of the user-input description text and decays exponentially with the network layer depth during the generation process. When a style deviation is detected, the gradient direction of the style weight vector is reversed, and the style deviation is determined based on the degree of deviation between the semantic consistency score and the preset standard.

7. The semantically guided portrait generation method according to claim 1, characterized in that, After decomposing the semantic guidance vector into a set of interpretable generation control parameters, the method further includes: Input the style weight vector into the inverse mapping network to generate the corresponding visual style heatmap; Decode the content constraint vector into readable attribute description text; The semantic differences between the attribute description text and the description text input by the user are compared, and the visual consistency between the style heatmap and the user's expected style is compared. If both semantic difference and visual deviation are below the preset threshold, the verification is deemed successful; otherwise, the parameter adjustment mechanism is triggered, and the user-input description text is re-parsed until the verification is successful.

8. A semantically guided human portrait generation device, characterized in that, The device includes: The association module is used to dynamically parse the descriptive text input by the user through the text semantic understanding model, establish feature association nodes that match the semantics of the parsed text in the latent space of the image generation network, and generate semantic guidance vectors. A generation module is used to decompose the semantic guidance vector into a set of interpretable generation control parameters; The fusion module is used to perform hierarchical cascade fusion of the generated control parameter group and random noise vector, dynamically adjust the feature response intensity of each layer of the generation network based on the result of the hierarchical cascade fusion, and generate a portrait image consistent with the text semantics based on the adjusted feature response intensity.

9. An electronic device, the device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the method as described in any one of claims 1 to 7.

10. A storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method as described in any one of claims 1 to 7.

Images