Multi-modal face attribute editing method based on text and image

Abstract

The invention relates to the technical field of face attribute editing processing, in particular to a text and image-based multi-modal face attribute editing method, which comprises the following steps of: constructing a face image data set; inputting the face image into an encoder E to generate a direct feature, and converting a given potential code into a style code by using a mapper M; and inputting the direct features and the style codes into a translator T to learn specific feature marks of the operation, and converting the translated features into an image by using a generator G. According to an existing method, expression of image attributes is only limited to an abstract and obscure monovision mode, dominant attribute description is lacked, and the problems that when face attribute editing is carried out, labeling and training cost is very high exist are solved.

Description

Technical Field

[0001] The present invention relates to the technical field of face attribute editing, and in particular to a multi-modal face attribute editing method based on text and image. Background Art

[0002] With the rapid development of computer vision and natural language processing technologies, traditional face attribute editing methods are undergoing profound changes;

[0003] Existing methods modify facial attributes based on image-based attribute editing methods; for example, in StarGANv2, the mixed style is learned based on the label-indexed style code of the target domain, continuing the assumption of StarGANv1 that the domain is a collection of images sharing the same label; however, this model often involves some unnecessary operations in the attribute editing process of glasses removal, such as changing the face identity and gender;

[0004] Facial-GAN is an improvement on StarGANv2, mainly using guided segmentation label masks to guide the generation of target images; it retains the facial features of the input while making the style of the generated image similar to the target image. However, this only retains the original blurred facial region features.

[0005] L2M-GAN uses a style converter to decompose the style encoding in the latent space, but its lack of other constraints results in not only the target domain changing, but also other parts of the converted face changing, such as only wanting to change the glasses, while the editing process usually changes the hair color and background, etc.

[0006] Although the above attribute editing methods have achieved certain results, their expression of image attributes is limited to the abstract and obscure single visual modality and lacks explicit attribute description. In order to improve the visual model's ability to understand facial semantics, they tend to use multiple labels with different attributes for joint training, so as to inject as much information as possible into the model. However, the cost of this idea is also very prominent, that is, using more labeled data will bring very high annotation and training costs. Summary of the invention

[0007] In view of the shortcomings of existing methods, the present invention firstly accurately projects visual attribute labels into a feature prompt synthesizer of text description; then uses the LLM model to understand the deep meaning when encoding text information and integrates it into the image editing process; enhances the model's understanding of the relationship between the two, and improves the performance and flexibility of attribute editing tasks.

[0008] The technical solution adopted by the present invention is: a multimodal face attribute editing method based on text and image comprises the following steps:

[0009] Step 1: Build a face image dataset;

[0010] Step 2: Input the face image into encoder E to generate direct features, and use mapper M to convert the given latent code into style code; input the direct features and style code into translator T to learn specific feature tags, and use generator G to convert the translated features into images;

[0011] As a preferred embodiment of the present invention, step 2 specifically includes:

[0012] Step 21, the source image x is encoded into direct features through the encoder E, the formula is:

[0013] e0=e=E(x) (1)

[0014] Step 22: Operate the source features into target attributes j1,...,j l For multiple markers i1,...,i l ; where l is the number of manipulation tokens, the features are input into the translator T one by one, and we get:

[0015] e k =T ik (e k-1 ,s ik,jk ) (2)

[0016] Among them, s ik,jk It is the potential guiding style;

[0017] M ik,jk (z) (3)

[0018] Step 23: Image generation is completed by the generator G, and we get:

[0019]

[0020] Among them, G() is the generator.

[0021] As a preferred embodiment of the present invention, the style code and the text code are fused using a multimodal feature fusion device, and the fused features are input into a translator T to learn and operate specific feature tags;

[0022] The text encoding generation process includes:

[0023] The attribute labels are input into the feature prompt synthesizer to convert the attribute labels into natural language descriptions; the natural language descriptions are input into the LLM model to generate text encodings.

[0024] As a preferred embodiment of the present invention, the construction of the feature prompt synthesizer includes:

[0025] First, using the attribute labels in the dataset, the attribute labels are passed into a list containing several attribute labels. Each label is represented by an array, and the elements in the array correspond to visual features.

[0026] Secondly, define an auxiliary function for processing a single attribute tag array. The auxiliary function determines the construction of the description based on the value of each attribute. When constructing the description, the attribute tag information is concatenated into a string one by one to form a complete sentence.

[0027] Finally, all input attribute labels are traversed, corresponding descriptions are generated for each attribute label, and all generated descriptions are summarized into a list to output natural text sentences.

[0028] As a preferred embodiment of the present invention, the LLM model generates text encoding, and the text encoding and style encoding are subjected to feature fusion through a multimodal feature fuser, including:

[0029] The LLM model is used to encode the semantic information of natural text sentences. Given a natural language sequence x=x1,…,x |x| As input, the decoder predicts the output sequence token by token; at the i-th step, the decoder predicts the i-th token y in a linearized form i , and the decoder status It is expressed as:

[0030]

[0031] The conditional probability of the entire output sequence p(y|x) is calculated by i |y <i ,x) are gradually combined into:

[0032]

[0033] Among them, y <i =y1…y i-1 , p(y i |y <i ,x) is the probability on the target vocabulary V;

[0034] For a natural language text sequence x=x1,…,x with a length of K |K| ,Pick As the sentence-level encoding of the text h t and the image encoding h with the output of the visual encoder v For splicing, the formula is:

[0035] h = concat(h v ,h t ) (7)

[0036] The image feature h v and text feature h t The common input is fed into the fully connected layer for feature dimension transformation.

[0037] As a preferred embodiment of the present invention, the generator G converts the translated features into images, and performs VLM description generation through the VLM model.

[0038] As a preferred embodiment of the present invention, the process of VLM description generation includes:

[0039] Divide the image I into grid image patches, each of which is encoded as a visual token; flatten the grid image patches into a sequence and add position encoding; obtain the encoded image representation x from the image I v ; The instruction text is recorded as x t ; Visual tokens are merged into the input sequence x:

[0040] x=[x v ,x t ] (8)

[0041] The sequence x is input into the text decoder of VLM, and the image description is generated by predicting the token; at each generation step i, the decoder predicts the next token y based on the predicted token and state i , and update y i Internal State

[0042]

[0043] The overall probability of sequence x generating the entire output sequence y is the probability of generating each token by multiplication, expressed as:

[0044]

[0045] Among them, y <i =y1,…y i-1 represents all tokens predicted before the i-th token; p(y i |y <i ,x) is the probability of token and predicted i-th token;

[0046] The goal of VLM is to maximize the given sequence X O In the case of the target output image description sequence X T The probability of output image description sentence X T .

[0047] As a preferred embodiment of the present invention, a multimodal facial attribute editing system based on text and images includes: a memory for storing instructions executable by a processor; a processor for executing the instructions to implement a multimodal facial attribute editing method based on text and images.

[0048] As a preferred embodiment of the present invention, a computer readable medium stores a computer program code, and when the computer program code is executed by a processor, a multimodal facial attribute editing method based on text and image is implemented.

[0049] Beneficial effects of the present invention:

[0050] 1. The present invention constructs a feature prompt synthesizer to project visual attributes into a natural language label sequence, and uses the LLM model to inject the natural language visual attribute features into the visual face attribute editing module based on the GAN network, which can inject more information into the model in one training, so that the model can focus on more details in one training and obtain higher quality attribute editing results;

[0051] 2. Build a feature prompt synthesizer from visual attribute labels to text descriptions, which can generate fluent and natural language descriptions based on the provided attribute labels and project visual attributes into natural language label sequences;

[0052] 3. An innovative attribute editing result evaluation method based on text description is designed to prove the accuracy and decoupling of the model of the present invention for specific attribute operations when performing attribute editing tasks from a text perspective. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Figure 1 It is a flow chart of the multimodal face attribute editing method based on text and image of the present invention;

[0054] Figure 2 It is a schematic diagram of the characteristic prompt synthesizer flow of the present invention;

[0055] Figure 3 is a multimodal feature fusion flow chart of the present invention;

[0056] Figure 4 is a schematic diagram of an image encoder of the present invention;

[0057] Figure 5 It is the qualitative result of facial attribute editing of the specific attribute (glasses) of the present invention;

[0058] Figure 6 It is the qualitative result of facial attribute editing of the specific attribute (gender) of the present invention;

[0059] Figure 7It is the qualitative result of facial attribute editing of the specific attribute (smile) of the present invention;

[0060] Figure 8 It is a schematic diagram of the ROUGE calculation process of the present invention;

[0061] Fig. 9 is the similarity ROUGE-1 of the text of the present invention;

[0062] Fig.10 It is the similarity ROUGE-2 of the text of the present invention. DETAILED DESCRIPTION

[0063] The present invention is further described below in conjunction with the accompanying drawings and embodiments. This figure is a simplified schematic diagram, which only illustrates the basic structure of the present invention in a schematic manner, and therefore it only shows the components related to the present invention.

[0064] like Figure 1 As shown, a multimodal face attribute editing method based on text and image includes the following steps:

[0065] Step 1: Build a face image dataset;

[0066] Step 2: Input the face image into encoder E to generate direct features, and use mapper M to convert the given latent code into style code; input the direct features and style code into translator T to learn the operational feature labels, and use generator G to convert the translated features into images;

[0067] Image feature extraction aims to generate, extract and manipulate tag-related attributes in images; for a given latent code z~N(0,1) and a label i of attribute j, the style code is generated by the mapper M to generate a style code s i,j =M i,j (z); Then, the generated or extracted style is used to guide the operation, using x to represent the source image at the beginning of training.

[0068] The input of image feature extraction is the original image, which is converted into direct features by encoder E. The direct features are given by e = E(x); then in order to operate on this feature, a translator T is introduced; given feature e and label i, the label-dependent style encoding s i,j , translator T through T i (e,s i,j ) Learn to operate specific feature tags; in order to obtain the translated image, the generator G is introduced to convert the translated features is converted to an image; the converted image is Representation; and introduce a discriminator D to determine whether the image given the label and attribute is real; for those modules that use labels and attributes as input, choose to use these labels and attributes to label the selection of specific layers instead of directly input;

[0069] During formal training, first, the source image x is encoded into direct features through the encoder E, the formula is:

[0070] e0=e=E(x) (1)

[0071] Secondly, the source features are manipulated into target attributes j1,...,j l For multiple markers i1,...,i l ; where l is the number of manipulation tokens, the features are input into the translator T one by one, and we get:

[0072] e k =T ik (e k-1 ,s ik,jk ) (2)

[0073] Among them, s ik,jk It is the potential guiding style;

[0074] M ik,jk (z) (3)

[0075] Finally, let Image generation is completed by the generator G, and we get:

[0076]

[0077] Construct text feature extraction and input the text description of the original image; organize the information of several attribute tags into natural text sentences and input them into the large language model (LLM) for text encoding;

[0078] Expanded attribute labels to six, including: gender (male and female), whether smiling, whether wearing glasses, nose size (big or small), lip size (big or small), and whether there are bangs;

[0079] like Figure 2 As shown, organizing the information of several attribute labels into natural text sentences is to convert the labels into natural language descriptions through a feature prompt synthesizer;

[0080] The main purpose of the feature prompt synthesizer is to generate natural language descriptions based on the provided attribute labels; its workflow can be divided into the following key steps:

[0081] First, use the attribute labels in the dataset to pass the labels into a list containing multiple attribute labels. Each label is represented by an array. The elements in these arrays correspond to specific visual features, such as gender, whether smiling, whether wearing glasses, etc.

[0082] Secondly, define a helper function specifically for processing a single attribute tag array. The helper function will decide how to construct the description based on the value of each attribute (usually 1 or -1); for example, if the gender tag is 1, the prompt description text will contain "male"; if the gender tag is -1, the prompt description text will contain "female"; the same logic applies to attributes such as smile, glasses, nose size, lip size, and bangs; when constructing the description, the feature prompt synthesizer will concatenate these attribute information into a string one by one to form a complete sentence; each attribute has a clear description format to ensure that the generated description is semantically coherent;

[0083] Finally, all input attribute labels are traversed, corresponding descriptions are generated for each label, and all generated descriptions are summarized into a list as the final output, that is, the output natural text sentence.

[0084] The feature cue synthesizer achieves effective conversion from visual attributes to text descriptions, providing clear and accurate text input for subsequent image processing tasks; by directly converting visual attributes into natural language descriptions, the generated text is easy to understand; the structured description ensures consistency and coherence, which is conducive to the learning and reasoning of the model in multimodal tasks; the text description synthesized by the feature cue synthesizer can not only enhance the model's understanding of the image content, but also promote the semantic association between image and text, so that the data is more fully utilized.

[0085] The construction process of the feature prompt synthesizer can be summarized into two steps. The first step is to traverse all samples in the input batch and generate a single prompt for each sample. The second step is to organize the corresponding prompt for a single sample based on the given 6 attribute labels. Use a to simply represent the attribute to be processed. Since there are 6 attributes involved in the training, a is assigned a value of 6. Focus on the six typical labels of the facial area, including male and female, whether smiling, whether wearing glasses, big / small nose, big / small lips, whether there is bangs, and organize them into natural language through the above method, such as Figure 2 For a face image, the corresponding Prompt is This image describes a male who is smiling, wearing glasses, having big nose and big lips, having no bangs.

[0086] like Figure 3 As shown in the figure, a multimodal feature fusion (MFF) is constructed. In traditional face attribute editing, in order to improve the model's understanding of face semantics, it is often inclined to use multiple labels of different attributes for joint training, so as to inject as much information as possible into the model; however, the cost of this idea is also very prominent, that is, using more labeled data will bring very large annotation and training costs.

[0087] Different from previous works that stick to the scope of pure images, this paper shifts its focus to the text approach; using the large language model LLM (Large Language Model), the LLM model relies on the scaling law and performs pre-training (Pretraining) and supervised fine-tuning (Supervised Finetuning) on ​​massive text data based on the Transformer Decoder structure; this paper organizes the label information of multiple attributes into natural text sentences and inputs them into the LLM model for text encoding; natural language text contains six different attribute editing label information, and uses the LLM model to encode its semantic information. Given a natural language sequence x=x1,…,x |x| As input, the decoder predicts the output sequence token by token; at the i-th generation step, the decoder predicts the i-th token y in a linearized form i , and the decoder status It is expressed as:

[0088]

[0089] The conditional probability of the entire output sequence p(y|x) is calculated by i |y <i ,x) are gradually combined into:

[0090]

[0091] Among them, y <i =y1…y i-1 , and p(y i |y <i ,x) is the probability on the target vocabulary V.

[0092] For a natural language text sequence x=x1,…,x with a length of K |K| ,Pick As the sentence-level encoding of the text h t , this encoding contains all 6 attribute label information; it is then combined with the image encoding h of the output of the visual encoder vFor splicing, the formula is:

[0093] h = concat(h v, h t ) (7)

[0094] In this process, the image feature h v and text feature h t They are input into the fully connected layer together, and the feature dimension is transformed into h so that they can be input into the model for corresponding operations.

[0095] By simultaneously inputting images and corresponding text descriptions during the training process, the text features are combined with the visual features, greatly improving the effect and flexibility of face attribute editing. In other words, applying LLM to feature injection for multimodal face attribute editing not only makes efficient use of data, but also brings higher accuracy to face editing tasks, which can cope with more complex editing needs and application scenarios.

[0096] The visual language model (VLM) combines image and text information to achieve in-depth understanding and generation by learning the features of the two modalities. The VLM model receives the image after attribute editing and generates a corresponding natural language description. By calculating the text similarity between the two paragraphs before and after the attribute editing, the effect of the attribute editing can be judged.

[0097] VLM image-text conversion: The visual language model (VLM) is a multimodal model that combines image and text information, aiming to achieve a deeper level of understanding and generation by jointly learning the features of these two modalities; the VLM model receives an image that has been processed by attribute editing and generates a corresponding natural language description; the present invention designs an attribute editing result evaluation method based on the VLM text description, and proves from the text perspective the accuracy and decoupling of the model's manipulation of specific attributes when completing the attribute editing task;

[0098] like Figure 4 As shown in Figure 2, the input image I is divided into a grid of image blocks, each of which is then encoded as a visual token; the grid is then flattened into a sequence and positional encoding is added; the encoded image representation x is obtained from the image I. v ; Input command text is recorded as x t ; These tokens are then combined to form an input sequence x with the following structure:

[0099] x=[x v ,x t ] (8)

[0100] The sequence x is input into the text decoder of VLM, which generates the output image description by predicting one token at a time; at each generation step i, the decoder predicts the next token y based on all previously predicted tokens and states i , and updates its internal state

[0101]

[0102] The overall probability of generating the entire output sequence y given an input x is calculated by multiplying the probability of generating each token, expressed as:

[0103]

[0104] Among them, y <i =y1,…y i-1 represents all tokens predicted before the i-th token; p(y i |y <i ,x) is the probability of predicting the i-th token given the previous tokens and the input.

[0105] The goal of the model is to maximize the O In the case of the target output image description sequence X T probability.

[0106] Then, for the output image statement X T .

[0107] Experimental process:

[0108] Database: In order to prove that the proposed method can perform attribute editing tasks well, experiments were conducted on the public dataset CelebA; CelebA is a large-scale face attribute dataset, which contains 202,599 face images of 10,177 famous people; this dataset covers images with various posture changes and background clutter, and is suitable for a wide range of face-related computer vision training tasks; the CelebA dataset was selected for model training to make full use of its rich data resources; on this basis, experiments and analysis were carried out to further verify the effectiveness and performance of the proposed model.

[0109] Two Tesla V100 GPUs (32GB video memory), CUDA 11.4 and CUDNN 8.0.5 were used. All codes were developed and implemented based on the deep learning framework PyTorch 1.4.0. During the experiment, the powerful GPU computing power and efficient deep learning algorithm implementation provided by PyTorch were fully utilized to ensure the accuracy and stability of the experiment. At the same time, the code was optimized and debugged in detail to ensure its efficient operation and excellent performance on the GPU. The model of the present invention was optimized using the Adam optimizer, with β1 = 0.0 and β2 = 0.99; the initial learning rate was 1×10 -4 , using the TTUR strategy to train the model of the present invention

[0110] Qualitative results:

[0111] Figure 5 , 6 7 show the qualitative results of the method of the present invention and the typical method on the three attributes of glasses, gender and smile; the first column is the input, the middle is the experimental results of Decent, HiSD, FacialGAN, L2M and StarGANv2, and the last column is the experimental result of the method of the present invention; although Decent can remove glasses, the overall sharpening is serious and the image distortion is unnatural; although HiSD has a more realistic arrangement style, it is not good enough in terms of decoupling; Figure 3 In the results of the first and third rows, the glasses are not completely removed, the ears and hands are deformed in the second row, and wrinkles appear on the cheeks in the fourth row. Although StarGAN and FacialGAN change the attribute-related style, they also change other attribute-irrelevant information, such as hair color, background, beard, gender, etc. In addition, they also change facial features such as skin color and facial texture. L2M can produce visually attractive images, but because it only uses latent space decomposition, it fails in some details, such as lack of accurate hair recovery, lack of texture details, and even destroys the overall style of the image. The accuracy of attribute editing is not well controlled. The model of the present invention can edit related attributes without changing other attributes, and has more natural and high-quality results.

[0112] Quantitative results:

[0113] FID reflects the quality of the generated image by calculating the distance between the generated image domain and the real image domain; the lower the FID, the closer the two distributions are, which means the higher the quality of the generated image; the following table shows the FID scores of editing the three attributes of "glasses, gender, smile" under different methods on the same dataset; the results are shown in Table 1, and it can be seen that the model of the present invention has better results, which shows that the results of the present invention have better image quality.

[0114] Table 1 FID of different methods

[0115]

[0116] Attribute editing precision:

[0117] In order to determine whether a specific attribute is correctly displayed on the image generated after the operation, the attribute operation accuracy can be used as a measure; in order to obtain this value, the present invention uses ResNet18 to train a binary classifier for the "glasses, gender, smile" attributes; Table 2 shows that although the model of the present invention did not use the attribute classifier during training, it can still achieve good results in operation accuracy, which shows that the method of the present invention can accurately manipulate specific attributes without changing other attributes, and also verifies the effectiveness of the model.

[0118] Table 2 Comparison of accuracy of different methods

[0119]

[0120]

[0121] ROUGE is the similarity index for evaluating two texts:

[0122] ROUGE (Recall-Oriented Understudy for Gisting Evaluation) is a commonly used indicator for evaluating text similarity. Specifically, Rouge-1 judges the similarity of two texts based on a single word. It considers whether there are words in the text that are the same as the keywords in the reference text, and if so, it is considered good. Rouge-2 not only considers whether there are keywords in the text, but also considers the order and combination of keywords. Rouge-L is a similarity evaluation indicator based on the longest common subsequence. It finds the longest common subsequence between the reference text and the abstract and calculates its similarity score. Since the order and combination of keywords are considered, the results are as follows: Figure 8 , 9 , as shown in 10.

[0123] Since the edited attributes are deleted from the text description when calculating ROUGE, the ROUGE results can indirectly prove the accuracy and decoupling of the model in attribute editing ability, that is, after the attribute editing of a specific attribute, other unedited attributes are not changed; in short, ROUGE can effectively evaluate the consistency of the generated text in content and structure. The experimental results of the model of the present invention are shown in Tables 3 and 4:

[0124] Table 3. ROUGE similarity of texts

[0125]

[0126]

[0127] Table 4. ROUGE similarity of texts from different models

[0128]

[0129] User Research:

[0130] To qualitatively evaluate the diversity of generated images, the Learned Patch Similarity of Perceptual Images (LPIPS) is widely used; however, in practical use, LPIPS encourages the model to manipulate images as much as possible, which is undesirable; furthermore, authenticity is a subjective measurement and cannot be fully represented by FID; therefore, a user study was chosen to compare diversity and authenticity; 50 volunteers were asked to evaluate the generation results from two aspects: image quality (Q) and disentanglement (D); since attribute editing is very challenging, a choice of “none of the methods achieves good results” is also provided to avoid that no method can manipulate the attributes well, but volunteers must choose the best one; images generated by different methods are displayed in random order for fair comparison; Table 6 shows the average user study results of all 50 volunteers

[0131] Table 5 User research results

[0132]

[0133] In order to inject more information into the model during training to obtain better attribute editing results, the present invention proposes a multimodal face attribute editing method assisted by large language model feature injection; firstly, a feature prompt synthesizer named Feature Prompt Synthesizer is constructed, which can generate natural and fluent language descriptions using the existing attribute labels in the data set; then, the attribute features processed by natural language are injected into the visual face attribute editing module based on the generative adversarial network (GAN) by utilizing the powerful language understanding ability of the large language model (LLM), and are incorporated into the training process together with the image features so that the model can grasp the image information more comprehensively; finally, a novel attribute editing result evaluation method based on text description is designed, and the accuracy and decoupling of the model of the present invention on specific attribute operations when completing the attribute editing task are verified from the text perspective; extensive subjective and objective experiments have proved that the model of the present invention has good performance in attribute editing.

[0134] Based on the above ideal embodiments of the present invention, the relevant staff can make various changes and modifications without departing from the technical concept of the present invention through the above description. The technical scope of the present invention is not limited to the contents of the specification, and its technical scope must be determined according to the scope of the claims.

Claims

1. A multimodal face attribute editing method based on text and image, characterized in that: The following steps are involved: Step 1: Build a face image dataset; Step 2: Input the face image into encoder E to generate direct features, and use mapper M to convert the given latent code into style code; input the direct features and style code into translator T to learn the operational feature labels, and use generator G to convert the translated features into images.

2. The multimodal face attribute editing method based on text and image according to claim 1, characterized in that: Step 2: Specifically include: Step 21: Encode the source image x into direct features. The formula is: e0=e=E(x) (1) Step 22: Operate the source features into target attributes j1,...,j l Used to label i1,...,i l ; where l is the number of manipulation tokens, the features are input into the translator T one by one, and we get: e k =T ik (e k-1 ,s ik,jk ) (2) Among them, s ik,jk It is the potential guiding style; M ik,jk (of) (3) Step 23: Image generation is completed by the generator G, and we get: Among them, G() is the generator and E() is the encoder.

3. The multimodal face attribute editing method based on text and image according to claim 1, characterized in that: The style encoding and text encoding are fused using a multimodal feature fuser, and the fused features are input into the translator T to learn the operational feature labels; The text encoding generation process includes: The attribute labels are input into the feature prompt synthesizer to convert the attribute labels into natural language descriptions; the natural language descriptions are input into the LLM model to generate text encodings.

4. The multimodal face attribute editing method based on text and image according to claim 3 is characterized in that: The construction of the feature hint synthesizer includes: First, using the attribute labels in the dataset, the attribute labels are passed into a list containing several attribute labels. Each label is represented by an array, and the elements in the array correspond to visual features. Secondly, define an auxiliary function for processing a single attribute tag array. The auxiliary function determines the construction of the description based on the value of each attribute. When constructing the description, the attribute tag information is concatenated into a string one by one to form a complete sentence. Finally, all input attribute labels are traversed, corresponding descriptions are generated for each attribute label, and all generated descriptions are summarized into a list to output natural text sentences.

5. The multimodal face attribute editing method based on text and image according to claim 3 is characterized in that: The LLM model generates text encoding, and the text encoding and style encoding are fused through a multimodal feature fuser, including: The LLM model is used to encode the semantic information of natural text sentences. Given a natural language sequence x=x1,…,x |x| As input, the decoder predicts the output sequence token by token; at the i-th step, the decoder predicts the i-th token y in a linearized form i , and the decoder status It is expressed as: The conditional probability of the entire output sequence p(y|x) is calculated by i |y <i ,x) are gradually combined into: Among them, y <i =y1…y i-1 , p(y i |y <i ,x) is the probability on the target vocabulary V; For a natural language text sequence x=x1,…,x with a length of K |K| ,Pick As the sentence-level encoding of the text h t and the image encoding h with the output of the visual encoder v For splicing, the formula is: h=concat(h v ,h t ) (7) The image feature h v and text feature h t The common input is fed into the fully connected layer for feature dimension transformation.

6. The multimodal face attribute editing method based on text and image according to claim 1, characterized in that: The generator G converts the translated features into images and generates VLM descriptions through the VLM model.

7. The multimodal face attribute editing method based on text and image according to claim 6, characterized in that: The process of VLM description generation includes: Divide image i into grid image patches, each of which is encoded as a visual token; flatten the grid image patches into a sequence and add position encoding; obtain the encoded image representation x from image I v ; The instruction text is recorded as x t ; Visual tokens are merged into the input sequence x: x=[x v ,x t ] (8) The sequence x is input into the text decoder of VLM, and the image description is generated by predicting the token; at each generation step i, the decoder predicts the next token y based on the predicted token and state i , and update y i Internal State The overall probability of sequence x generating the entire output sequence y is the probability of generating each token by multiplying them together, expressed as: Among them, y <i =y1,…y i-1 represents all tokens predicted before the i-th token; p(y i |y <i ,x) is the probability of token and predicted i-th token; The goal of VLM is to maximize the given sequence X O In the case of the target output image description sequence X T The probability of output image description sentence X T .

8. A multimodal face attribute editing system based on text and image, characterized in that: include: a memory for storing instructions executable by a processor; A processor, configured to execute instructions to implement the text- and image-based multimodal facial attribute editing method as described in any one of claims 1 to 7.

9. A computer readable medium storing a computer program code, characterized in that: When the computer program code is executed by a processor, the computer program code implements the text- and image-based multimodal facial attribute editing method as described in any one of claims 1 to 7.

Images