Customized action image generation method and system based on general action space

Abstract

The invention discloses a customized action image generation method and system based on a general action space. The method comprises the following three key steps: firstly, constructing a general action semantic space based on a group of representative action phrases; secondly, simulating a customized action in the universal action semantic space; and finally, semantic similarity of customized actions is optimized through action similarity loss, so that accurate customized action images irrelevant to context are generated in different situations, and identity consistency of different subjects including animals, human beings and even customized figures is kept at the same time. The method is particularly suitable for few-sample motion image synthesis of a text-guided diffusion model, and constructs a general motion semantic space to solve the challenge of decoupling motion from other semantic features such as figure appearance in a reference motion image under the condition of few-sample images. In addition, the superiority of the method in the aspect of generating the customized action image is verified through experiments.

Description

Technical Field

[0001] The present invention belongs to the field of image generation, and in particular relates to a method and system for generating customized action images based on a universal action space. Background Art

[0002] The goal of customized action image generation is to generate an image that contains a reference action and conforms to the text prompt based on a given reference action image and text prompt. It is an interdisciplinary field between computer vision and natural language processing. It has important value in multiple application scenarios, such as video surveillance, game development, virtual reality, animation production, and social media.

[0003] However, action image generation faces different challenges. It requires not only understanding the semantic content of the action, but also being able to generate new and realistic action images based on text descriptions or other conditions. Traditional action image generation methods rely on complex models such as generative adversarial networks (GANs) or variational autoencoders (VAEs), which have been successful in generating high-quality images but often require a lot of training data and computing resources.

[0004] Existing reports use a small number of samples for fine-tuning to generate images that match specific text descriptions. However, a given reference action image contains many fine-grained semantic features, such as background, style, and appearance of the person. Most existing methods decouple customized actions from other semantic features in the image through contrastive learning. However, due to the large number of fine-grained semantic features, it is very difficult to collect complete negative samples, resulting in the generated customized action images containing information other than the action.

[0005] In summary, the existing technology cannot effectively solve the problem of customized action image generation, resulting in limited performance in actual scene applications and inability to accurately and quickly generate customized action images. Summary of the invention

[0006] The purpose of the present invention is to solve the problems existing in the prior art and provide a method and system for generating customized action images based on a universal action space. The method of the present invention enables the model to generate actions independently of specific actors or contexts by mapping action phrases to a common semantic feature space. In this way, the specific features of actions and actors can be effectively decoupled, so that action information can be reused in different situations and roles, effectively improving model performance.

[0007] In order to achieve the above-mentioned invention object, the present invention specifically adopts the following technical solutions:

[0008] In a first aspect, the present invention provides a method for generating a customized action image based on a universal action space, which comprises the following steps:

[0009] S1. Obtain a reference action generated image dataset and preprocess the reference action generated image dataset, wherein the reference action generated image dataset includes a reference action image and a description text with a custom identifier, and the text in the description text other than the custom identifier is used as context text;

[0010] S2. Generate corresponding images based on the acquired action phrases by using the pre-trained text feature extractor and the UNet network, evaluate the images generated by the UNet network, and filter out action phrases that cannot generate correct action images based on the evaluation results; wherein the text feature extractor includes a text segmenter and a text encoder;

[0011] S3. Input the filtered action phrases into the pre-trained text segmenter to extract the universal action basis vector, and reduce the dimension of the extracted universal action basis vector, and construct the universal action semantic space from the universal action basis vector after the dimension reduction;

[0012] S4. Input the context text to be customized action image generated into the text segmenter to obtain the text segmentation vector of the context text, output the text segmentation vector of the custom identifier by the trained multi-layer perceptron, splice the text segmentation vector of the context text and the text segmentation vector of the custom identifier in the sequence dimension, input the spliced ​​text segmentation vector into the text encoder to obtain the final text prompt feature vector, input the final text prompt feature vector into the trained text-image generation model to generate a customized action image; the text-image generation model is a UNet network including a LoRA layer;

[0013] The multi-layer perceptron and the text-image generation model are jointly trained. During the joint training process, the pre-processed reference action image is input into the pre-trained action encoder to extract action semantic features. The extracted action semantic features are input into the multi-layer perceptron to modulate the coefficients of the action basis in the universal action space, and finally the text segmentation vector of the custom identifier is obtained.

[0014] The total loss used in the joint training is the sum of the reconstruction loss of the diffusion model and the constructed action similarity loss. The parameters of the multi-layer perceptron and the LoRA layer are updated based on minimizing the total loss, and the parameters of the UNet network in the text-image generation model are fixed.

[0015] Based on the above scheme, each step can be implemented in the following preferred specific manner.

[0016] As a preferred embodiment of the above-mentioned first aspect, in the reference action generation image data set of step S1, each character action contains 10 reference action images, 4 description texts are annotated for each reference action image, and the action phrases in the description texts are replaced with custom identifiers to form description texts with custom identifiers.

[0017] As a preferred embodiment of the above-mentioned first aspect, in step S1, the specific process of preprocessing the reference action generation image data set is: preprocessing the reference action images in the reference action generation image data set, first performing super-resolution reconstruction to improve the clarity of the reference action images, then adjusting the reference action images after super-resolution reconstruction to a preset size, and finally performing image enhancement on the resized reference action images to obtain the preprocessed reference action images.

[0018] As a preferred embodiment of the first aspect, the specific process of step S2 is as follows:

[0019] S21. Fill the acquired action phrase into the pre-built text prompt, input the filled text prompt into the pre-trained text feature extractor to obtain the text prompt feature vector of the action phrase, input the text prompt feature vector of the action phrase into the pre-trained UNet network, and the UNet network generates a set of images for each action phrase;

[0020] S22. Evaluate the quality of each set of images and the accuracy and recognizability of the actions in each set of images, and delete action phrases that fail to generate correct actions or cannot recognize actions or whose image quality does not meet the standards.

[0021] As a preferred embodiment of the first aspect, in step S3, a principal component analysis method or a linear discriminant analysis method is used to reduce the dimension of the extracted universal action basis vectors.

[0022] As a preferred embodiment of the first aspect, in step S4, the coefficients of the action base in the universal action space output by the multilayer perceptron are weighted summed with the main components obtained by dimensionality reduction to obtain the text segmentation vector of the custom identifier.

[0023] As a preferred embodiment of the above-mentioned first aspect, in the joint training, after obtaining the text segmentation vector of the custom identifier, the context text in the description text is input into the text segmenter, and then the text segmentation vector output by the text segmenter and the text segmentation vector of the custom identifier output by the multi-layer perceptron are spliced ​​in the sequence dimension, and the spliced ​​text segmentation vector is processed by the text encoder and the text-image generation model in turn to generate a customized action image, and the total loss is calculated based on the generated customized action image and the preprocessed reference action image.

[0024] As a preferred embodiment of the first aspect, in step S4, the action similarity loss The functional form is:

[0025]

[0026] in, represents the reconstruction loss of the diffusion model; Represents action similarity loss; cos_sim() represents cosine similarity; is the action semantic feature extracted by the action encoder from the customized action image; X p represents the action semantic features extracted by the action encoder from the preprocessed reference action image; represents the expectation; x represents the generated customized action image; p(x|c) represents the probability distribution of the customized action image x under condition c; c represents the text prompt feature describing the text.

[0027] In a second aspect, the present invention provides a customized action image generation system based on a universal action space, comprising:

[0028] A preprocessing module is used to obtain a reference action generated image data set and preprocess the reference action generated image data set, wherein the reference action generated image data set includes a reference action image and a description text with a custom identifier, and the text in the description text other than the custom identifier is used as context text;

[0029] The action phrase screening module is used to generate corresponding images according to the acquired action phrases through the pre-trained text feature extractor and the UNet network, evaluate the images generated by the UNet network, and filter out action phrases that cannot generate correct action images according to the evaluation results; wherein the text feature extractor includes a text segmenter and a text encoder;

[0030] A universal action semantic space construction module is used to input the filtered action phrases into a pre-trained text segmenter to extract universal action basis vectors, and to reduce the dimension of the extracted universal action basis vectors, and to construct a universal action semantic space from the universal action basis vectors after the dimension reduction;

[0031] A customized action image generation module is used to input the context text to be customized action image generated into the text segmenter to obtain the text segmentation vector of the context text, output the text segmentation vector of the custom identifier by the trained multi-layer perceptron, splice the text segmentation vector of the context text and the text segmentation vector of the custom identifier in the sequence dimension, input the spliced ​​text segmentation vector into the text encoder to obtain the final text prompt feature vector, and input the final text prompt feature vector into the trained text-image generation model to generate a customized action image; the text-image generation model is a UNet network including a LoRA layer;

[0032] The multi-layer perceptron and the text-image generation model are jointly trained. During the joint training process, the pre-processed reference action image is input into the pre-trained action encoder to extract action semantic features. The extracted action semantic features are input into the multi-layer perceptron to modulate the coefficients of the action basis in the universal action space, and finally the text segmentation vector of the custom identifier is obtained.

[0033] The total loss used in the joint training is the sum of the reconstruction loss of the diffusion model and the constructed action similarity loss. The parameters of the multi-layer perceptron and the LoRA layer are updated based on minimizing the total loss, and the parameters of the UNet network in the text-image generation model are fixed.

[0034] In a third aspect, the present invention provides a computer electronic device comprising a memory and a processor;

[0035] The memory is used to store computer programs;

[0036] The processor is used to implement the customized action image generation method based on the general action space as described in any solution of the first aspect when executing the computer program.

[0037] Compared with the traditional customized action image generation field, the method of the present invention proposes a universal action semantic space and effectively improves the performance of customized action image generation. The present invention has the following beneficial effects:

[0038] (1) By constructing a universal action semantic space and using a multi-layer perceptron (MLP) to modulate the coefficients of the action basis, the present invention can generate highly customized action images based on a small number of sample images. This customization is not limited to imitating specific actions, but can also adapt to different characters and contextual texts, thereby providing users with greater creative freedom.

[0039] (2) By introducing action similarity loss, the present invention optimizes the action semantic similarity between the generated customized action image and the reference action image. This method ensures that the generated customized action image is highly consistent with the original description text or reference action image while maintaining action details and expressiveness.

[0040] (3) The method of the present invention allows the reuse of action information in different application scenarios, whether in animation production, game development or virtual reality applications. In addition, the scalability of the method supports the continuous introduction of new action phrases to enrich the action space, thereby adapting to the ever-changing user needs and technological development. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 is a flow chart of the steps of the method of the present invention;

[0042] Figure 2 A schematic diagram of the process of the method of the present invention when training a multi-layer perceptron and a text-image generation model;

[0043] Figure 3 A schematic diagram of the construction of a general action semantic space according to an embodiment of the present invention;

[0044] Figure 4 It is a system block diagram of the present invention. DETAILED DESCRIPTION

[0045] In order to make the above-mentioned purpose, features and advantages of the present invention more obvious and easy to understand, the specific implementation mode of the present invention is described in detail below in conjunction with the accompanying drawings. In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention can be implemented in many other ways different from those described herein, and those skilled in the art can make similar improvements without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below. The technical features in each embodiment of the present invention can be combined accordingly without conflicting with each other.

[0046] In the description of the present invention, it should be understood that the terms "first" and "second" are only used for the purpose of distinguishing descriptions, and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features.

[0047] like Figure 1 As shown, in a preferred implementation of the present invention, the above-mentioned customized action image generation method based on the universal action space includes the following steps S1 to S4. The specific implementation process is described in detail below.

[0048] S1. Obtain a reference action generated image dataset and preprocess the reference action generated image dataset, wherein the reference action generated image dataset includes a reference action image and a description text with a custom identifier, and the text in the description text except the custom identifier is used as context text.

[0049] It should be noted that in the reference action generation image data set in step S1 of the present invention, each character action contains 10 reference action images, 4 description texts are annotated for each reference action image, and the action phrases in the description texts are replaced with custom identifiers to form description texts with custom identifiers.

[0050] It should be noted that in step S1 of the present invention, the specific process of preprocessing the reference action generation image data set is: preprocessing the reference action image in the reference action generation image data set, first performing super-resolution reconstruction to improve the clarity of the reference action image, then adjusting the reference action image after super-resolution reconstruction to a preset size, and finally performing image enhancement on the reference action image after size adjustment to obtain the preprocessed reference action image.

[0051] In this embodiment, the size of the reference action image is 1024*1024, and when performing image enhancement, image enhancement methods such as random cropping and rotation may be used.

[0052] S2. The pre-trained text feature extractor and the UNet network generate corresponding images based on the acquired action phrases, evaluate the images generated by the UNet network, and filter out action phrases that cannot generate correct action images based on the evaluation results. The text feature extractor includes a text segmenter and a text encoder.

[0053] It should be noted that in step S2 of the present invention, a large number of action phrases are obtained, which cover various human body movements from finger gestures to whole body postures. These action phrases are derived from various action description resources, including but not limited to sports movements, daily life behaviors, dance movements and action terms in other specific fields.

[0054] It should be noted that the specific process of step S2 of the present invention is as follows:

[0055] S21. Fill the acquired action phrase into the pre-built text prompt, input the filled text prompt into the pre-trained text feature extractor to obtain the text prompt feature vector of the action phrase, input the text prompt feature vector of the action phrase into the pre-trained UNet network, and the UNet network generates a set of images for each action phrase.

[0056] In this embodiment, if Figure 3As shown, the filled text prompt may be "a photo of someone*", where "*" is a custom identifier that replaces the action phrase.

[0057] S22. Evaluate the quality of each set of images and the accuracy and recognizability of the actions in each set of images, and delete action phrases that fail to generate correct actions or cannot recognize actions or whose image quality does not meet the standards.

[0058] In this embodiment, when evaluating the matching degree between the generated image and the action phrase, the evaluation criteria include the accuracy of the action in the image, the image quality (whether the details of the action are clearly expressed), the recognizability of the action, etc. Through the evaluation, the action phrases that can accurately generate the image are screened out, and the action phrases that fail the evaluation will be excluded to ensure the accuracy and reliability of the action space established subsequently.

[0059] S3. Input the filtered action phrases into the pre-trained text segmenter to extract the universal action basis vector, and reduce the dimension of the extracted universal action basis vector, and construct the universal action semantic space from the universal action basis vector after the dimension reduction.

[0060] It should be noted that if Figure 3 As shown, in step S3 of the present invention, the principal component analysis method (PCA) or the linear discriminant analysis method (LDA) is used to reduce the dimension of the extracted universal action basis vectors.

[0061] In this embodiment, after filtering all action phrases according to the pre-trained text feature extractor and UNet network, a total of 832 representative action phrases covering finger movements to whole body postures are obtained. These action phrases are input into the pre-trained text segmenter for encoding, such as the tokenizer of the CLIP model, to extract the universal action basis vector of the i-th action phrase where i∈{1,…,m}, represents the 1st, ..., kth word segment of the universal action basis vector of the ith action phrase, m is the number of action phrases, and k is the length of the action phrase. In this embodiment, m = 832, and the length of the action phrase is uniformly set to 3, that is, k = 3. Therefore, in this embodiment, the first word of each word segment in the universal action basis vector of each action phrase constitutes Q 1 vector, the second word constitutes Q 2 vector, the third word constitutes Q 3 vector, then all action basis vectors can be represented by Q 1 , Q 2 and Q 3 Indicates that Qk′ Represents the set consisting of the k′th word of each participle; They represent the k′th word of each participle in the 1st,…,mth action phrases respectively.

[0062] Then the set Q consisting of the k′th word of each word of all action phrases k′ Send it to the PCA algorithm for dimensionality reduction. Specifically, use the function PCA(Q k′ ,p) reduce Dimensions, and generate p main components to form a set This dimensionality reduction transformation plays a vital role in reducing redundant information within the data and improving the efficiency of multi-layer perceptron and LoRA optimization, specifically:

[0063]

[0064] in, represents the Q after dimensionality reduction k′ The main component set of ; p is the specified dimension of dimensionality reduction; Denote the 1st, ..., pth principal components respectively. In this embodiment, p is set to 512, which captures the basic features of the data in a more compact form.

[0065] Finally, this embodiment obtains a set of action bases [B 1 ,B 2 ,B 3 ], these bases construct an action space that does not contain any action-irrelevant information, namely the universal action semantic space, which provides a strong inductive bias for achieving disentangled action label embedding.

[0066] S4. Input the context text to be customized action image generated into the text segmenter to obtain the text segmentation vector of the context text, output the text segmentation vector of the custom identifier by the trained multi-layer perceptron, splice the text segmentation vector of the context text and the text segmentation vector of the custom identifier in the sequence dimension, input the spliced ​​text segmentation vector into the text encoder to obtain the final text prompt feature vector, input the final text prompt feature vector into the trained text-image generation model to generate a customized action image; the text-image generation model is a UNet network including a LoRA layer;

[0067] The multi-layer perceptron and the text-image generation model are jointly trained. During the joint training process, the pre-processed reference action image is input into the pre-trained action encoder to extract action semantic features. The extracted action semantic features are input into the multi-layer perceptron (MLP) to modulate the coefficients of the action basis in the universal action space, and finally the text segmentation vector of the custom identifier is obtained;

[0068] The total loss used in the joint training is the sum of the reconstruction loss of the diffusion model and the constructed action similarity loss. The parameters of the multi-layer perceptron and the LoRA layer are updated based on minimizing the total loss, and the parameters of the UNet network in the text-image generation model are fixed.

[0069] It should be noted that in the joint training of step S4 of the present invention, the action semantic features X in the preprocessed reference action image X are first extracted by the pre-trained action encoder ψ(·). p , and take it as a priori:

[0070] X p =ψ(X)

[0071] In this embodiment, the action encoder specifically adopts CLIP. Then the extracted action semantic feature X p Input to a multilayer perceptron to modulate the coefficients of the action basis in the general action space:

[0072]

[0073] Among them, W k′ The coefficient matrix representing the action basis; They represent the 1st,…,pth coefficients in the coefficient matrix of the action basis respectively.

[0074] Then, the coefficients of the action bases output by the MLP are applied to the universal action semantic space, and the coefficients of each action base in the universal action space output by the multilayer perceptron are weighted and summed with the main components obtained by dimensionality reduction to obtain the text segmentation vector of the custom identifier, which is:

[0075]

[0076] in, represents the coefficient of the j-th action basis in the universal action space.

[0077] The present invention is based on the action base [B 1 ,B 2 ,B 3 ] is combined with the corresponding coefficients [W1, W2, W3] to form a text segmentation vector set of custom identifiers In this way, is a linear combination of the action basis. Therefore, the vector interpolation in the action space will not contain any features unrelated to the action, achieving the goal of decoupling the action from the character in the reference action image.

[0078] It should be noted that in the joint training of the present invention, Figure 2 As shown, after obtaining the text segmentation vector of the custom identifier, the context text in the description text is input into the text segmenter, and then the text segmentation vector output by the text segmenter and the text segmentation vector of the custom identifier output by the multi-layer perceptron are spliced ​​in the sequence dimension, and the spliced ​​text segmentation vector is processed by the text encoder and the text-image generation model in turn to generate a customized action image, and the total loss is calculated based on the generated customized action image and the preprocessed reference action image.

[0079] It should be noted that the customized action images generated by the text-image generation model are compared with the reference action images, and the total loss is optimized. The multi-layer perceptron and the LoRA layer in the text-image generation model are jointly trained until a trained customized text-guided image generation model is obtained. The function form of the total loss is:

[0080]

[0081] in, represents the reconstruction loss of the diffusion model; Represents action similarity loss; cos_sim() represents cosine similarity; is the action semantic feature extracted by the action encoder from the customized action image; X p represents the action semantic features extracted by the action encoder from the preprocessed reference action image; represents the expectation; x represents the generated customized action image; p(x|c) represents the probability distribution of the customized action image x under condition c; c represents the text prompt feature describing the text.

[0082] In the embodiment of the present invention, the reconstruction loss of the diffusion model is specifically:

[0083]

[0084] in, represents Gaussian noise with mean 0 and variance 1; t is the time step; x 0 is the original image; ∈ θ represents a parameterized function, which is used in the diffusion model to predict the noise under given conditions; θ represents the parameters of the text-image generation model; x t Indicates that in x 0The noisy image obtained by adding Gaussian noise ∈ to it; τ θ (c) Represents the text prompt feature vector extracted by the text encoder.

[0085] In addition, the embodiments of the present invention take into account that the existing Lrec aims to faithfully reconstruct the appearance of the sample image and focuses on low-level pixel features. This narrow focus limits the model's ability to identify and invert high-level action features in the image. To address this limitation, the embodiments of the present invention introduce action similarity loss to enhance the model's grasp of high-level action semantics. Starting from the preprocessed reference action image X and the corresponding textual cue features c of the description text, the embodiments of the present invention randomly initialize a latent variable x t . This variable is gradually denoised until it reaches a randomly chosen time step t. At this point, the denoised image x pred Directly from the latent variable x t The goal is to evaluate the action similarity between the two images. Specifically, the action encoder ψ(·) is used to extract the action features in the reference action image and the action features in the generated customized action image. The cosine similarity between these action features is then calculated to measure the action consistency during the generation process. Therefore, the functional form of the action similarity loss is:

[0086]

[0087] Among them, cos_sim() represents cosine similarity; is the action semantic feature extracted by the action encoder from the customized action image; X p represents the action semantic features extracted by the action encoder from the preprocessed reference action image; represents the expectation; x represents the generated customized action image; p(x|c) represents the probability distribution of customized action image x under condition c.

[0088] The present invention will now use a specific example to demonstrate the application effect of a customized action image generation method based on a general action space described in S1 to S4 in the above embodiments on a specific data set, so as to facilitate understanding of the essence of the present invention.

[0089] Example

[0090] The specific implementation process of the customized action image generation method based on the universal action space adopted in this embodiment is as described above and will not be repeated here.

[0091] The embodiment of the present invention is based on the ActionBench benchmark test, which includes 12 actions involving multiple body parts, such as fingers, arms, legs, and whole body movements. Each action contains about 10 reference action images. In addition to actions performed by a single character, more complex actions involving multiple characters are also included.

[0092] In terms of test evaluation criteria, the embodiment of the present invention uses objective indicators to evaluate the quality of the generated image. The CLIP score is used to calculate the action consistency between the reference action image and the generated customized action image, that is, S action In addition, ensuring the consistency of the person's identity is a key aspect of the present invention, so the present invention uses a pre-trained facial recognition encoder S actor To evaluate the identity similarity of actors. In addition, the present invention also conducts user research, including U action and U actor , respectively represent the user’s preferences for action fidelity and character identity consistency in customized action images.

[0093] The implementation details are as follows:

[0094] In the method of the present invention, the AdamW optimizer is used with a learning rate of 2e-4. CLIP is used as a preprocessor (action encoder) to estimate the action features contained in the reference action image. Stable Diffusion XL is selected as the text feature extractor and UNet network, and the image generation resolution is 1024×1024. All experiments are performed on an A-100 GPU.

[0095] Table 1 shows the experimental results of the method of the present invention on the ActionBench dataset. Among them, TextInversion, DreamBooth, Reversion, Custom Diffusion and P+ are the latest customized image generation methods, and ADI is the latest customized action image generation method. TwinAct represents the method proposed by the present invention.

[0096] Table 1. Experimental results of different methods on the ActionBench dataset

[0097]

[0098]

[0099] It can be seen from Table 1 that since the present invention uses an action base based on a universal action semantic space to customize the generation of action images, the method of the present invention is superior to other methods in terms of both action and character similarity, proving its effectiveness in imitating customized actions and filtering irrelevant information that may confuse the performance of new characters.

[0100] In addition, we conducted a quantitative comparison involving human evaluators to further verify the effectiveness of our TwinAct method. We invited 100 users to rate each pair of person-action images generated by our TwinAct method and the baseline, with the ratings ranging from 1 (worst) to 5 (best), focusing on the consistency of the action and the person. The results in the last two rows of Table 1 show that users showed a clear preference for the results generated by our TwinAct method.

[0101] It should also be noted that the customized action image generation method based on the universal action space in the above embodiment can essentially be executed by a computer program or module. Therefore, similarly, based on the same inventive concept, another preferred embodiment of the present invention also provides a customized action image generation system based on the universal action space corresponding to the customized action image generation method based on the universal action space provided in the above embodiment, such as Figure 4 As shown, it includes:

[0102] A preprocessing module is used to obtain a reference action generated image data set and preprocess the reference action generated image data set, wherein the reference action generated image data set includes a reference action image and a description text with a custom identifier, and the text in the description text other than the custom identifier is used as context text;

[0103] The action phrase screening module is used to generate corresponding images according to the acquired action phrases through the pre-trained text feature extractor and the UNet network, evaluate the images generated by the UNet network, and filter out action phrases that cannot generate correct action images according to the evaluation results; wherein the text feature extractor includes a text segmenter and a text encoder;

[0104] A universal action semantic space construction module is used to input the filtered action phrases into a pre-trained text segmenter to extract universal action basis vectors, and to reduce the dimension of the extracted universal action basis vectors, and to construct a universal action semantic space from the universal action basis vectors after the dimension reduction;

[0105] A customized action image generation module is used to input the context text to be customized action image generated into the text segmenter to obtain the text segmentation vector of the context text, output the text segmentation vector of the custom identifier by the trained multi-layer perceptron, splice the text segmentation vector of the context text and the text segmentation vector of the custom identifier in the sequence dimension, input the spliced ​​text segmentation vector into the text encoder to obtain the final text prompt feature vector, and input the final text prompt feature vector into the trained text-image generation model to generate a customized action image; the text-image generation model is a UNet network including a LoRA layer;

[0106] The multi-layer perceptron and the text-image generation model are jointly trained. During the joint training process, the pre-processed reference action image is input into the pre-trained action encoder to extract action semantic features. The extracted action semantic features are input into the multi-layer perceptron to modulate the coefficients of the action basis in the universal action space, and finally the text segmentation vector of the custom identifier is obtained.

[0107] The total loss used in the joint training is the sum of the reconstruction loss of the diffusion model and the constructed action similarity loss. The parameters of the multi-layer perceptron and the LoRA layer are updated based on minimizing the total loss, and the parameters of the UNet network in the text-image generation model are fixed.

[0108] Similarly, based on the same inventive concept, another preferred embodiment of the present invention further provides a computer electronic device corresponding to the customized action image generation method based on the general action space provided in the above embodiment, which includes a memory and a processor;

[0109] The memory is used to store computer programs;

[0110] The processor is used to implement the customized action image generation method based on the general action space in the above embodiment when executing the computer program.

[0111] In addition, the logic instructions in the above-mentioned memory can be implemented in the form of software functional units and can be stored in a computer-readable storage medium when sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention.

[0112] It is understandable that the above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0113] It should also be noted that those skilled in the art can clearly understand that for the convenience and simplicity of description, the specific working process of the system described above can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here. In the various embodiments provided in this application, the division of steps or modules in the system and method is only a logical function division, and there may be other division methods in actual implementation, such as multiple modules or steps can be combined or integrated together, and a module or step can also be split.

[0114] The above-described embodiment is only a preferred solution of the present invention, but it is not intended to limit the present invention. A person skilled in the relevant technical field may make various changes and modifications without departing from the spirit and scope of the present invention. Therefore, any technical solution obtained by equivalent replacement or equivalent transformation falls within the protection scope of the present invention.

Claims

1. A customized action image generation method based on a universal action space, characterized in that: The following steps are involved: S1. Obtain a reference action generated image dataset and preprocess the reference action generated image dataset, wherein the reference action generated image dataset includes a reference action image and a description text with a custom identifier, and the text in the description text other than the custom identifier is used as context text; S2. Generate corresponding images based on the acquired action phrases by using the pre-trained text feature extractor and the UNet network, evaluate the images generated by the UNet network, and filter out action phrases that cannot generate correct action images based on the evaluation results; wherein the text feature extractor includes a text segmenter and a text encoder; S3. Input the filtered action phrases into the pre-trained text segmenter to extract the universal action basis vector, and reduce the dimension of the extracted universal action basis vector, and construct the universal action semantic space from the universal action basis vector after the dimension reduction; S4. Input the context text to be customized action image generated into the text segmenter to obtain the text segmentation vector of the context text, output the text segmentation vector of the custom identifier by the trained multi-layer perceptron, splice the text segmentation vector of the context text and the text segmentation vector of the custom identifier in the sequence dimension, input the spliced ​​text segmentation vector into the text encoder to obtain the final text prompt feature vector, input the final text prompt feature vector into the trained text-image generation model to generate a customized action image; the text-image generation model is a UNet network including a LoRA layer; The multi-layer perceptron and the text-image generation model are jointly trained. During the joint training process, the pre-processed reference action image is input into the pre-trained action encoder to extract action semantic features. The extracted action semantic features are input into the multi-layer perceptron to modulate the coefficients of the action basis in the universal action space, and finally the text segmentation vector of the custom identifier is obtained. The total loss used in the joint training is the sum of the reconstruction loss of the diffusion model and the constructed action similarity loss. The parameters of the multi-layer perceptron and the LoRA layer are updated based on minimizing the total loss, and the parameters of the UNet network in the text-image generation model are fixed.

2. The method for generating a customized action image based on a general action space according to claim 1, characterized in that: In the reference action generation image dataset of step S1, each character action contains 10 reference action images, 4 description texts are annotated for each reference action image, and the action phrases in the description texts are replaced with custom identifiers to form description texts with custom identifiers.

3. The method for generating a customized action image based on a general action space according to claim 1, characterized in that: In step S1, the specific process of preprocessing the reference action generated image data set is: preprocessing the reference action image in the reference action generated image data set, first performing super-resolution reconstruction to improve the clarity of the reference action image, then adjusting the reference action image after super-resolution reconstruction to a preset size, and finally performing image enhancement on the resized reference action image to obtain a preprocessed reference action image.

4. The method for generating a customized action image based on a general action space according to claim 1, characterized in that: The specific process of step S2 is as follows: S21. Fill the acquired action phrase into the pre-built text prompt, input the filled text prompt into the pre-trained text feature extractor to obtain the text prompt feature vector of the action phrase, input the text prompt feature vector of the action phrase into the pre-trained UNet network, and the UNet network generates a set of images for each action phrase; S22. Evaluate the quality of each set of images and the accuracy and recognizability of the actions in each set of images, and delete action phrases that fail to generate correct actions or cannot recognize actions or whose image quality does not meet the standards.

5. The method for generating customized action images based on a general action space according to claim 1, characterized in that: In step S3, the principal component analysis method or the linear discriminant analysis method is used to reduce the dimension of the extracted universal action basis vector.

6. The method for generating customized action images based on a universal action space according to claim 1, characterized in that: In step S4, the coefficients of the action base in the universal action space output by the multilayer perceptron are weighted and summed with the main components obtained by dimensionality reduction to obtain the text segmentation vector of the custom identifier.

7. The method for generating customized action images based on a general action space according to claim 1, characterized in that: In the joint training, after obtaining the text segmentation vector of the custom identifier, the context text in the description text is input into the text segmenter, and then the text segmentation vector output by the text segmenter and the text segmentation vector of the custom identifier output by the multi-layer perceptron are spliced ​​in the sequence dimension. The spliced ​​text segmentation vector is processed by the text encoder and the text-image generation model in turn to generate a customized action image, and the total loss is calculated based on the generated customized action image and the preprocessed reference action image.

8. The method for generating customized action images based on a universal action space according to claim 7, characterized in that: In step S4, the action similarity loss The functional form is: in, represents the reconstruction loss of the diffusion model; Represents action similarity loss; cos_sim() represents cosine similarity; is the action semantic feature extracted by the action encoder from the customized action image; X p represents the action semantic features extracted by the action encoder from the preprocessed reference action image; represents the expectation; x represents the generated customized action image; p(x|c) represents the probability distribution of the customized action image x under condition c; c represents the text prompt feature describing the text.

9. A customized action image generation system based on a universal action space, characterized in that: include: A preprocessing module is used to obtain a reference action generated image data set and preprocess the reference action generated image data set, wherein the reference action generated image data set includes a reference action image and a description text with a custom identifier, and the text in the description text other than the custom identifier is used as context text; The action phrase screening module is used to generate corresponding images according to the acquired action phrases through the pre-trained text feature extractor and the UNet network, evaluate the images generated by the UNet network, and filter out action phrases that cannot generate correct action images according to the evaluation results; wherein the text feature extractor includes a text segmenter and a text encoder; A universal action semantic space construction module is used to input the filtered action phrases into a pre-trained text segmenter to extract universal action basis vectors, and to reduce the dimension of the extracted universal action basis vectors, and to construct a universal action semantic space from the universal action basis vectors after the dimension reduction; A customized action image generation module is used to input the context text to be customized action image generated into the text segmenter to obtain the text segmentation vector of the context text, output the text segmentation vector of the custom identifier by the trained multi-layer perceptron, splice the text segmentation vector of the context text and the text segmentation vector of the custom identifier in the sequence dimension, input the spliced ​​text segmentation vector into the text encoder to obtain the final text prompt feature vector, and input the final text prompt feature vector into the trained text-image generation model to generate a customized action image; the text-image generation model is a UNet network including a LoRA layer; The multi-layer perceptron and the text-image generation model are jointly trained. During the joint training process, the pre-processed reference action image is input into the pre-trained action encoder to extract action semantic features. The extracted action semantic features are input into the multi-layer perceptron to modulate the coefficients of the action basis in the universal action space, and finally the text segmentation vector of the custom identifier is obtained. The total loss used in the joint training is the sum of the reconstruction loss of the diffusion model and the constructed action similarity loss. The parameters of the multi-layer perceptron and the LoRA layer are updated based on minimizing the total loss, and the parameters of the UNet network in the text-image generation model are fixed.

10. A computer electronic device, characterized in that: including memory and processor; The memory is used to store computer programs; The processor is configured to implement the customized action image generation method based on a general action space as described in any one of claims 1 to 8 when executing the computer program.

Images