Digital human generation method and device and digital human generation system

Abstract

The invention discloses a digital human generation method, a digital human generation device and a digital human generation system. The method comprises the following steps: acquiring a face image of an object to be modeled and digital human style text description; an image processing and text semantic understanding algorithm based on artificial intelligence and deep learning is introduced at the rear end to analyze the face image of the object to be modeled and the digital human style text description; in this way, cross-modal semantic significance fusion representation between the face semantic features of the object to be modeled and the semantic features of the digital human style text is captured and described, and the semantic fusion representation is utilized to automatically generate a digital human head portrait. Thus, feature capture and simulation of face features of a real object are realized, and personalized rendering of a digital human head portrait can be performed in combination with digital human style description input by a user, so that a digital human image with personalized features is generated. The intelligent degree and the automation level of digital human generation are improved, and the user experience is improved.

Description

Technical Field

[0001] The present application relates to the field of intelligent generation, and more specifically, to a method and device for generating a digital human, and a system for generating a digital human. Background Art

[0002] Digital Human, also known as virtual human or synthetic human, refers to a virtual person created through computer technology. These digital humans can be completely fictional or digital representations of real people. They can have similar appearance, movements and behavior patterns to real humans, and can interact and perform tasks in various digital environments. These digital humans are not only used in filmmaking and video game development, but also have a wide range of applications in education, medical care, social platforms and other fields.

[0003] However, traditional digital human generation solutions usually require professional technicians to manually adjust every detail, including the digital human's facial features, expressions, clothing style, etc., to achieve the desired effect. This process is time-consuming and laborious, and requires the experience and technical knowledge of high-level technicians. In addition, in the traditional digital human generation process, users can often only provide limited input or rely entirely on the design of the professional team, lacking the means to directly participate in the creation process. This means that the final result may be different from the user's expectations and cannot meet the user's personalized needs.

[0004] Therefore, an optimized digital human generation scheme is desired. Summary of the invention

[0005] In order to solve the above technical problems, the present application is proposed. The embodiment of the present application provides a method, device and system for generating a digital human. After collecting the facial image of the object to be modeled and the digital human style text description, the image processing and text semantic understanding algorithm based on artificial intelligence and deep learning is introduced at the back end to analyze the facial image of the object to be modeled and the digital human style text description, so as to capture and characterize the cross-modal semantic saliency fusion representation between the facial semantic features of the object to be modeled and the digital human style text semantic features, and use the semantic fusion representation to automatically generate a digital human avatar. In this way, not only the feature capture and simulation of the facial features of the real object is realized, but also the personalized rendering of the digital human avatar can be combined with the digital human style description input by the user, so as to generate a digital human image with personalized features. In this way, the intelligence and automation level of digital human generation are improved, and the user experience is improved.

[0006] According to one aspect of the present application, a method for generating a digital human is provided, which comprises: Obtain a face image of the object to be modeled; Obtain a text description of the style of the target generated digital human input by the user; Extracting facial features from the facial image of the object to be modeled to obtain facial features of the object to be modeled; Performing semantic coding on the style text description of the target generated digital human to obtain the style semantic features of the target generated digital human; Performing cross-modal optimization and significant fusion on the face features of the object to be modeled and the style semantic features of the target generated digital human to obtain cross-modal significant fusion features of the face and style semantics of the object to be modeled; A digital human portrait is generated based on the face-style semantic cross-modal significant fusion features of the object to be modeled to obtain a digital human portrait.

[0007] According to another aspect of the present application, a digital human generation system is provided, comprising: A face image acquisition module for the object to be modeled, used to acquire the face image of the object to be modeled; A style text description acquisition module is used to acquire the style text description of the target generated digital human input by the user; A face feature extraction module, used for extracting face features from the face image of the object to be modeled to obtain face features of the object to be modeled; A target generated digital human style semantic encoding module is used to perform semantic encoding on the style text description of the target generated digital human to obtain the target generated digital human style semantic features; A cross-modal optimal salient fusion module, used for performing cross-modal optimal salient fusion on the face features of the object to be modeled and the style semantic features of the target generated digital human to obtain cross-modal salient fusion features of the face and style semantics of the object to be modeled; The digital human portrait generation module is used to generate a digital human portrait based on the face-style semantic cross-modal significant fusion features of the object to be modeled to obtain a digital human portrait.

[0008] Compared with the prior art, the digital human generation method, device and digital human generation system provided by the present application collect the face image of the object to be modeled and the digital human style text description, and then introduce the image processing and text semantic understanding algorithm based on artificial intelligence and deep learning at the back end to analyze the face image of the object to be modeled and the digital human style text description, so as to capture and depict the cross-modal semantic saliency fusion representation between the face semantic features of the object to be modeled and the digital human style text semantic features, and use the semantic fusion representation to automatically generate the digital human avatar. In this way, not only the feature capture and simulation of the face features of the real object is realized, but also the personalized rendering of the digital human avatar can be combined with the digital human style description input by the user, so as to generate a digital human image with personalized features. In this way, the intelligence and automation level of digital human generation are improved, and the user experience is improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] By describing the embodiments of the present application in more detail in conjunction with the accompanying drawings, the above and other purposes, features and advantages of the present application will become more apparent. The accompanying drawings are used to provide a further understanding of the embodiments of the present application and constitute a part of the specification. Together with the embodiments of the present application, they are used to explain the present application and do not constitute a limitation of the present application. In the accompanying drawings, the same reference numerals generally represent the same components or steps.

[0010] Figure 1 is a flow chart of a method for generating a digital human according to an embodiment of the present application; Figure 2 A data flow diagram of a method for generating a digital human according to an embodiment of the present application; Figure 3 is a flowchart of sub-step S5 of the method for generating a digital human according to an embodiment of the present application; Figure 4 4 is a block diagram of a digital human generation system according to an embodiment of the present application. DETAILED DESCRIPTION

[0011] Below, the exemplary embodiments according to the present application will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments of the present application, and it should be understood that the present application is not limited to the exemplary embodiments described here.

[0012] As shown in this application and claims, unless the context clearly indicates an exception, the words "a", "an", "an" and / or "the" do not refer to the singular and may also include the plural. Generally speaking, the terms "include" and "comprise" only indicate the inclusion of the steps and elements that have been clearly identified, and these steps and elements do not constitute an exclusive list. The method or device may also include other steps or elements.

[0013] Although the present application makes various references to certain modules in the system according to the embodiments of the present application, any number of different modules can be used and run on the user terminal and / or server. The modules are only illustrative, and different aspects of the system and method can use different modules.

[0014] Flowcharts are used in the present application to illustrate the operations performed by the system according to the embodiments of the present application. It should be understood that the preceding or following operations are not necessarily performed accurately in order. On the contrary, various steps may be processed in reverse order or simultaneously as required. Meanwhile, other operations may also be added to these processes, or a certain step or several steps of operations may be removed from these processes.

[0015] Below, the exemplary embodiments according to the present application will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments of the present application, and it should be understood that the present application is not limited to the exemplary embodiments described here.

[0016] Traditional digital human generation solutions usually require professional technicians to manually adjust every detail, including the digital human's facial features, expressions, clothing style, etc., to achieve the desired effect. This process is time-consuming and laborious, and requires the experience and technical knowledge of high-level technicians. In addition, in the traditional digital human generation process, users can often only provide limited input or rely entirely on the design of the professional team, lacking the means to directly participate in the creation process. This means that the final result may be different from the user's expectations and cannot meet the user's personalized needs. Therefore, an optimized digital human generation solution is desired.

[0017] In the technical solution of the present application, a method for generating a digital human is proposed. Figure 1 Flow chart of a method for generating a digital human according to an embodiment of the present application. Figure 2 Schematic diagram of data flow of the method for generating a digital human according to an embodiment of the present application. Figure 1 and Figure 2As shown, according to the embodiment of the present application, the method for generating a digital human includes the following steps: S1, obtaining a facial image of an object to be modeled; S2, obtaining a style text description of a target generated digital human input by a user; S3, performing facial feature extraction on the facial image of the object to be modeled to obtain facial features of the object to be modeled; S4, performing semantic encoding on the style text description of the target generated digital human to obtain style semantic features of the target generated digital human; S5, performing cross-modal preferential significant fusion on the facial features of the object to be modeled and the style semantic features of the target generated digital human to obtain cross-modal significant fusion features of the face-style semantics of the object to be modeled; S6, generating a digital human avatar based on the cross-modal significant fusion features of the face-style semantics of the object to be modeled to obtain a digital human avatar.

[0018] In particular, the steps S1 and S2, obtain a face image of the object to be modeled; and obtain a style text description of the digital human generated by the target input by the user. It should be understood that the face is the core component of the digital human, and the real face feature capture technology aims to accurately extract and simulate various subtle features of the face, including facial contours, facial features, skin color, expressions, etc., which are helpful to generate a digital human image that is more in line with the actual state of the face of the modeled object. In addition, since users' expectations of digital humans are often highly personalized, and the style requirements of digital humans vary in different scenes. Therefore, in order to allow users to participate in the design and generation process of digital humans, the digital human avatar is personalized according to the user's style requirements and preferences and the user's actual facial state, and a digital human image with personalized characteristics is generated, thereby enhancing the user's experience. Therefore, first, obtain a face image of the object to be modeled; and obtain a style text description of the digital human generated by the target input by the user.

[0019] In particular, S3 extracts facial features from the facial image of the object to be modeled to obtain facial features of the object to be modeled. In a specific example of the present application, the facial image of the object to be modeled is input into a facial feature extractor based on a deep separable convolutional neural network model for feature mining to extract implicit semantic feature information about the face to be modeled in the facial image of the object to be modeled, thereby obtaining a facial feature map of the object to be modeled as the facial features of the object to be modeled. It is worth mentioning that the facial feature map of the object to be modeled contains important facial features and semantics of the object to be modeled, such as the position and shape of the eyes, nose, and mouth. The facial feature map of the object to be modeled extracted by the feature extractor can be used as the basis for subsequent digital human avatar generation, helping the digital human avatar generator to better understand the facial structure and style, and helping to generate personalized digital human avatars that better meet user needs.

[0020] In particular, the S4 performs semantic encoding on the style text description of the target generated digital person to obtain the style semantic features of the target generated digital person. In a specific example of the present application, the style text description of the target generated digital person is semantically encoded to obtain the style semantic encoding vector of the target generated digital person as the style semantic features of the target generated digital person. By performing semantic encoding on the style text description of the target generated digital person, the semantic feature information in the style text description of the target generated digital person is extracted, thereby obtaining the style semantic encoding vector of the target generated digital person. In one example, the process of semantically encoding the style text description of the target generated digital person includes: performing word segmentation on the style text description of the target generated digital person to convert the style text description of the target generated digital person into a word sequence composed of multiple words; using the embedding layer of the semantic encoder including the embedding layer to map each word in the word sequence into a word embedding vector to obtain a sequence of word embedding vectors; using the converter of the semantic encoder including the embedding layer to perform global context semantic encoding based on the converter idea on the sequence of word embedding vectors to obtain multiple global context semantic feature vectors; and, cascading the multiple global context semantic feature vectors to obtain the style semantic encoding vector of the target generated digital person.

[0021] In particular, the S5 performs cross-modal optimization and significant fusion of the facial features of the object to be modeled and the style semantic features of the target generated digital human to obtain the cross-modal significant fusion features of the face-style semantics of the object to be modeled. It should be understood that since the facial feature map of the object to be modeled contains the facial semantics and implicit feature information of the object to be modeled; and the target generated digital human style semantic encoding vector contains the digital human style description semantics input by the user, which represents the user's personalized needs and preferences. Therefore, in order to comprehensively utilize the complementary information between the two different modal data features of the facial semantic features of the object to be modeled and the style semantic features of the digital human described by the user to enhance the effect of the final digital human avatar generation, so that the generated digital human avatar is more in line with the user's expected style, in the technical solution of the present application, the facial features of the object to be modeled and the style semantic features of the target generated digital human are further cross-modally optimized and significantly fused to obtain the cross-modal significant fusion features of the face-style semantics of the object to be modeled. In particular, the cross-modal preferred salient fusion method can generate a unified and optimized cross-modal salient fusion feature representation by analyzing and fusing semantic feature information from different perception channels (i.e., the semantic features of the face image of the object to be modeled and the semantic features of the user input style description). This fusion encoding method aims to identify and utilize the potential connections between different data sources, even if these connections are not directly apparent or obvious. This is very important for generating high-quality digital human avatars that meet user needs. In a specific example of this application, Figure 3As shown, the S5 includes: S51, performing feature association coding on the facial features of the object to be modeled and the style semantic features of the target generated digital human to obtain the implicit common features of the face-style semantics of the object to be modeled; S52, performing preliminary feature screening on the implicit common features of the face-style semantics of the object to be modeled to obtain the preliminary screened face-style semantic cross-modal joint coding features of the object to be modeled; S53, performing secondary feature screening on the preliminary screened face-style semantic cross-modal joint coding features of the object to be modeled to obtain the face-style semantic cross-modal significant fusion features of the object to be modeled.

[0022] Specifically, the S51 performs feature association coding on the face features of the object to be modeled and the style semantic features of the target generated digital human to obtain the face-style semantic implicit common features of the object to be modeled. Specifically, first, the face feature map of the object to be modeled is feature distilled to obtain the face semantic distillation feature vector of the object to be modeled; by performing feature distillation on the face feature map of the object to be modeled, redundant information is removed while retaining the expression of the key face semantic features of the object to be modeled, and the components that contribute the most to the subsequent digital human head portrait generation task are retained, thereby obtaining a more compact and representative face feature expression of the object to be modeled. More importantly, the feature distillation technology used here can also ensure that the data semantic features of different modalities of image and text can be compared and fused in the same semantic space, promoting effective interaction in subsequent stages. Then, the target generated digital human style semantic coding vector and the face semantic distillation feature vector of the object to be modeled are input into the multimodal implicit association feature capture network to obtain the face-style semantic implicit common feature vector of the object to be modeled as the face-style semantic implicit common feature of the object to be modeled. That is, the target generated digital human style semantic encoding vector and the face distillation feature vector of the object to be modeled are input into a multimodal implicit correlation feature capture network together, so as to identify and capture the potential but not directly manifested common patterns or interactive information between the target generated digital human style description semantics and the face semantics of the object to be modeled. Specifically, this network uses a collaborative attention mechanism to enhance the consistent expression between the two different modal features of the target generated digital human style description semantic features and the face semantic features of the object to be modeled, identify and capture the potential common patterns or internal connections between the two sets of data features, and construct a face-style semantic implicit common feature vector of the object to be modeled that combines the face semantic features and the digital human style semantic features.

[0023] The specific process of performing feature distillation on the face feature map of the object to be modeled to obtain the face semantic distillation feature vector of the object to be modeled includes: performing point convolution coding on the face feature map of the object to be modeled to obtain the face semantic feature coding channel modulation feature map of the object to be modeled; performing global mean pooling processing on each feature matrix along the channel dimension in the face semantic feature coding channel modulation feature map of the object to be modeled to obtain the face semantic channel feature vector of the object to be modeled; inputting the face semantic channel feature vector of the object to be modeled into The function is used to process the face semantic distillation feature vector of the object to be modeled.

[0024] More specifically, the specific process of inputting the target generated digital human style semantic coding vector and the face semantic distillation feature vector of the object to be modeled into the multimodal implicit association feature capture network to obtain the face-style semantic implicit common feature vector of the object to be modeled as the face-style semantic implicit common feature of the object to be modeled includes: cascading the target generated digital human style semantic coding vector and the face semantic distillation feature vector of the object to be modeled to obtain the face-style semantic cascade feature vector of the object to be modeled; multiplying the face-style semantic cascade feature vector of the object to be modeled with the linear transformation weight matrix and then adding them to the linear transformation bias vector by position to obtain the linear transformation face-style semantic cascade feature vector of the object to be modeled; and concatenating the linear transformation face-style semantic cascade feature vector of the object to be modeled through The function is used to process the face-style semantic implicit common feature vector of the object to be modeled.

[0025] Specifically, the S52 performs a preliminary feature screening on the implicit common features of the face-style semantics of the object to be modeled to obtain the preliminary screened face-style semantic cross-modal joint coding features of the object to be modeled. In the technical solution of the present application, a gating unit is used to implement a first-level feature screening mechanism to process the implicit common feature vector of the face-style semantics of the object to be modeled, so as to realize the importance evaluation and selection of features. Specifically, the gating unit dynamically controls the information flow through the learned weight parameters, allowing the key features related to the subsequent digital human portrait generation task to pass while suppressing other irrelevant or minor parts, and then calculates the first-level face-style semantic feature preliminary screening weight vector of the object to be modeled. The advantage of doing this is that it can significantly reduce the interference of noise on the subsequent digital human portrait generation task, and make the model pay more attention to the key features that are really helpful for digital human generation, while also improving the efficiency of the entire system, because only the most valuable information is retained for subsequent calculations. Furthermore, the weight vector is used to perform a preliminary feature screening on the face feature map of the object to be modeled, and the semantic content of the original face image is reorganized in a weighted average manner according to the importance, so as to generate a feature that is more focused on the subsequent digital human head portrait generation task, that is, the preliminary screening face-style cross-modal joint encoding feature map of the object to be modeled. Since the target generated digital human style semantic feature information from another modality has been taken into account at this time, the obtained result not only reflects the semantic features of the single face image of the object to be modeled and the semantic features of the digital human style text, but also reflects the potential complementary effect between the semantics of the two modalities of image and text.

[0026] Specifically, the process of performing preliminary feature screening on the implicit common features of the face-style semantics of the object to be modeled includes: first, inputting the implicit common feature vector of the face-style semantics of the object to be modeled into a feature preliminary screening module based on a gating unit to obtain a first-level face-style semantic feature preliminary screening weight vector of the object to be modeled; then, based on the first-level face-style semantic feature preliminary screening weight vector of the object to be modeled, performing preliminary feature screening on the face feature map of the object to be modeled to obtain a preliminary screened face-style semantic cross-modal joint coding feature map of the object to be modeled as the preliminary screened face-style semantic cross-modal joint coding feature of the object to be modeled.

[0027] The method comprises: inputting the face-style semantic implicit common feature vector of the object to be modeled into a feature preliminary screening module based on a gating unit to obtain a first-level face-style semantic feature preliminary screening weight vector of the object to be modeled, including: multiplying the face-style semantic implicit common feature vector of the object to be modeled with a preliminary screening weight matrix and then adding the vector with the preliminary screening bias vector by position to obtain a preliminary screening linear transformation face-style semantic implicit common feature vector of the object to be modeled; and transmitting the preliminary screening linear transformation face-style semantic implicit common feature vector of the object to be modeled through The function is used to process the face-style semantic feature preliminary screening modulation vector of the first-level object to be modeled; the face-style semantic feature preliminary screening modulation vector of the first-level object to be modeled is input into the mask function for processing to obtain the face-style semantic feature preliminary screening weight vector of the first-level object to be modeled; wherein, in response to the eigenvalue in the face-style semantic feature preliminary screening modulation vector of the first-level object to be modeled being greater than twice a preset threshold, the eigenvalue greater than twice the preset threshold is taken as the original value; in response to the eigenvalue in the face-style semantic feature preliminary screening modulation vector of the first-level object to be modeled being less than the preset threshold, the eigenvalue less than the preset threshold is set to 0; in response to the eigenvalue in the face-style semantic feature preliminary screening modulation vector of the first-level object to be modeled being greater than the preset threshold and less than twice the preset threshold, half of the eigenvalue is taken.

[0028] More specifically, based on the primary face-style semantic feature preliminary screening weight vector of the object to be modeled, the face feature map of the object to be modeled is subjected to preliminary feature screening to obtain a preliminary screened face-style semantic cross-modal joint coding feature map of the object to be modeled as the preliminary screened face-style semantic cross-modal joint coding feature of the object to be modeled, including: using each feature value in the primary face-style semantic feature preliminary screening weight vector of the object to be modeled as a channel weighting weight, performing weighted processing on the face feature map of the object to be modeled based on the channel dimension to obtain a channel preliminary screened face feature map of the object to be modeled; inputting the channel preliminary screened face feature map of the object to be modeled The function is used to process the face-style semantic cross-modal joint encoding feature map of the initially screened object to be modeled.

[0029] Specifically, in S53, the face-style semantic cross-modal joint coding features of the primary screening object to be modeled are subjected to secondary feature screening to obtain the face-style semantic cross-modal significant fusion features of the object to be modeled. That is, the implicit common feature vector of the face-style semantics of the object to be modeled is sent to the feature secondary screening module composed of multiple hierarchical gating units. Compared with the single-level design, the multi-level architecture provides more abundant control means and can manage the process of information transmission more finely. Each layer can focus on a specific type of feature extraction or pattern discovery, thereby enhancing the different modal consistency expressions between the semantics of the face image of the object to be modeled and the semantics of the digital human style text, and identifying and capturing the potential common patterns or structures between the two modal features. This method effectively enhances the ability of the model to capture the complex interaction patterns between the target generated digital human style semantic features and the face features of the object to be modeled, and also improves the quality of the final output results. Finally, the face-style cross-modal joint coding feature map of the primary screening object to be modeled previously formed is refined for the second time using the latest secondary face-style semantic feature screening weight vector of the object to be modeled. After two consecutive targeted selections and reconstructions, we finally obtained a highly concentrated face-style semantic cross-modal significant fusion feature map of the object to be modeled that is rich in significant correlation fusion feature information between the two modalities of face image and style text.

[0030] Specifically, the process of subjecting the initially screened face-style semantic cross-modal joint encoding features of the object to be modeled to secondary feature screening includes: first, inputting the implicit common feature vector of the face-style semantics of the object to be modeled into a secondary feature screening module based on a multi-level gating unit to obtain a secondary face-style semantic feature screening weight vector of the object to be modeled; then, based on the secondary face-style semantic feature screening weight vector of the object to be modeled, performing secondary feature screening on the initially screened face-style semantic cross-modal joint encoding feature map of the object to be modeled to obtain a face-style semantic cross-modal significant fusion feature map of the object to be modeled as the face-style semantic cross-modal significant fusion feature of the object to be modeled.

[0031] The method comprises: inputting the face-style semantic implicit common feature vector of the object to be modeled into a secondary feature screening module based on a multi-level gating unit to obtain a secondary face-style semantic feature screening weight vector of the object to be modeled, including: multiplying the face-style semantic implicit common feature vector of the object to be modeled with a preliminary screening weight matrix and then adding the initial screening bias vector by position to obtain the preliminary screening linear transformation face-style semantic implicit common feature vector of the object to be modeled; and filtering the face-style semantic implicit common feature vector of the object to be modeled by the preliminary screening linear transformation. The face-style semantic feature initial screening modulation vector of the first-level object to be modeled is processed in the function to obtain the face-style semantic feature initial screening modulation vector of the first-level object to be modeled; the face-style semantic feature initial screening modulation vector of the first-level object to be modeled is input into the mask function for processing to obtain the face-style semantic feature screening weight vector of the second-level object to be modeled.

[0032] More specifically, based on the secondary face-style semantic feature screening weight vector of the object to be modeled, the primary screened face-style semantic cross-modal joint coding feature map of the object to be modeled is subjected to secondary feature screening to obtain a face-style semantic cross-modal significant fusion feature map of the object to be modeled as the face-style semantic cross-modal significant fusion feature of the object to be modeled, including: using each feature value in the secondary face-style semantic feature screening weight vector of the object to be modeled as a channel weighting weight, performing weighted processing based on the channel dimension on the primary screened face-style semantic cross-modal joint coding feature map of the object to be modeled to obtain a channel-screened face-style semantic cross-modal joint coding feature map of the object to be modeled; inputting the channel-screened face-style semantic cross-modal joint coding feature map of the object to be modeled The function is used to process the face-style semantic cross-modal significant fusion feature map of the object to be modeled.

[0033] In summary, in the above embodiment, the facial features of the object to be modeled and the style semantic features of the target generated digital human are cross-modally optimized and significantly fused to obtain the face-style semantic cross-modal significant fusion features of the object to be modeled, including: using the following cross-modal fusion formula to perform cross-modal optimization and significant fusion on the facial features of the object to be modeled and the style semantic features of the target generated digital human to obtain the face-style semantic cross-modal significant fusion features of the object to be modeled; Among them, the cross-modal fusion formula is: ;in, is the facial feature map of the object to be modeled, is point convolutional coding, is the mean pooling process, for function, The semantic distillation feature vector of the face of the object to be modeled, Generate a digital human style semantic encoding vector for the target, For vector cascade processing, and are the linear transformation weight matrix and the linear transformation bias vector respectively, for function, is the implicit common feature vector of face-style semantics of the object to be modeled, and are the initial screening weight matrix and the initial screening bias vector respectively, is matrix multiplication, is the initial screening modulation vector of the face-style semantic features of the first-level object to be modeled, is the mask function, is the initial screening weight vector of face-style semantic features of the first-level object to be modeled, is the channel point product by position, To initially screen the face-style semantic cross-modal joint encoding feature map of the object to be modeled, and are the screening weight matrix and the screening bias vector respectively, Filter the modulation vector for the face-style semantic features of the secondary object to be modeled, Filter the weight vector for the face-style semantic features of the secondary object to be modeled, is the predetermined threshold, The face-style semantic cross-modal significant fusion feature map of the object to be modeled.

[0034] In particular, the S6 generates a digital human portrait based on the face-style semantic cross-modal significant fusion feature of the object to be modeled to obtain a digital human portrait. In a specific example of the present application, the face-style semantic cross-modal significant fusion feature map of the object to be modeled is input into a digital human portrait generator based on a diffusion model to obtain the digital human portrait. That is, the digital human portrait is automatically generated by using the cross-modal semantic saliency fusion representation between the semantic features of the face image of the object to be modeled and the semantic features of the digital human style text description. In this way, not only the feature capture and simulation of the facial features of the real object is achieved, but also the personalized rendering of the digital human portrait can be combined with the digital human style description input by the user, thereby generating a digital human image with personalized features. In this way, the intelligence and automation level of digital human generation are improved, and the user experience is improved.

[0035] Preferably, when the facial feature map of the object to be modeled and the style semantic coding vector of the target generated digital human respectively represent the image semantic features of the facial image of the object to be modeled and the text semantic features of the style text description of the target generated digital human, when performing cross-modal preferential encoding based on multimodal implicit associations, the difference in the distribution of cross-modal semantic implicit associations will also make the cross-modal spatial composite coding semantic representation of the face-style semantic cross-modal significant fusion feature map of the object to be modeled have a complex spatial structure based on different semantic association distribution structures, thereby expecting to improve the convergence and generalization effects of its diffuse generative regression under complex spatial structures.

[0036] Therefore, in a preferred example of the present application, when the face-style semantic cross-modal significant fusion feature map of the object to be modeled is input into a digital human portrait generator based on a diffusion model to obtain a digital human portrait, the face-style semantic cross-modal significant fusion feature map of the object to be modeled is optimized, specifically including: The absolute value sum of each eigenvalue of the face-style semantic cross-modal significant fusion feature map of the object to be modeled is calculated to obtain the first face-style semantic cross-modal significant fusion spatial structure value of the object to be modeled, and the square root of the sum of the squares of each eigenvalue of the face-style semantic cross-modal significant fusion feature map of the object to be modeled is calculated to obtain the second face-style semantic cross-modal significant fusion spatial structure value of the object to be modeled, that is, ;in, , are the eigenvalues ​​of the face-style semantic cross-modal significant fusion feature map of the object to be modeled, is the spatial structure value of the face-style semantic cross-modal significant fusion of the first object to be modeled, is the second object to be modeled face-style semantic cross-modal significant fusion spatial structure value; Multiplying each eigenvalue of the face-style semantic cross-modal significant fusion feature map of the object to be modeled by the first face-style semantic cross-modal significant fusion spatial structure value of the object to be modeled and the second face-style semantic cross-modal significant fusion spatial structure value of the object to be modeled to obtain a first face-style semantic cross-modal significant fusion structure reference value of the object to be modeled and a second face-style semantic cross-modal significant fusion structure reference value of the object to be modeled corresponding to each eigenvalue; Multiplying each eigenvalue of the face-style semantic cross-modal significant fusion feature map of the object to be modeled by the scale of the face-style semantic cross-modal significant fusion feature map of the object to be modeled and the square root of the scale to obtain a first face-style semantic cross-modal significant fusion scale transformation value of the object to be modeled and a second face-style semantic cross-modal significant fusion scale transformation value of the object to be modeled corresponding to each eigenvalue; Dividing the first to-be-modeled object face-style semantic cross-modal significant fusion structure reference value by the difference between the first to-be-modeled object face-style semantic cross-modal significant fusion space structure value and the first to-be-modeled object face-style semantic cross-modal significant fusion scale transformation value to obtain the first to-be-modeled object face-style semantic cross-modal significant fusion transformation adjustment value; Dividing the second to-be-modeled object face-style semantic cross-modal significant fusion structure reference value by the difference between the second to-be-modeled object face-style semantic cross-modal significant fusion spatial structure value and the second to-be-modeled object face-style semantic cross-modal significant fusion scale transformation value to obtain the second to-be-modeled object face-style semantic cross-modal significant fusion transformation adjustment value; Calculate the weighted sum of the first object to be modeled face-style semantic cross-modal significant fusion transformation adjustment value and the second object to be modeled face-style semantic cross-modal significant fusion transformation adjustment value to obtain each feature value of the optimized object to be modeled face-style semantic cross-modal significant fusion feature map.

[0037] Among them, the optimized face-style semantic cross-modal significant fusion feature map of the object to be modeled It is expressed as: ;in, .and , , and are the feature matrix width, feature matrix height and number of channels of the face-style semantic cross-modal significant fusion feature map of the object to be modeled, respectively. is the face-style semantic cross-modal significant fusion transformation adjustment value of the first object to be modeled, is the second object to be modeled face-style semantic cross-modal significant fusion transformation adjustment value, is the scale of the face-style semantic cross-modal significant fusion feature map of the object to be modeled, is a first to-be-modeled object face-style semantic cross-modal significant fusion transformation adjustment image composed of the first to-be-modeled object face-style semantic cross-modal significant fusion transformation adjustment value, is a second to-be-modeled object face-style semantic cross-modal significant fusion transformation adjustment image composed of the second to-be-modeled object face-style semantic cross-modal significant fusion transformation adjustment value, is a weighted hyperparameter, To add by position, For point-by-position multiplication, It is the optimized face-style semantic cross-modal significant fusion feature map of the object to be modeled.

[0038] That is, in this preferred example, with respect to the spatial structure information of the feature set of the face-style semantic cross-modal significant fusion feature map of the object to be modeled in the high-dimensional space, a scale-based frame transformation of each feature value of the face-style semantic cross-modal significant fusion feature map of the object to be modeled is performed by taking the class-norm space structured representation of the face-style semantic cross-modal significant fusion feature map of the object to be modeled as a reference window, and a spatial structure-based frame attention weight adjustment of each feature value of the face-style semantic cross-modal significant fusion feature map of the object to be modeled is implemented to ensure the spatial transformation invariance of the face-style semantic cross-modal significant fusion feature map of the object to be modeled under the interaction of feature space, thereby improving the convergence and generalization effect of the diffusion generative regression of the feature set of the face-style semantic cross-modal significant fusion feature map of the object to be modeled under the complex spatial structure representation, and improving the image quality of the obtained digital human portrait. In this way, not only the feature capture and simulation of the facial features of the real object is achieved, but also the personalized rendering of the digital human avatar can be performed in combination with the digital human style description input by the user, thereby generating a digital human image with personalized characteristics. In this way, the intelligence level of digital human generation and the user experience are improved.

[0039] In summary, the method for generating a digital human according to the embodiment of the present application is explained. After collecting the facial image of the object to be modeled and the digital human style text description, an image processing and text semantic understanding algorithm based on artificial intelligence and deep learning is introduced at the back end to analyze the facial image of the object to be modeled and the digital human style text description, so as to capture and characterize the cross-modal semantic saliency fusion representation between the facial semantic features of the object to be modeled and the digital human style text semantic features, and use the semantic fusion representation to automatically generate a digital human avatar. In this way, not only the feature capture and simulation of the facial features of the real object is achieved, but also the personalized rendering of the digital human avatar can be combined with the digital human style description input by the user, so as to generate a digital human image with personalized features. In this way, the intelligence and automation level of digital human generation are improved, and the user experience is improved.

[0040] Furthermore, a digital human generation system is also provided.

[0041] Figure 4 FIG. 1 is a block diagram of a digital human generation system according to an embodiment of the present application. Figure 4As shown, according to the embodiment of the present application, the digital human generation system 300 includes: a face image acquisition module 310 for the object to be modeled, which is used to acquire the face image of the object to be modeled; a style text description acquisition module 320, which is used to acquire the style text description of the target generated digital human input by the user; a face feature extraction module 330, which is used to extract the face features of the face image of the object to be modeled to obtain the face features of the object to be modeled; a target generated digital human style semantic encoding module 340, which is used to semantically encode the style text description of the target generated digital human to obtain the style semantic features of the target generated digital human; a cross-modal preferred significant fusion module 350, which is used to cross-modally preferred and significantly fuse the face features of the object to be modeled and the style semantic features of the target generated digital human to obtain the face-style semantic cross-modal significant fusion features of the object to be modeled; a digital human avatar generation module 360, which is used to generate a digital human avatar based on the face-style semantic cross-modal significant fusion features of the object to be modeled to obtain a digital human avatar.

[0042] As described above, the digital human generation system 300 according to the embodiment of the present application can be implemented in various wireless terminals, such as a server with a digital human generation algorithm. In a possible implementation, the digital human generation system 300 according to the embodiment of the present application can be integrated into the wireless terminal as a software module and / or a hardware module. For example, the digital human generation system 300 can be a software module in the operating system of the wireless terminal, or can be an application developed for the wireless terminal; of course, the digital human generation system 300 can also be one of the many hardware modules of the wireless terminal.

[0043] Alternatively, in another example, the digital human generation system 300 and the wireless terminal may be separate devices, and the digital human generation system 300 may be connected to the wireless terminal via a wired and / or wireless network and transmit interactive information in accordance with an agreed data format.

[0044] The embodiments of the present disclosure have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and changes will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The selection of terms used herein is intended to best explain the principles of the embodiments, practical applications, or improvements to the technology in the market, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein.

Claims

1. A method for generating a digital human, characterized in that: include: Obtain a face image of the object to be modeled; Obtain a text description of the style of the target generated digital human input by the user; Extracting facial features from the facial image of the object to be modeled to obtain facial features of the object to be modeled; Performing semantic coding on the style text description of the target generated digital human to obtain the style semantic features of the target generated digital human; The facial features of the object to be modeled and the style semantic features of the target generated digital human are cross-modally preferentially significantly fused to obtain the face-style semantic cross-modal significant fusion features of the object to be modeled, including: performing feature association coding on the facial features of the object to be modeled and the style semantic features of the target generated digital human to obtain the face-style semantic implicit common features of the object to be modeled; performing preliminary feature screening on the face-style semantic implicit common features of the object to be modeled to obtain the pre-screened face-style semantic cross-modal joint coding features of the object to be modeled; performing secondary feature screening on the pre-screened face-style semantic cross-modal joint coding features of the object to be modeled to obtain the face-style semantic cross-modal significant fusion features of the object to be modeled; A digital human portrait is generated based on the face-style semantic cross-modal significant fusion features of the object to be modeled to obtain a digital human portrait.

2. The method for generating a digital human according to claim 1, characterized in that: Performing facial feature extraction on the facial image of the object to be modeled to obtain facial features of the object to be modeled, including: inputting the facial image of the object to be modeled into a facial feature extractor based on a deep separable convolutional neural network model to obtain a facial feature map of the object to be modeled as the facial features of the object to be modeled.

3. The method for generating a digital human according to claim 2, characterized in that: The style text description of the target generated digital human is semantically encoded to obtain the style semantic features of the target generated digital human, including: the style text description of the target generated digital human is semantically encoded to obtain the style semantic encoding vector of the target generated digital human as the style semantic features of the target generated digital human.

4. The method for generating a digital human according to claim 3, characterized in that: Performing feature association coding on the face features of the object to be modeled and the style semantic features of the target generated digital human to obtain the face-style semantic implicit common features of the object to be modeled, including: Performing feature distillation on the face feature map of the object to be modeled to obtain a semantic distillation feature vector of the face of the object to be modeled; The target generated digital human style semantic encoding vector and the face semantic distillation feature vector of the object to be modeled are input into a multimodal implicit association feature capture network to obtain the face-style semantic implicit common feature vector of the object to be modeled as the face-style semantic implicit common feature of the object to be modeled.

5. The method for generating a digital human according to claim 4, characterized in that: Performing feature distillation on the face feature map of the object to be modeled to obtain a semantic distillation feature vector of the face of the object to be modeled, including: Performing point convolution coding on the face feature map of the object to be modeled to obtain a channel modulation feature map of the semantic feature coding of the face of the object to be modeled; Performing global mean pooling processing on each feature matrix along the channel dimension in the modulation feature map of the facial semantic feature encoding channel of the object to be modeled to obtain a facial semantic channel feature vector of the object to be modeled; Input the semantic channel feature vector of the face of the object to be modeled The function is used to process the face semantic distillation feature vector of the object to be modeled.

6. The method for generating a digital human according to claim 5, characterized in that: The target generated digital human style semantic encoding vector and the face semantic distillation feature vector of the object to be modeled are input into a multimodal implicit association feature capture network to obtain the face-style semantic implicit common feature vector of the object to be modeled as the face-style semantic implicit common feature of the object to be modeled, including: Cascading the target generated digital human style semantic encoding vector and the face semantic distillation feature vector of the object to be modeled to obtain a face-style semantic cascade feature vector of the object to be modeled; The face-style semantic cascade feature vector of the object to be modeled is multiplied by the linear transformation weight matrix, and then added to the linear transformation bias vector by position to obtain the linear transformation face-style semantic cascade feature vector of the object to be modeled; The linear transformation of the face-style semantic cascade feature vector of the object to be modeled is passed through The function is used to process the face-style semantic implicit common feature vector of the object to be modeled.

7. The method for generating a digital human according to claim 6, characterized in that: Performing a preliminary feature screening on the implicit common features of the face-style semantics of the object to be modeled to obtain the preliminary screened face-style semantic cross-modal joint encoding features of the object to be modeled, including: Inputting the implicit common feature vector of the face-style semantics of the object to be modeled into a feature preliminary screening module based on a gate control unit to obtain a primary face-style semantic feature preliminary screening weight vector of the object to be modeled; Based on the preliminary screening weight vector of the face-style semantic features of the first-level object to be modeled, the face feature map of the object to be modeled is subjected to preliminary feature screening to obtain a preliminary screened face-style semantic cross-modal joint coding feature map of the object to be modeled as the preliminary screened face-style semantic cross-modal joint coding feature of the object to be modeled.

8. The method for generating a digital human according to claim 7, characterized in that: The face-style semantic cross-modal joint encoding features of the initially screened object to be modeled are subjected to secondary feature screening to obtain the face-style semantic cross-modal significant fusion features of the object to be modeled, including: Inputting the implicit common feature vector of the face-style semantics of the object to be modeled into a secondary feature screening module based on a multi-level gating unit to obtain a secondary face-style semantic feature screening weight vector of the object to be modeled; Based on the secondary face-style semantic feature screening weight vector of the object to be modeled, the primary screened face-style semantic cross-modal joint encoding feature map of the object to be modeled is subjected to secondary feature screening to obtain the face-style semantic cross-modal significant fusion feature map of the object to be modeled as the face-style semantic cross-modal significant fusion feature of the object to be modeled.

9. The method for generating a digital human according to claim 8, characterized in that: A digital human portrait is generated based on the face-style semantic cross-modal significant fusion feature of the object to be modeled to obtain the digital human portrait, including: inputting the face-style semantic cross-modal significant fusion feature map of the object to be modeled into a digital human portrait generator based on a diffusion model to obtain the digital human portrait.

10. A digital human generation system, characterized in that: include: A face image acquisition module for the object to be modeled, used to acquire the face image of the object to be modeled; A style text description acquisition module is used to acquire the style text description of the target generated digital human input by the user; A face feature extraction module, used for extracting face features from the face image of the object to be modeled to obtain face features of the object to be modeled; A target generated digital human style semantic encoding module is used to perform semantic encoding on the style text description of the target generated digital human to obtain the target generated digital human style semantic features; A cross-modal optimal salient fusion module, used for performing cross-modal optimal salient fusion on the face features of the object to be modeled and the style semantic features of the target generated digital human to obtain cross-modal salient fusion features of the face and style semantics of the object to be modeled; The digital human portrait generation module is used to generate a digital human portrait based on the face-style semantic cross-modal significant fusion features of the object to be modeled to obtain a digital human portrait.

Images