Method and system for automatically generating digital character

Abstract

The invention discloses a method and system for automatically generating a digital figure, and relates to the field of digital figure intelligent generation. The method comprises the following steps: obtaining a face image of a to-be-modeled object, and obtaining a digital figure expected style text description input by a user; performing multi-scale feature analysis on the face image of the to-be-modeled object by adopting an artificial intelligence-based image processing and text analysis technology, wherein the digital human expects semantic coding of style text description; therefore, the three-dimensional model of the digital person object is intelligently generated according to the information obtained by cross-modal combination between the shallow and deep fusion features of the to-be-modeled object image and the expected semantic features of the digital person, and the generation of the digital person can be guided through text description input by a user. According to the method, each digital figure can better reflect the personalized demand and style preference of a user, meanwhile, the dependence on manual design and the limitation of traditional template design are reduced, more flexible design options are provided, and intelligent generation of the digital figure is achieved.

Description

Technical Field

[0001] The present application relates to the field of intelligent generation of digital characters, and more specifically, to a method and system for automatically generating digital characters. Background Art

[0002] Digital characters refer to virtual characters created through computer technology and software tools. These virtual characters can be completely fictional characters or simulated images based on the appearance, movements and behaviors of real people. The application areas of digital characters are very wide, from entertainment to education, to business and social services, digital characters are playing an important role.

[0003] However, traditional digital character generation methods rely on manual modeling by designers, a process that often requires deep artistic skills and proficiency in software tools. This not only means that each step from concept design to final rendering requires a lot of time and effort, but also the fine adjustment of details is extremely time-consuming. This manual production method leads to long production cycles, high costs, and difficulty in large-scale application.

[0004] In addition, traditional methods tend to use templated designs, which are difficult to meet the personalized needs of users. Specifically, templated design means that designers usually use pre-set models and styles, which can be generated quickly but lack flexibility and diversity. When users put forward specific personalized requirements, traditional templated designs often cannot fully meet these needs, thereby reducing user satisfaction.

[0005] Therefore, a solution for automatically generating digital characters is needed to solve the above technical problems. Summary of the invention

[0006] In order to solve the above technical problems, the present application is proposed. The embodiment of the present application provides a method and system for automatically generating digital characters, which obtains the facial image of the object to be modeled, obtains the text description of the expected style of the digital person input by the user, and uses the image processing and text analysis technology based on artificial intelligence to perform multi-scale feature analysis of the facial image of the object to be modeled, and the semantic encoding of the text description of the expected style of the digital person, so as to intelligently generate a three-dimensional model of the digital human object based on the information obtained by cross-modal combination between the shallow and deep fusion features of the image of the object to be modeled and the expected semantic features of the digital person, and can guide the generation of digital characters through the text description input by the user, so that each digital character can better reflect the personalized needs and style preferences of the user. At the same time, it reduces the dependence on professional artists and designers, breaks the limitations of traditional template design, provides more flexible design options, and realizes the intelligent generation of digital characters.

[0007] According to one aspect of the present application, a method for automatically generating a digital character is provided, which includes: Obtain a face image of the object to be modeled; Obtain the text description of the expected style of the digital person input by the user; Performing a multi-scale facial feature scan on the face image of the object to be modeled to obtain a shallow feature map of the face of the object to be modeled and a deep feature map of the face of the object to be modeled; Combine the shallow feature map of the face of the object to be modeled and the deep feature map of the face of the object to be modeled with the shallow-deep fine-grained global features of the face of the object to be modeled to obtain the shallow-deep fusion features of the face of the object to be modeled; Performing cross-modal semantic joint encoding on the expected style text description of the digital human and the shallow-deep fusion features of the face of the object to be modeled to obtain the face features of the object to be modeled with style semantic constraints; Based on the style semantics and the facial features of the object to be modeled, a three-dimensional model of the digital human object is obtained.

[0008] According to another aspect of the present application, a system for automatically generating a digital character is provided, comprising: An image acquisition module is used to acquire a face image of an object to be modeled; A text description acquisition module, used to acquire a text description of the expected style of the digital human input by the user; A face multi-scale feature scanning module, used for performing face multi-scale feature scanning on the face image of the object to be modeled to obtain a shallow feature map of the face of the object to be modeled and a deep feature map of the face of the object to be modeled; A fine-grained global feature combination module is used to combine the shallow feature map of the face of the object to be modeled and the deep feature map of the face of the object to be modeled to obtain a shallow-deep fusion feature of the face of the object to be modeled; A cross-modal semantic joint encoding module, used for performing cross-modal semantic joint encoding on the text description of the expected style of the digital human and the shallow-deep fusion features of the face of the object to be modeled to obtain the face features of the object to be modeled with style semantic constraints; The three-dimensional model generation module is used to obtain a three-dimensional model of a digital human object based on the facial features of the object to be modeled according to the style semantic constraints.

[0009] The present application has at least the following technical effects: Compared with the prior art, the present application provides a method and system for automatically generating digital characters, which obtains the face image of the object to be modeled, obtains the text description of the expected style of the digital person input by the user, and uses the image processing and text analysis technology based on artificial intelligence to perform multi-scale feature analysis of the face image of the object to be modeled, and the semantic encoding of the text description of the expected style of the digital person, so as to intelligently generate a three-dimensional model of the digital human object based on the information obtained by cross-modal combination between the shallow and deep fusion features of the image of the object to be modeled and the expected semantic features of the digital person, and can guide the generation of digital characters through the text description input by the user, so that each digital character can better reflect the personalized needs and style preferences of the user. At the same time, it reduces the dependence on professional artists and designers, breaks the limitations of traditional template design, provides more flexible design options, and realizes the intelligent generation of digital characters. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] By describing the embodiments of the present application in more detail in conjunction with the accompanying drawings, the above and other purposes, features and advantages of the present application will become more apparent. The accompanying drawings are used to provide a further understanding of the embodiments of the present application and constitute a part of the specification. Together with the embodiments of the present application, they are used to explain the present application and do not constitute a limitation of the present application. In the accompanying drawings, the same reference numerals generally represent the same components or steps.

[0011] Figure 1 A flowchart of a method for automatically generating a digital character according to an embodiment of the present application; Figure 2 A data flow diagram of a method for automatically generating a digital character according to an embodiment of the present application; Figure 3 is a flowchart of sub-step S4 of the method for automatically generating a digital character according to an embodiment of the present application; Figure 4 The block diagram is a system for automatically generating a digital character according to an embodiment of the present application. DETAILED DESCRIPTION

[0012] Below, the exemplary embodiments according to the present application will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments of the present application, and it should be understood that the present application is not limited to the exemplary embodiments described here.

[0013] As shown in this application and claims, unless the context clearly indicates an exception, the words "a", "an", "an" and / or "the" do not refer to the singular and may also include the plural. Generally speaking, the terms "include" and "comprise" only indicate the inclusion of the steps and elements that have been clearly identified, and these steps and elements do not constitute an exclusive list. The method or device may also include other steps or elements.

[0014] Although the present application makes various references to certain modules in the system according to the embodiments of the present application, any number of different modules can be used and run on the user terminal and / or server. The modules are only illustrative, and different aspects of the system and method can use different modules.

[0015] Flowcharts are used in the present application to illustrate the operations performed by the system according to the embodiments of the present application. It should be understood that the preceding or following operations are not necessarily performed accurately in order. On the contrary, various steps may be processed in reverse order or simultaneously as required. Meanwhile, other operations may also be added to these processes, or a certain step or several steps of operations may be removed from these processes.

[0016] Below, the exemplary embodiments according to the present application will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments of the present application, and it should be understood that the present application is not limited to the exemplary embodiments described here.

[0017] In the technical solution of the present application, a method for automatically generating a digital character is proposed. Figure 1 The figure is a flowchart of a method for automatically generating a digital character according to an embodiment of the present application. Figure 2 Schematic diagram of data flow of the method for automatically generating a digital character according to an embodiment of the present application. Figure 1 and Figure 2 As shown, according to the embodiment of the present application, the method for automatically generating a digital human comprises the following steps: S1, obtaining a facial image of an object to be modeled; S2, obtaining a text description of the expected style of the digital human input by a user; S3, performing a multi-scale facial feature scan on the facial image of the object to be modeled to obtain a shallow feature map of the face of the object to be modeled and a deep feature map of the face of the object to be modeled; S4, performing a shallow-deep fine-grained global feature combination of the shallow feature map of the face of the object to be modeled and the deep feature map of the face of the object to be modeled to obtain a shallow-deep fusion feature of the face of the object to be modeled; S5, performing a cross-modal semantic joint encoding on the text description of the expected style of the digital human and the shallow-deep fusion feature of the face of the object to be modeled to obtain the facial features of the object to be modeled with style semantic constraints; S6, obtaining a three-dimensional model of the digital human object based on the facial features of the object to be modeled with style semantic constraints.

[0018] In particular, the S1 and S2 obtain a facial image of the object to be modeled; and obtain a text description of the expected style of the digital human input by the user. Specifically, the facial image of the object to be modeled can provide the basic shape and features of the three-dimensional model, and the text description of the expected style of the digital human input by the user can guide the model to generate a three-dimensional model that meets the user's style preferences. For example, the user can choose styles such as "natural", "cartoon" or "realistic". In the technical solution of the present application, a three-dimensional model of the digital human object is intelligently generated by fusing the shallow and deep fusion features of the image of the object to be modeled and the expected semantic features of the digital human. In this way, the generation of digital characters can be guided by the text description input by the user, so that each digital character can better reflect the user's personalized needs and style preferences. At the same time, it reduces the dependence on professional artists and designers, breaks the limitations of traditional template design, provides more flexible design options, and realizes the intelligent generation of digital characters.

[0019] In particular, the S3 performs a multi-scale facial feature scan on the face image of the object to be modeled to obtain a shallow feature map of the face of the object to be modeled and a deep feature map of the face of the object to be modeled. In a specific example of the present application, the face image of the object to be modeled is input into a multi-scale facial feature scanner based on a hollow pyramid network model to obtain a shallow feature map of the face of the object to be modeled and a deep feature map of the face of the object to be modeled. Considering that the face image of the object to be modeled contains feature information of different scales, such as shallow edge information, color, etc.; deep facial structure and other more complex and abstract features. The hollow pyramid network model allows the network to perceive details of different scales without increasing the number of parameters or the amount of calculation, and can better understand the structural features of different scales in the image. Based on this, in order to capture and mine facial features of different scales, so as to better understand and model facial features, in the technical solution of the present application, the face image of the object to be modeled is input into a multi-scale facial feature scanner based on a hollow pyramid network model to obtain a shallow feature map of the face of the object to be modeled and a deep feature map of the face of the object to be modeled. It is worth mentioning that the Dilated Pyramid Network is a convolutional neural network model that uses dilated convolutions to expand the receptive field while maintaining spatial resolution. Specifically, the Dilated Pyramid Network consists of a series of convolutional layers, each of which uses dilated convolutions with different dilation rates. The dilation rate refers to the spacing between zero-filled elements in the convolution kernel. A larger dilation rate results in a larger receptive field.

[0020] In particular, the S4 combines the shallow feature map of the face of the object to be modeled and the deep feature map of the face of the object to be modeled with the shallow-deep fine-grained global features of the face of the object to be modeled to obtain the shallow-deep fusion features of the face of the object to be modeled. Considering that the shallow feature map of the face of the object to be modeled and the deep feature map of the face of the object to be modeled respectively represent the features of the face of the object to be modeled at different scales, that is, the shallow features provide basic details to help capture the physical characteristics of the face; while the deep features provide a higher level of abstraction, which helps to identify and distinguish different individuals. And there is a mutual influence correlation relationship between different local features. Therefore, in the technical solution of the present application, the shallow feature map of the face of the object to be modeled and the deep feature map of the face of the object to be modeled are subjected to feature fine-grained decomposition to obtain a set of local feature matrices of the shallow channel dimension of the face of the object to be modeled and a set of local feature matrices of the deep channel dimension of the face of the object to be modeled as the local features of the shallow channel dimension of the face of the object to be modeled and the local features of the deep channel dimension of the face of the object to be modeled. In particular, the shallow-deep fine-grained global feature combination of the face of the object to be modeled is an advanced feature fusion technology, the core goal of which is to improve the overall perception level of the model for semantic information through fine-grained feature decomposition, fine-grained bidirectional interaction based on the attention mechanism, and global interaction of features. In a specific example of the present application, Figure 3 As shown, the S4 includes: S41, performing fine-grained feature decomposition on the shallow feature map of the face of the object to be modeled and the deep feature map of the face of the object to be modeled to obtain shallow channel dimension local features of the face of the object to be modeled and deep channel dimension local features of the face of the object to be modeled; S42, performing bidirectional attention fusion on the shallow channel dimension local features of the face of the object to be modeled and the deep channel dimension local features of the face of the object to be modeled to obtain shallow-deep fusion features of the face of the object to be modeled.

[0021] Specifically, the S41 performs fine-grained feature decomposition on the shallow feature map of the face of the object to be modeled and the deep feature map of the face of the object to be modeled to obtain local features of the shallow channel dimension of the face of the object to be modeled and local features of the deep channel dimension of the face of the object to be modeled. In a specific example of the present application, the shallow feature map of the face of the object to be modeled and the deep feature map of the face of the object to be modeled are subjected to fine-grained feature decomposition to obtain a set of local feature matrices of the shallow channel dimension of the face of the object to be modeled and a set of local feature matrices of the deep channel dimension of the face of the object to be modeled as the local features of the shallow channel dimension of the face of the object to be modeled and the local features of the deep channel dimension of the face of the object to be modeled. That is, the fine-grained feature decomposition is performed along the channel dimension of the feature map to divide each feature map into smaller local feature matrices, which can better capture local details in the image, such as slight changes in facial features such as eyes, nose, mouth, etc., and provide input for subsequent attention interaction. Next, considering the mutual correlation and interaction between each shallow feature and the deep feature, the shallow features can focus on the important information of the deep features, and the deep features can also obtain useful information from the shallow features.

[0022] Specifically, the S42 performs bidirectional attention fusion on the local features of the shallow channel dimension of the face of the object to be modeled and the local features of the deep channel dimension of the face of the object to be modeled to obtain the shallow-deep fusion features of the face of the object to be modeled. In an embodiment of the present application, first, each local feature matrix of the shallow channel dimension of the face of the object to be modeled in the set of local feature matrices of the shallow channel dimension of the face of the object to be modeled is used as a query feature matrix, and the set of local feature matrices of the deep channel dimension of the face of the object to be modeled is used as a set of key feature matrices, and the query feature matrix and the set of key feature matrices are input into a one-way global attention interaction module based on a first converter structure to obtain a set of local feature matrices of the shallow channel dimension of the face of the object to be modeled that are optimized by one-way global attention; and each local feature matrix of the deep channel dimension of the face of the object to be modeled in the set of local feature matrices of the deep channel dimension of the face of the object to be modeled is used as a query feature matrix, and the set of local feature matrices of the shallow channel dimension of the face of the object to be modeled is used as a set of key feature matrices, and the set of query feature matrices and the set of key feature matrices are input into a one-way global attention interaction module based on a second converter structure to obtain a set of local feature matrices of the deep channel dimension of the face of the object to be modeled that are optimized by one-way global attention. Specifically, after the feature map is fine-grainedly decomposed, a unidirectional global attention module based on the Transformer architecture is used, and the relationship between different feature matrices is measured through a multi-attention head mechanism to strengthen the interaction between them, so that the model can focus more on those areas with dense and critical feature information. Specifically, in the present application, each local feature matrix of the shallow feature map of the face of the object to be modeled can have in-depth semantic interaction with all local feature matrices of the deep feature map of the face of the object to be modeled in a single direction, so as to more comprehensively capture the semantic association between each local shallow feature matrix and the entire deep feature map of the face of the object to be modeled. Similarly, each local feature matrix in the deep feature map of the face of the object to be modeled can also be globally semantically associated with all local matrices of the shallow feature map of the face of the object to be modeled. Therefore, through this two-way comprehensive semantic communication, it can be ensured that the relationship between the two feature maps is fully utilized during the analysis process.Next, the set of local feature matrices of the shallow channel dimension of the face of the object to be modeled optimized by one-way global attention and the set of local feature matrices of the deep channel dimension of the face of the object to be modeled optimized by one-way global attention are coupled along the channel dimension to obtain a shallow feature map of the face of the object to be modeled optimized by one-way global interaction and a deep feature map of the face of the object to be modeled optimized by one-way global interaction; here, the set of local feature matrices of the shallow channel dimension of the face of the object to be modeled optimized by one-way global attention and the set of local feature matrices of the deep channel dimension of the face of the object to be modeled optimized by one-way global attention are coupled along the channel dimension to more comprehensively and integratedly integrate the overall semantic expressions of the shallow and deep layers, so as to obtain a shallow feature map of the face of the object to be modeled optimized by one-way global interaction and a deep feature map of the face of the object to be modeled optimized by one-way global interaction. Subsequently, the position-weighted sum between the shallow feature map of the face of the object to be modeled by the one-way global interaction optimization and the deep feature map of the face of the object to be modeled by the one-way global interaction optimization is calculated to obtain a shallow-deep fine-grained fusion feature map of the face of the object to be modeled as the shallow-deep fusion feature of the face of the object to be modeled; specifically, in the example of the present application, after the shallow feature map of the face of the object to be modeled by the one-way global interaction optimization and the deep feature map of the face of the object to be modeled by the one-way global interaction optimization are multiplied by the first weighted hyperparameter and the second weighted hyperparameter respectively by position, the obtained shallow weighted feature map of the face of the object to be modeled by the one-way global interaction optimization and the deep weighted feature map of the face of the object to be modeled by the one-way global interaction optimization are added by position to obtain the shallow-deep fine-grained fusion feature map of the face of the object to be modeled. In this way, the model not only captures the basic features of the image (such as edges and textures), but also includes more advanced abstract features (such as facial structure and expressions), thereby ensuring that the final feature map is more focused on key information and improving the accuracy of subsequent character model production. Among them, each local feature matrix of the shallow channel dimension of the face of the object to be modeled in the set of local feature matrices of the shallow channel dimension of the face of the object to be modeled is used as a query feature matrix, and the set of local feature matrices of the deep channel dimension of the face of the object to be modeled is used as a set of key feature matrices, and the query feature matrix and the set of key feature matrices are input into a one-way global attention interaction module based on a first converter structure to obtain a set of local feature matrices of the shallow channel dimension of the face of the object to be modeled by one-way global attention optimization. The process includes: calculating the product of the local feature matrix of the shallow channel dimension of the face of the object to be modeled and the transposed matrix of each local feature matrix of the deep channel dimension of the face of the object to be modeled in the set of local feature matrices of the deep channel dimension of the face of the object to be modeled to obtain a set of shallow-deep correlation feature matrices of the face of the object to be modeled; calculating the local feature matrix of the shallow channel dimension of the face of the object to be modeled The shallow-deep correlation feature matrix of each face of the object to be modeled in the set of shallow-deep correlation feature matrices is divided by position between the square root of the scale of the shallow-deep correlation feature matrix of the face of the object to be modeled to obtain a set of shallow-deep correlation modulation feature matrices of the face of the object to be modeled; the set of shallow-deep correlation modulation feature matrices of the face of the object to be modeled is input into the softmax function for activation processing, and the obtained set of shallow-deep correlation activation feature matrices of the face of the object to be modeled is matrix-multiplied with the local feature matrix of the shallow channel dimension of the face of the object to be modeled to obtain a set of shallow-deep correlation attention interaction feature matrices of the face of the object to be modeled; the set of shallow-deep correlation attention interaction feature matrices of the face of the object to be modeled is multiplied by position point to obtain the unidirectional global attention optimization local feature matrix of the shallow channel dimension of the face of the object to be modeled.The process of respectively using each local feature matrix of the deep channel dimension of the face of the object to be modeled in the set of local feature matrices of the deep channel dimension of the face of the object to be modeled as a query feature matrix, and using the set of local feature matrices of the shallow channel dimension of the face of the object to be modeled as a set of key feature matrices, and inputting the query feature matrix and the set of key feature matrices into a one-way global attention interaction module based on a second converter structure to obtain a set of local feature matrices of the deep channel dimension of the face of the object to be modeled that are optimized by one-way global attention includes: performing matrix multiplication on the local feature matrix of the deep channel dimension of the face of the object to be modeled and the transposed matrix of each local feature matrix of the shallow channel dimension of the face of the object to be modeled in the set of local feature matrices of the shallow channel dimension of the face of the object to be modeled to obtain a set of deep-shallow correlation feature matrices of the face of the object to be modeled; calculating the deep channel dimension of the face of the object to be modeled; and obtaining a set of deep-shallow correlation feature matrices of the face of the object to be modeled. The method comprises the following steps: performing positional division between each deep-shallow correlation feature matrix of the face of the object to be modeled and the square root of the scale of the deep-shallow correlation feature matrix of the face of the object to be modeled in the set of deep-shallow correlation feature matrices to obtain a set of deep-shallow correlation modulation feature matrices of the face of the object to be modeled; inputting the set of deep-shallow correlation modulation feature matrices of the face of the object to be modeled into the softmax function for activation processing, and performing matrix multiplication of the set of deep-shallow correlation activation feature matrices of the face of the object to be modeled with the local feature matrix of the deep channel dimension of the face of the object to be modeled to obtain a set of deep-shallow correlation attention interaction feature matrices of the face of the object to be modeled; performing positional multiplication on the set of deep-shallow correlation attention interaction feature matrices of the face of the object to be modeled to obtain the unidirectional global attention optimization local feature matrix of the deep channel dimension of the face of the object to be modeled.

[0023] In summary, in the above embodiment, the shallow feature map of the face of the object to be modeled and the deep feature map of the face of the object to be modeled are combined with the shallow-deep fine-grained global features of the face of the object to be modeled to obtain the shallow-deep fusion features of the face of the object to be modeled, including: combining the shallow feature map of the face of the object to be modeled and the deep feature map of the face of the object to be modeled with the following formula to obtain the shallow-deep fine-grained fusion feature map of the face of the object to be modeled; wherein the formula is: in, and are respectively a shallow feature map of the face of the object to be modeled and a deep feature map of the face of the object to be modeled, It is a feature fine-grained decomposition operation. , , ..., , ..., are respectively the first, second, ..., and third local feature matrices of the shallow channel dimension of the face of the object to be modeled. ,..., The local feature matrix of the shallow channel dimension of the face of the object to be modeled, , , ..., , ..., are respectively the first, second, ..., and third local feature matrices of the deep channel dimension of the face of the object to be modeled. ,..., The local feature matrix of the deep channel dimension of the face of the object to be modeled, is the number of feature matrices in the set of local feature matrices of the shallow channel dimension of the face of the object to be modeled and the set of local feature matrices of the deep channel dimension of the face of the object to be modeled, and They are and The transposed matrix of represents matrix multiplication, and are the scales of the local feature matrix of the shallow channel dimension of the face of the object to be modeled and the local feature matrix of the deep channel dimension of the face of the object to be modeled, is the softmax function, The first local feature matrix in the set of shallow channel dimensions of the face of the object to be modeled is optimized for unidirectional global attention. A one-way global attention optimizes the local feature matrix of the shallow channel dimension of the face of the object to be modeled, The first local feature matrix in the set of deep channel dimensions of the face of the object to be modeled is optimized for unidirectional global attention. A one-way global attention optimizes the local feature matrix of the deep channel dimension of the face of the object to be modeled, To couple the features of each feature matrix along the channel dimension, and They are respectively the shallow feature map of the face of the object to be modeled by the one-way global interaction optimization and the deep feature map of the face of the object to be modeled by the one-way global interaction optimization, and is the corresponding weighted hyperparameter, It is a shallow-deep fine-grained fusion feature map of the face of the object to be modeled.

[0024] Among them, in other specific examples of the present application, the shallow feature map of the face of the object to be modeled and the deep feature map of the face of the object to be modeled can also be combined with the shallow-deep fine-grained global features of the face of the object to be modeled in other ways to obtain the shallow-deep fusion features of the face of the object to be modeled, for example: input the shallow feature map of the face of the object to be modeled and the deep feature map of the face of the object to be modeled; ensure that the shallow feature map and the deep feature map have the same spatial resolution through upsampling; use global pooling operations (such as average pooling or maximum pooling) to extract global features from the shallow feature map and the deep feature map; splice the shallow global features and the deep global features together to obtain the shallow-deep fusion features of the face of the object to be modeled.

[0025] In particular, the S5 performs cross-modal semantic joint encoding on the text description of the digital human's expected style and the shallow-deep fusion features of the face of the object to be modeled to obtain the face features of the object to be modeled with style semantic constraints. In a specific example of the present application, first, the text description of the digital human's expected style is semantically encoded to obtain a semantic encoding feature vector of the digital human's expected style; considering that the text description of the digital human's expected style contains the semantics and intentions of the user's expected style, based on this, in the technical solution of the present application, the text description of the digital human's expected style is semantically encoded to capture and extract the relationship and contextual information between words in the text, and obtain a semantic encoding feature vector of the digital human's expected style, which helps to understand the true intention and meaning of the text description, and the capture model can better understand the style description input by the user, and generate a digital human model that meets the user's expectations accordingly. Furthermore, in order to utilize the shallow-deep fine-grained fusion feature map of the face of the object to be modeled to capture all important details of the face, and use the digital human expected style semantic encoding feature vector to ensure that the generated face model meets the user's expectations in style, in the technical solution of the present application, the shallow-deep fine-grained fusion feature map of the face of the object to be modeled and the digital human expected style semantic encoding feature vector are input into the cross-modal semantic joint encoder based on MetaNet to obtain the style semantic constraint face feature map of the object to be modeled as the style semantic constraint face feature of the object to be modeled. That is, MetaNet can effectively process data from different modalities (such as images and text). By inputting the fused image feature map and semantic encoding feature vector into the cross-modal semantic joint encoder, deep fusion between image features and text descriptions can be achieved to achieve effective fusion of cross-modal information, ensuring that the generated three-dimensional digital human model not only retains the features of the original image, but also meets the user's style expectations.

[0026] In particular, the S6 obtains a three-dimensional model of a digital human object based on the facial features of the object to be modeled according to the style semantic constraints. In a specific example of the present application, the facial feature map of the object to be modeled according to the style semantic constraints is input into a digital human generator based on a diffusion model to obtain a three-dimensional model of the digital human object. That is, the facial features of the object to be modeled according to the style semantic constraints obtained by cross-modal semantic union of the shallow-deep fine-grained fusion feature map of the face of the object to be modeled and the expected style semantic encoding feature vector of the digital human are generated and processed to intelligently generate a three-dimensional model of the digital human object, and the generation of digital characters can be guided by the text description input by the user, so that each digital character can better reflect the user's personalized needs and style preferences. At the same time, it reduces the dependence on professional artists and designers, breaks the limitations of traditional templated design, provides more flexible design options, and realizes the intelligent generation of digital characters.

[0027] That is, considering that the shallow-deep fine-grained fusion feature map of the face of the object to be modeled and the digital human expected style semantic encoding feature vector respectively represent the image multi-scale semantic joint encoding features of the face image of the object to be modeled and the text semantic features of the text description of the digital human expected style, when they are passed through the MetaNet-based cross-modal semantic joint encoder, the different dimensional association structures based on the cross-modal feature distribution will also have cross-modal fusion context association encoding differences. Therefore, it is expected to improve the detail semantic aggregation expression effect of the face feature map of the object to be modeled based on the multi-modal semantic association representation differences of the style semantic constraints.

[0028] In a preferred example, inputting the face feature map of the object to be modeled with style semantic constraints into a digital human generator based on a diffusion model to obtain a three-dimensional model of the digital human object includes: Expanding the face feature map of the object to be modeled with style semantic constraints into a face feature vector of the object to be modeled with style semantic constraints; Performing L random sampling on the facial feature vector of the object to be modeled with the style semantic constraint to obtain L randomly disturbed feature values ​​of the facial features of the object to be modeled; Arrange the L randomly perturbed eigenvalues ​​of the facial features of the object to be modeled in order from large to small to obtain a randomly perturbed ordered vector of the facial features of the object to be modeled; Calculating the inner product of the facial feature vector of the object to be modeled with the style semantic constraint to obtain the facial feature energy modulation feature value of the object to be modeled; Based on the energy modulation characteristic value of the facial feature of the object to be modeled, the randomly disturbed ordered vector of the facial feature of the object to be modeled is semantically scaled to obtain a semantically enhanced randomly disturbed ordered vector of the facial feature of the object to be modeled; Calculating the product of the randomly disturbed ordered vector of the semantically enhanced object's facial features to be modeled and the transposed vector of the style semantically constrained object's facial features vector to be modeled to obtain an optimal sample prompt orthogonal modulation matrix of the object's facial features to be modeled; Calculating the matrix product between the optimal sample hint orthogonal modulation matrix of the facial features of the object to be modeled and the facial feature vector of the object to be modeled with style semantic constraints to obtain an optimized facial feature vector of the object to be modeled with style semantic constraints; The optimized face feature vector of the object to be modeled with style semantic constraints is input into the diffusion model-based digital human generator to obtain a three-dimensional model of the digital human object.

[0029] The above optimization process is expressed as follows: in, represents a normal distribution, represents the Gaussian density probability function, and They represent the mean and variance of the facial feature vector of the object to be modeled under the style semantic constraint, represents a random variable that follows a normal distribution, represents the number of random samplings, Indicates The randomly perturbed eigenvalues ​​of the facial features of the object to be modeled, represents the face feature vector of the object to be modeled under the style semantic constraint, The face feature vector of the object to be modeled under the style semantic constraint is represented by eigenvalues, It means to calculate the average value of the vector. represents the length of the face feature vector of the object to be modeled under the style semantic constraint, It means that the randomly disturbed eigenvalues ​​of the facial features of the modeling object are arranged in order from large to small. represents the randomly disturbed ordered vector of the facial features of the object to be modeled, represents the energy modulation eigenvalue of the facial feature of the object to be modeled, represents the square of the vector's two-norm, It means point multiplication by position. represents matrix multiplication, represents the transpose of a vector, The best sample prompt orthogonal modulation matrix representing the facial features of the object to be modeled, Represents the optimized style semantic constraint face feature vector of the object to be modeled.

[0030] Accordingly, the face feature vector of the object to be modeled with style semantic constraints obtained by expanding the face features of the object to be modeled with style semantic constraints is randomly sampled L times to obtain a set of new feature values ​​based on the original features but containing slight changes, so as to introduce randomness to explore different aspects of the feature space, and by simulating the noise or variation in the actual data, it helps the model to better generalize to unseen data. Then, the randomly perturbed feature values ​​of the face features of the object to be modeled are arranged in order from large to small to form an ordered vector of randomly perturbed face features of the object to be modeled, so as to identify which perturbation directions may be the most important for the current problem. Such an ordering not only helps the algorithm focus on the feature differences that can best distinguish different categories, but also helps to reveal hidden data structures or patterns. Subsequently, the self-inner product of the facial feature vector of the object to be modeled with style semantic constraints is calculated to obtain the energy modulation eigenvalue of the facial feature of the object to be modeled, which reflects the overall strength or "energy" of the original feature vector and provides a benchmark for measuring the importance of the feature. Using the energy modulation eigenvalue of the facial feature of the object to be modeled as a reference point, the previously formed randomly perturbed ordered vector of the facial features of the object to be modeled is semantically scaled to construct a semantically enhanced randomly perturbed ordered vector of the facial features of the object to be modeled. In this way, those disturbances that appear to be more important in the original feature space are amplified, thereby enhancing the model's sensitivity to changes in key features, while also retaining relatively small but potentially useful disturbance information.

[0031] Then, the product of the randomly disturbed ordered vector of the facial features of the semantically enhanced object to be modeled and the transposed vector of the facial features of the object to be modeled with style semantic constraints is calculated to obtain the orthogonal modulation matrix of the best sample hint of the facial features of the object to be modeled. The purpose is to find the way in which the features interact with each other under two different perspectives. The formed matrix can be regarded as a conversion tool for capturing and representing the complex linear relationship between features. Finally, the final feature structure modulation is completed by applying the best sample hint orthogonal modulation matrix of the facial features of the object to be modeled constructed above to the original facial feature vector of the object to be modeled with style semantic constraints, so as to make the features more in line with the requirements of the objective function, reduce redundant information, and maximize the retention of useful information, so as to improve learning efficiency and prediction performance.

[0032] In summary, the feature manifold optimization process of the face feature map of the object to be modeled with style semantic constraints is explained. It extracts the best hint sample based on the fine-grained target retrieval of random perturbations, and then guides the adversarial robustness learning based on the best hint sample to complete the structure of the high-dimensional manifold of the feature vector in the feature space, so as to improve the detail and structure of the feature expression, thereby improving the perception effect of the aggregated semantics of the face feature map of the object to be modeled with style semantic constraints on the change of detailed semantics, so as to improve the expression effect of the face feature map of the object to be modeled with style semantic constraints, and improve the model quality of the three-dimensional model of the digital human object obtained by the digital human generator based on the diffusion model. The generation of digital characters can be guided by the text description input by the user, so that each digital character can better reflect the personalized needs and style preferences of the user. At the same time, it reduces the dependence on professional artists and designers, breaks the limitations of traditional template design, provides more flexible design options, and realizes the intelligent generation of digital characters.

[0033] In summary, the method for automatically generating a digital person according to the embodiment of the present application is explained, which obtains the face image of the object to be modeled, obtains the text description of the expected style of the digital person input by the user, and uses the image processing and text analysis technology based on artificial intelligence to perform multi-scale feature analysis of the face image of the object to be modeled, and the semantic encoding of the text description of the expected style of the digital person, so as to intelligently generate a three-dimensional model of the digital person object based on the information obtained by cross-modal combination between the shallow and deep fusion features of the image of the object to be modeled and the expected semantic features of the digital person. In this way, the generation of digital characters can be guided by the text description input by the user, so that each digital character can better reflect the personalized needs and style preferences of the user. At the same time, it reduces the dependence on professional artists and designers, breaks the limitations of traditional template design, provides more flexible design options, and realizes the intelligent generation of digital characters.

[0034] Furthermore, a system for automatically generating digital characters is also provided.

[0035] Figure 4 FIG. 1 is a block diagram of a system for automatically generating a digital character according to an embodiment of the present application. Figure 4As shown, according to the embodiment of the present application, the automatic generation of digital characters system 300 includes: an image acquisition module 310, which is used to acquire a face image of a to-be-modeled object; a text description acquisition module 320, which is used to acquire a text description of the expected style of a digital person input by a user; a face multi-scale feature scanning module 330, which is used to perform face multi-scale feature scanning on the face image of the to-be-modeled object to obtain a shallow feature map of the face of the to-be-modeled object and a deep feature map of the face of the to-be-modeled object; a fine-grained global feature combination module 340, which is used to combine the shallow feature map of the face of the to-be-modeled object with the deep feature map of the face of the to-be-modeled object; The feature map and the deep feature map of the face of the object to be modeled are used to perform a shallow-deep fine-grained global feature combination of the face of the object to be modeled to obtain a shallow-deep fusion feature of the face of the object to be modeled; a cross-modal semantic joint encoding module 350 is used to perform a cross-modal semantic joint encoding of the expected style text description of the digital human and the shallow-deep fusion feature of the face of the object to be modeled to obtain the face features of the object to be modeled with style semantic constraints; a three-dimensional model generation module 360 ​​is used to obtain a three-dimensional model of the digital human object based on the face features of the object to be modeled with style semantic constraints.

[0036] As described above, the automatic digital character generation system 300 according to the embodiment of the present application can be implemented in various wireless terminals, such as a server with an automatic digital character generation algorithm. In a possible implementation, the automatic digital character generation system 300 according to the embodiment of the present application can be integrated into the wireless terminal as a software module and / or a hardware module. For example, the automatic digital character generation system 300 can be a software module in the operating system of the wireless terminal, or it can be an application developed for the wireless terminal; of course, the automatic digital character generation system 300 can also be one of the many hardware modules of the wireless terminal.

[0037] Alternatively, in another example, the automatic digital character generation system 300 and the wireless terminal may be separate devices, and the automatic digital character generation system 300 may be connected to the wireless terminal via a wired and / or wireless network and transmit interactive information in a predetermined data format.

[0038] The embodiments of the present disclosure have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and changes will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The selection of terms used herein is intended to best explain the principles of the embodiments, practical applications, or improvements to the technology in the market, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein.

Claims

1. A method for automatically generating a digital character, characterized in that: include: Obtain a face image of the object to be modeled; Obtain the text description of the expected style of the digital person input by the user; Performing a multi-scale facial feature scan on the facial image of the object to be modeled to obtain a shallow feature map of the face of the object to be modeled and a deep feature map of the face of the object to be modeled; Combine the shallow feature map of the face of the object to be modeled and the deep feature map of the face of the object to be modeled with the shallow-deep fine-grained global features of the face of the object to be modeled to obtain the shallow-deep fusion features of the face of the object to be modeled; Among them, the shallow feature map of the face of the object to be modeled and the deep feature map of the face of the object to be modeled are combined with the shallow-deep fine-grained global features of the face of the object to be modeled to obtain the shallow-deep fusion features of the face of the object to be modeled, including: performing fine-grained feature decomposition on the shallow feature map of the face of the object to be modeled and the deep feature map of the face of the object to be modeled to obtain the shallow channel dimension local features of the face of the object to be modeled and the deep channel dimension local features of the face of the object to be modeled; performing bidirectional attention fusion on the shallow channel dimension local features of the face of the object to be modeled and the deep channel dimension local features of the face of the object to be modeled to obtain the shallow-deep fusion features of the face of the object to be modeled; Performing cross-modal semantic joint encoding on the expected style text description of the digital human and the shallow-deep fusion features of the face of the object to be modeled to obtain the face features of the object to be modeled with style semantic constraints; Based on the style semantics and the facial features of the object to be modeled, a three-dimensional model of the digital human object is obtained.

2. The method for automatically generating a digital character according to claim 1, characterized in that: The face image of the object to be modeled is subjected to a multi-scale face feature scanning to obtain a shallow face feature map of the object to be modeled and a deep face feature map of the object to be modeled, including: inputting the face image of the object to be modeled into a multi-scale face feature scanner based on a diaphanous pyramid network model to obtain the shallow face feature map of the object to be modeled and the deep face feature map of the object to be modeled.

3. The method for automatically generating a digital character according to claim 2, characterized in that: Performing fine-grained feature decomposition on the shallow feature map of the face of the object to be modeled and the deep feature map of the face of the object to be modeled to obtain shallow channel dimension local features of the face of the object to be modeled and deep channel dimension local features of the face of the object to be modeled, including: Perform fine-grained feature decomposition on the shallow feature map of the face of the object to be modeled and the deep feature map of the face of the object to be modeled to obtain a set of local feature matrices of the shallow channel dimension of the face of the object to be modeled and a set of local feature matrices of the deep channel dimension of the face of the object to be modeled as the local features of the shallow channel dimension of the face of the object to be modeled and the local features of the deep channel dimension of the face of the object to be modeled.

4. The method for automatically generating a digital character according to claim 3, characterized in that: Performing bidirectional attention fusion on the local features of the shallow channel dimension of the face of the object to be modeled and the local features of the deep channel dimension of the face of the object to be modeled to obtain the shallow-deep fusion features of the face of the object to be modeled, including: Using each local feature matrix of the shallow channel dimension of the face of the object to be modeled in the set of local feature matrices of the shallow channel dimension of the face of the object to be modeled as a query feature matrix, and using the set of local feature matrices of the deep channel dimension of the face of the object to be modeled as a set of key feature matrices, inputting the query feature matrix and the set of key feature matrices into a unidirectional global attention interaction module based on the first converter structure to obtain a set of local feature matrices of the shallow channel dimension of the face of the object to be modeled that are optimized by unidirectional global attention; Respectively use each local feature matrix of the deep channel dimension of the face of the object to be modeled in the set of local feature matrices of the deep channel dimension of the face of the object to be modeled as a query feature matrix, use the set of local feature matrices of the shallow channel dimension of the face of the object to be modeled as a set of key feature matrices, input the query feature matrix and the set of key feature matrices into a unidirectional global attention interaction module based on a second converter structure to obtain a set of local feature matrices of the deep channel dimension of the face of the object to be modeled that are optimized by unidirectional global attention; The set of local feature matrices of shallow channel dimension of the face of the object to be modeled by the one-way global attention optimization and the set of local feature matrices of deep channel dimension of the face of the object to be modeled by the one-way global attention optimization are coupled along the channel dimension to obtain a shallow feature map of the face of the object to be modeled by the one-way global interaction optimization and a deep feature map of the face of the object to be modeled by the one-way global interaction optimization; The position-weighted sum between the shallow feature map of the face of the object to be modeled by the one-way global interaction optimization and the deep feature map of the face of the object to be modeled by the one-way global interaction optimization is calculated to obtain a shallow-deep fine-grained fusion feature map of the face of the object to be modeled as the shallow-deep fusion feature of the face of the object to be modeled.

5. The method for automatically generating a digital character according to claim 4, characterized in that: The method comprises: taking each local feature matrix of the shallow channel dimension of the face of the object to be modeled in the set of local feature matrices of the shallow channel dimension of the face of the object to be modeled as a query feature matrix, taking the set of local feature matrices of the deep channel dimension of the face of the object to be modeled as a set of key feature matrices, inputting the query feature matrix and the set of key feature matrices into a unidirectional global attention interaction module based on a first converter structure to obtain a set of local feature matrices of the shallow channel dimension of the face of the object to be modeled optimized by unidirectional global attention, including: Calculate the product of the local feature matrix of the shallow channel dimension of the face of the object to be modeled and the transposed matrix of each local feature matrix of the deep channel dimension of the face of the object to be modeled in the set of local feature matrices of the deep channel dimension of the face of the object to be modeled to obtain a set of shallow-deep correlation feature matrices of the face of the object to be modeled; Calculating the positional division between each shallow-deep correlation feature matrix of the face of the object to be modeled in the set of shallow-deep correlation feature matrices of the face of the object to be modeled and the square root of the scale of the shallow-deep correlation feature matrix of the face of the object to be modeled to obtain a set of shallow-deep correlation modulation feature matrices of the face of the object to be modeled; Input the set of shallow-deep correlation modulation feature matrices of the face of the object to be modeled into the softmax function for activation processing, and perform matrix multiplication of the obtained set of shallow-deep correlation activation feature matrices of the face of the object to be modeled with the shallow channel dimension local feature matrix of the face of the object to be modeled to obtain the set of shallow-deep correlation attention interaction feature matrices of the face of the object to be modeled; The set of shallow-deep correlation attention interaction feature matrices of the face of the object to be modeled is processed by position point to obtain the unidirectional global attention optimization shallow channel dimension local feature matrix of the face of the object to be modeled.

6. The method for automatically generating a digital character according to claim 5, characterized in that: The method comprises: taking each local feature matrix of the deep channel dimension of the face of the object to be modeled in the set of local feature matrices of the deep channel dimension of the face of the object to be modeled as a query feature matrix, taking the set of local feature matrices of the shallow channel dimension of the face of the object to be modeled as a set of key feature matrices, inputting the query feature matrix and the set of key feature matrices into a unidirectional global attention interaction module based on a second converter structure to obtain a set of local feature matrices of the deep channel dimension of the face of the object to be modeled optimized by unidirectional global attention, including: Performing matrix multiplication of the local feature matrix of the deep channel dimension of the face of the object to be modeled and the transposed matrix of each local feature matrix of the shallow channel dimension of the face of the object to be modeled in the set of local feature matrices of the shallow channel dimension of the face of the object to be modeled to obtain a set of deep-shallow correlation feature matrices of the face of the object to be modeled; Calculating the positional division between each deep-shallow correlation feature matrix of the face of the object to be modeled in the set of deep-shallow correlation feature matrices of the face of the object to be modeled and the square root of the scale of the deep-shallow correlation feature matrix of the face of the object to be modeled to obtain a set of deep-shallow correlation modulation feature matrices of the face of the object to be modeled; Input the set of deep-shallow correlation modulation feature matrices of the face of the object to be modeled into the softmax function for activation processing, and perform matrix multiplication of the obtained set of deep-shallow correlation activation feature matrices of the face of the object to be modeled with the local feature matrix of the deep channel dimension of the face of the object to be modeled to obtain the set of deep-shallow correlation attention interaction feature matrices of the face of the object to be modeled; The set of deep-shallow correlation attention interaction feature matrices of the face of the object to be modeled is processed by position point to obtain the local feature matrix of the deep channel dimension of the face of the object to be modeled by unidirectional global attention optimization.

7. The method for automatically generating a digital character according to claim 6, characterized in that: The method comprises calculating the position-weighted sum between the shallow feature map of the face of the object to be modeled by the one-way global interaction optimization and the deep feature map of the face of the object to be modeled by the one-way global interaction optimization to obtain a shallow-deep fine-grained fusion feature map of the face of the object to be modeled, including: multiplying the shallow feature map of the face of the object to be modeled by the one-way global interaction optimization and the deep feature map of the face of the object to be modeled by the one-way global interaction optimization respectively with the first weighted hyperparameter and the second weighted hyperparameter by position, and then adding the obtained shallow weighted feature map of the face of the object to be modeled by the one-way global interaction optimization and the deep weighted feature map of the face of the object to be modeled by the one-way global interaction optimization to obtain the shallow-deep fine-grained fusion feature map of the face of the object to be modeled.

8. The method for automatically generating a digital character according to claim 7, characterized in that: Performing cross-modal semantic joint encoding on the expected style text description of the digital human and the shallow-deep fusion features of the face of the object to be modeled to obtain the face features of the object to be modeled with style semantic constraints, including: Performing semantic coding on the text description of the digital human's expected style to obtain a semantic coding feature vector of the digital human's expected style; The shallow-deep fine-grained fusion feature map of the face of the object to be modeled and the expected style semantic encoding feature vector of the digital human are input into the cross-modal semantic joint encoder based on MetaNet to obtain the face feature map of the object to be modeled with style semantic constraints as the face feature of the object to be modeled with style semantic constraints.

9. The method for automatically generating a digital character according to claim 8, characterized in that: Based on the facial features of the object to be modeled with style semantic constraints, a three-dimensional model of a digital human object is obtained, including: inputting the facial feature map of the object to be modeled with style semantic constraints into a digital human generator based on a diffusion model to obtain the three-dimensional model of the digital human object.

10. A system for automatically generating digital characters, characterized in that: include: An image acquisition module is used to acquire a face image of an object to be modeled; A text description acquisition module, used to acquire a text description of the expected style of the digital human input by the user; A face multi-scale feature scanning module, used for performing face multi-scale feature scanning on the face image of the object to be modeled to obtain a shallow feature map of the face of the object to be modeled and a deep feature map of the face of the object to be modeled; A fine-grained global feature combination module is used to combine the shallow feature map of the face of the object to be modeled and the deep feature map of the face of the object to be modeled to obtain a shallow-deep fusion feature of the face of the object to be modeled; A cross-modal semantic joint encoding module, used for performing cross-modal semantic joint encoding on the text description of the expected style of the digital human and the shallow-deep fusion features of the face of the object to be modeled to obtain the face features of the object to be modeled with style semantic constraints; The three-dimensional model generation module is used to obtain a three-dimensional model of a digital human object based on the facial features of the object to be modeled according to the style semantic constraints.

Images