Digital human video synthesis system

Through the digital human video synthesis system, 3D modeling and deep learning algorithms are used to generate natural digital human movements and expressions, and combined with multi-sensing touch screens, the problems of unnatural and insensitive digital human interaction in the existing technology are solved. Achieve efficient user interaction and operation experience.

CN119011752AInactive Publication Date: 2024-11-22ANHUI HAIXUAN EDUCATION TECH CO LTD
View PDF 7 Cites 0 Cited by

Patent Information

Application Number
CN202410964070.2
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-18
Publication Date
2024-11-22
Estimated Expiration
Not applicable · inactive patent

AI Technical Summary

Technical Problem

There are unnatural problems in the movement and expression generation of existing digital human video systems, and their interaction capabilities with users are relatively limited, especially the interaction of the touch screen is not accurate and flexible enough to respond to the user's operation.

Method used

The digital human video synthesis system is adopted, including model building module, speech synthesis module, expression generation module, rendering engine module and interaction control module. Through 3D modeling, deep learning algorithms, real-time rendering engine and multi-sensing touch screen, digital humans are realized. Rapid construction of models, natural movements and expression generation, and sensitive user interaction.

Benefits of technology

It realizes natural and smooth interaction between digital people and users, improves user experience and operational response sensitivity, and provides intelligent and flexible services.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119011752A_ABST
    Figure CN119011752A_ABST
Patent Text Reader

Abstract

The invention discloses a digital human video synthesis system, and relates to the technical field of intellectualization. Compared with a previous digital human video synthesis system, the digital human video synthesis system disclosed by the invention solves the problems that the action and expression generation of the existing digital human video system is still unnatural, and the interaction ability of the current digital human with the user is limited, especially the problems that the operation response to the user is not accurate and flexible enough due to the interaction of a touch screen; the character model can be quickly built according to a real character image, the long period of traditional video production is greatly shortened, real-time voice and character interaction with a user can be carried out in combination with a multi-mode interaction mode of voice, action, expression and the like, interaction between a digital person and the user is more natural and smooth, and the user experience is improved. The content and the expression mode are adjusted according to the feedback and the demand of the user, and more intelligent and flexible services are provided; meanwhile, the touch screen is optimized, the sensitivity of response to user operation is improved, and the user experience is improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of intelligent technology, and in particular to a digital human video synthesis system. Background Art

[0002] Digital human video synthesis systems have gradually emerged with the development of technologies such as computer graphics, artificial intelligence, computer vision, and natural language processing.

[0003] In the field of computer graphics, the continuous advancement of 3D modeling and animation technology has made it possible to create realistic digital human models. By using professional modeling software and rendering engines, it is possible to build digital human appearance and movements with high details and realism.

[0004] The development of artificial intelligence technology, especially deep learning technology, has provided strong support for the speech synthesis, expression generation and action drive of digital humans. For example, the speech synthesis model based on deep neural network can generate natural and fluent speech, while the image generation model based on generative adversarial network (GAN) and variational autoencoder (VAE) can be used to generate facial expressions and actions of digital humans.

[0005] Natural language processing technology enables digital humans to understand and process natural language text, thereby generating corresponding voices and actions based on the input text content. Computer vision technology can be used to capture the actions and expressions of real people and map them to digital human models to achieve more natural and realistic interaction effects.

[0006] Although great progress has been made in the appearance modeling and rendering technology of digital humans, in some cases, the movement and expression generation of digital humans still have unnatural problems, such as stiff movements, incoherent expressions, lack of subtle emotional expressions, etc. In addition, the current digital humans are still relatively limited in their ability to interact with users, especially the interaction with touch screens, which are not accurate and flexible enough to respond to user operations.

[0007] In order to solve the above problems, the present invention proposes a digital human video synthesis system. Summary of the invention

[0008] The purpose of the present invention is to provide a digital human video synthesis system to solve the problems raised in the background technology:

[0009] The existing digital human video system still has the problem of unnatural movement and expression generation, and the current digital human's ability to interact with users is still relatively limited, especially the interaction with the touch screen, which is not accurate and flexible enough to respond to user operations.

[0010] In order to achieve the above object, the present invention adopts the following technical solutions:

[0011] The digital human video synthesis system includes: 1. The digital human video synthesis system includes:

[0012] Model building module: used to model digital humans based on real people;

[0013] Speech synthesis module: used to convert the input text content into speech audio through the speech synthesis engine, and adjust and optimize the synthesized speech;

[0014] Expression generation module: used to generate digital human expressions based on real people’s facial expressions;

[0015] Rendering engine module: used to render digital human video;

[0016] Interaction control module: used for data interaction with users and third parties;

[0017] The model building module includes:

[0018] Character modeling unit: used to create appearance models based on 3D modeling technology, and to scan real people based on scanning technology to build digital human models;

[0019] Bone binding and animation unit: used to establish a bone system for the digital human model, determine the position and range of motion of the joints, and perform action recognition and animation production through bone binding;

[0020] The expression generation module comprises:

[0021] Expression Capture and Mapping Unit: It is used to capture the facial expression data of real people through facial capture equipment and map it to the digital human model to achieve real-time driving of expressions; it also uses deep learning algorithms to analyze the input voice or text content and automatically generate corresponding digital human expressions;

[0022] Expression library and animation curve unit: used to build expression library and control expression change curve, amplitude and transition effect by adjusting animation curve;

[0023] The rendering engine module includes:

[0024] Real-time rendering unit: used to render the digital human model using a real-time rendering engine;

[0025] Rendering output and optimization unit: used to output the rendered digital human video into video files of different formats and resolutions, and compress and encode the rendered output video;

[0026] The interactive control module comprises:

[0027] User interface and operation control unit: used to provide the user with a user interface through a touch control screen to operate the system and set parameters;

[0028] Data management and communication interface unit: used to manage and store system data and provide communication interfaces with other systems or applications.

[0029] Preferably, the character modeling unit creates appearance models of different parts of the digital human based on 3D modeling technology and stores them in a database.

[0030] Preferably, the skeleton binding and animation unit detects the 3D human body posture and compares it with the real human body joints, and analyzes and calculates the coordinate displacement of the vth joint point of the 33 joint points of the human body based on the video frame, as follows:

[0031] Based on the calculated 3D coordinate offset d of the j-1th frame data and the jth frame data j-1,j :

[0032]

[0033] Among them, (x j-1 ,y j-1 ,z j-1 ) is the joint position coordinate of the j-1th frame; (x j ,y j ,z j ) is the joint position coordinate of the jth frame;

[0034] Offset the 3D coordinate by d j-1,j Calculate the ratio with the current real person's height and update the 3D coordinate offset:

[0035]

[0036] Among them, d' j-1,j is the updated 3D coordinate offset;

[0037] Will d' j-1,j By taking the time difference Δt between two adjacent frames as the quotient, at the jth frame of the u action video, each joint point v generates a corresponding rate change value V ujv ;

[0038]

[0039] In a P-frame video, the average velocity of each joint point v for:

[0040]

[0041] Different actions are generated into corresponding action sets according to different action amplitudes, where each joint point corresponds to a different speed average, and the speed threshold V of other actions in the set is generated based on this. v :

[0042]

[0043] Where U is the number of actions in the action set;

[0044] The action set is divided based on the cosine annealing strategy, and cosine annealing is performed with the action recognition judgment error value corresponding to the division of the action set as the maximum judgment error value.

[0045] Preferably, the expression capture and mapping unit captures and recognizes labels based on a facial expression recognition network, wherein the facial expression recognition network includes an input module, a Transformer module fused with a self-attention mechanism, a multi-scale feature extraction module, and a fusion recognition module;

[0046] The input module collects real person's facial image data X0∈R H×W×C , where H and W are the height and width of the image data, and C is the number of image channels; the collected facial expression image data is divided into equal distances based on the numerical common divisor λ of the height and width of the image data and serialized to obtain X', each image block is position-encoded, each image block is linearly projected and the position encoding vector corresponding to each image is added to obtain the real input X;

[0047] The Transformer module integrating the self-attention mechanism includes a multi-head attention layer, a feedforward neural network layer and a layer normalization, wherein a residual connection module is further added between the input and output of the multi-head attention layer and the feedforward neural network layer;

[0048] The multi-scale feature extraction module divides the facial expressions of real people into several regions, each region corresponds to a square matrix with a dimension of the number of nodes in the region. The output of the multi-scale feature extraction module is integrated into the image block position parameter feature α and the node number parameter feature β, and is integrated into the self-attention mechanism. The output of the value vector is as follows:

[0049] The value vector matrix of the node e m After incorporating the number of nodes and the self-attention mechanism and extracting features, a new value vector e' is obtained n :

[0050]

[0051] Among them, f q is the dimension of the matching vector q; m and n are the total horizontal and vertical amounts of nodes in the image block, respectively;

[0052] The output Q of the multi-scale feature extraction module is as follows:

[0053]

[0054] Among them, A, B, and E are the query vector matrix, matching vector matrix, and value vector matrix composed of a, b, and e respectively; g α and g β is the calculation matrix of the corresponding position parameter features α and β; T is the transpose of the vector; G l Corresponding to different facial areas of real people;

[0055] The fusion recognition module concatenates the outputs of the Transformer module fusion self-attention mechanism and the multi-scale feature extraction module to obtain feature X R,Q , input into the spatial attention module and the channel attention module in sequence, in the spatial attention module,

[0056] The input features are processed along the channel dimension through the pooling operation:

[0057]

[0058] Where c∈[1,C] is the number of channels; M avg (X R,Q ), M max (X R,Q ) are the average pooling features and the maximum pooling features respectively;

[0059] M avg (X R,Q ) and M max (X R,Q ) are concatenated, and then a convolution layer is used for convolution operation to obtain the output spatial attention feature Y1:

[0060] Y1=σ(f conv,e (M avg (X R,Q );M max (X R,Q )))

[0061] The channel attention mechanism is as follows:

[0062] The input features are processed along the spatial dimensions through the pooling operation:

[0063]

[0064] in, is the image block height and width; avg (X R,Q ), Omax (X R,Q ) are the average pooling features and the maximum pooling features respectively;

[0065] O avg (X R,Q ) and O max (X R,Q ) are concatenated, and then a convolution layer is used for convolution operation to obtain the output channel attention Y1:

[0066] Y2=σ(MLP(O avg (X R,Q );O max (X R,Q )))

[0067] The output of the spatial attention module and the output of the channel attention module are fused to obtain the final fused output Z:

[0068] Z=δY1+(1-δ)Y2

[0069] Among them, δ is the fusion weight of the spatial attention module and the channel attention module;

[0070] The fusion weight δ is obtained by optimizing based on the cosine annealing strategy; the cosine annealing strategy adds a warm up strategy for initial transition to optimize the fusion weight, as follows:

[0071] The model recognition error value corresponding to the initial fusion weight is used as the cosine annealing of the maximum recognition error value. The recognition error value gradually decreases during the training process and reaches the minimum value at the end. The details are as follows:

[0072]

[0073] Among them, γ t is the fusion weight value of the tth iteration; is the minimum fusion weight; is the highest fusion weight; τ is the number of cycles currently being learned; N k is the total number of cycles in the current operating environment;

[0074] Finally, the Softmax layer outputs the recognition results of the facial expression recognition network on the image data.

[0075] Preferably, the expression library and animation curve unit also adjust the animation curve based on the cosine annealing strategy, and perform cosine annealing processing based on the comprehensive effect of the expression change speed, amplitude and transition effect.

[0076] Preferably, the touch feedback screen is designed to be a curved or bendable shape to adapt to different application scenarios and device forms; the touch feedback screen is also spliced ​​together by several independent small touch feedback screen modules to form a complete screen, which is divided into several functional partitions according to user-defined requirements, and each functional partition has different touch functions and feedback methods.

[0077] Preferably, the touch feedback screen is designed with a three-layer sensing structure, corresponding to pressure sensing, position sensing and gesture sensing respectively; and the touch sensing circuit of the sensing structure is made of nanomaterials.

[0078] Preferably, a microlens array is constructed on the surface of the touch feedback screen, and a microfluidic channel is arranged below the microlens array, which is filled with different liquids with electrical or optical characteristics. The liquid pressure and flow state in the microfluidic channel dynamically adjust the curvature, spacing or direction of the microlens array; the information of the microlens array is also fed back to the microfluidic channel to dynamically adjust the flow and distribution of the liquid.

[0079] Preferably, the touch feedback screen also utilizes sensing technology to enable the touch feedback screen to detect the movement of a finger or stylus suspended within a certain distance above the screen, thereby constructing a touch feedback screen with three-dimensional space sensing capabilities.

[0080] Compared with the prior art, the present invention provides a digital human video synthesis system having the following features:

[0081] Beneficial effects:

[0082] The present invention can quickly build a character model based on the real character image, greatly shortening the long cycle of traditional video production. It also combines multimodal interaction methods such as voice, action, and expression, and can interact with users in real time by voice and text, making the interaction between digital people and users more natural and smooth, and adjusting the content and expression methods according to user feedback and needs to provide more intelligent and flexible services; at the same time, the touch screen is optimized to improve the sensitivity of responding to user operations and enhance user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0083] Figure 1 This is a system block diagram mentioned in Embodiment 1 of the present invention;

[0084] Figure 2 This is a schematic diagram of the human body joints mentioned in Example 1 of the present invention;

[0085] Figure 3 This is a schematic diagram of the residual link module structure mentioned in Embodiment 1 of the present invention;

[0086] Figure 4 This is a schematic diagram of the sensing structure mentioned in Example 1 of the present invention;

[0087] Figure 5 This is a schematic diagram of the surface structure of the touch feedback screen mentioned in Example 1 of the present invention.

[0088] Meaning of the symbols in the figure:

[0089] 1. Model building module; 11. Character modeling unit; 12. Bone binding and animation unit; 2. Speech synthesis module; 3. Expression generation module; 31. Expression capture and mapping unit; 32. Expression library and animation curve unit; 4. Rendering engine module; 41. Real-time rendering unit; 42. Rendering output and optimization unit; 5. Interaction control module; 51. User interface and operation control unit; 52. Data management and communication interface unit. DETAILED DESCRIPTION

[0090] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments.

[0091] The present invention can quickly build a character model based on real character images, greatly shortening the long cycle of traditional video production. It also combines multimodal interaction methods such as voice, action, and expression to interact with users in real time through voice and text, making the interaction between digital people and users more natural and smooth. It also adjusts the content and expression methods according to user feedback and needs to provide more intelligent and flexible services. At the same time, it optimizes the touch screen to improve the sensitivity of responding to user operations and improve user experience. Specifically, it includes the following contents.

[0092] Embodiment 1:

[0093] See also Figure 1-5 The digital human video synthesis system of the present invention comprises:

[0094] Model building module 1: used to model digital humans based on real people;

[0095] Model building module 1 includes:

[0096] Character modeling unit 11: used to create the appearance model of the digital human body, face, hairstyle, clothing, etc. based on 3D modeling technology, for example, using Maya, 3ds Max and other software for modeling, and store it in the database for easy access at any time. And based on scanning technology, scan the real person to obtain the shape, texture and other data of the body and face, and build the digital human model; for example, collect data through structured light scanning, laser scanning and other technologies.

[0097] Skeleton binding and animation unit 12: used to establish a skeleton system for the digital human model, determine the position and range of motion of the joints, so as to achieve natural action performance, and perform action recognition and animation production through skeleton binding; the steps of action recognition through skeleton binding are as follows:

[0098] Detect 3D human body posture and compare it with real human joints. The 33 joints of the human body can be referenced Figure 2 The coordinate displacement of the vth joint point is analyzed and calculated based on the video frame, as follows:

[0099] Based on the calculated 3D coordinate offset d of the j-1th frame data and the jth frame data j-1,j :

[0100]

[0101] Among them, (x j-1 ,y j-1 ,z j-1 ) is the joint position coordinate of the j-1th frame; (x j ,y j ,z j ) is the joint position coordinate of the jth frame;

[0102] Offset the 3D coordinate by d j-1,j Calculate the ratio with the current real person's height and update the 3D coordinate offset:

[0103]

[0104] Among them, d' j-1,j is the updated 3D coordinate offset;

[0105] Will d' j-1,j By taking the time difference Δt between two adjacent frames as the quotient, at the jth frame of the u action video, each joint point v generates a corresponding rate change value V ujv ;

[0106]

[0107] In a P-frame video, the average velocity of each joint point v for:

[0108]

[0109] Different actions are generated into corresponding action sets according to different action amplitudes, where each joint point corresponds to a different speed average, and the speed threshold V of other actions in the set is generated based on this. v :

[0110]

[0111] Where U is the number of actions in the action set;

[0112] The action set is divided based on the cosine annealing strategy, and cosine annealing is performed with the action recognition judgment error value corresponding to the division of the action set as the maximum judgment error value. The action set obtained based on the cosine annealing strategy can more accurately identify the actions made by real people through the speed threshold.

[0113] It can also be used to develop animation control systems that support keyframe animation, motion capture data import, physical simulation and other animation generation methods, such as using MotionBuilder software for bone binding and animation production.

[0114] Speech synthesis module 2: used to convert the input text content into speech audio through the speech synthesis engine. Common speech synthesis technologies include speech synthesis based on parameter synthesis, splicing synthesis and deep learning. The timbre, intonation, and speaking speed of the synthesized speech are adjusted and optimized to adapt to different digital human roles and application scenarios. For example, iFlytek's speech synthesis technology can generate speech with a variety of timbres and styles. For example, the voice broadcast in the navigation software is realized through text-to-speech technology, providing users with clear and accurate route guidance.

[0115] Expression generation module 3: used to generate the expression of the digital human according to the facial expression of the real person;

[0116] Expression generation module 3 includes:

[0117] Expression capture and mapping unit 31: used to capture facial expression data of real people through facial capture equipment such as cameras, depth sensors, etc., and map it to the digital human model to achieve real-time driving of expressions;

[0118] The expression capture and mapping unit 31 captures and recognizes labels based on a facial expression recognition network, wherein the facial expression recognition network includes an input module, a Transformer module fused with a self-attention mechanism, a multi-scale feature extraction module, and a fusion recognition module;

[0119] The input module collects real person’s facial image data X0∈R H×W×C , where H and W are the height and width of the image data, and C is the number of image channels; the collected facial expression image data is divided into equal distances based on the numerical common divisor λ of the height and width of the image data and serialized to obtain X', each image block is position-encoded, each image block is linearly projected and the position encoding vector corresponding to each image is added to obtain the real input X;

[0120] The Transformer module integrating the self-attention mechanism includes a multi-head attention layer, a feedforward neural network layer, and layer normalization. A residual connection module is added between the input and output of the multi-head attention layer and the feedforward neural network layer. The residual connection module can be referred to Figure 3 ,The upper path and the lower path are the skip connection and the main path respectively. The skip connection merges the stacked layer input and output through the identity mapping process without additional parameters. The gradient is back-propagated to the first few layers, which allows faster training through more additional layers, making training easier.

[0121] The multi-scale feature extraction module divides the facial expressions of real people into several regions. Each region corresponds to a square matrix with a dimension of the number of nodes in the region. The output of the multi-scale feature extraction module is integrated into the image block position parameter feature α and the node number parameter feature β, and the self-attention mechanism is integrated. The output of the value vector is as follows:

[0122] The value vector matrix of the node e m After incorporating the number of nodes and the self-attention mechanism and extracting features, a new value vector e' is obtained n :

[0123]

[0124] Among them, f q is the dimension of the matching vector q; m and n are the total horizontal and vertical amounts of nodes in the image block, respectively;

[0125] The output Q of the multi-scale feature extraction module is as follows:

[0126]

[0127] Among them, A, B, and E are the query vector matrix, matching vector matrix, and value vector matrix composed of a, b, and e respectively; g α and g β is the calculation matrix of the corresponding position parameter features α and β; T is the transpose of the vector; G l Corresponding to different facial areas of real people;

[0128] The fusion recognition module concatenates the outputs of the Transformer module with the self-attention mechanism and the multi-scale feature extraction module to obtain the feature X R,Q , input into the spatial attention module and channel attention module in turn, in the spatial attention module,

[0129] The input features are processed along the channel dimension through the pooling operation:

[0130]

[0131] Where c∈[1,C] is the number of channels; M avg (X R,Q ), M max (X R,Q ) are the average pooling features and the maximum pooling features respectively;

[0132] M avg (X R,Q ) and M max (X R,Q ) are concatenated, and then a convolution layer is used for convolution operation to obtain the output spatial attention feature Y1:

[0133] Y1=σ(f conv,e (M avg (X R,Q );M max (X R,Q )))

[0134] The channel attention mechanism is as follows:

[0135] The input features are processed along the spatial dimensions through the pooling operation:

[0136]

[0137] in, is the image block height and width; avg (X R,Q ), O max (X R,Q ) are the average pooling features and the maximum pooling features respectively;

[0138] O avg (X R,Q ) and O max (X R,Q ) are concatenated, and then a convolution layer is used for convolution operation to obtain the output channel attention Y1:

[0139] Y2=σ(MLP(O avg (X R,Q );O max (X R,Q )))

[0140] The output of the spatial attention module and the output of the channel attention module are fused to obtain the final fused output Z:

[0141] Z=δY1+(1-δ)Y2

[0142] Among them, δ is the fusion weight of the spatial attention module and the channel attention module;

[0143] The fusion weight δ is obtained by optimization based on the cosine annealing strategy. The cosine annealing strategy adds a warm up strategy for initial transition and optimizes the fusion weight as follows:

[0144] The model recognition error value corresponding to the initial fusion weight is used as the cosine annealing of the maximum recognition error value. The recognition error value gradually decreases during the training process and reaches the minimum value at the end. The details are as follows:

[0145]

[0146] Among them, γ t is the fusion weight value of the tth iteration; is the minimum fusion weight; is the highest fusion weight; τ is the number of cycles currently being learned; N k is the total number of cycles in the current operating environment;

[0147] Finally, the Softmax layer outputs the recognition results of the facial expression recognition network on the image data.

[0148] The convolutional network is used as model one, the convolutional network with self-attention mechanism is used as model two, the convolutional network with self-attention mechanism and spatial attention mechanism is used as model three, the convolutional network with self-attention mechanism and channel attention mechanism is used as model four, the convolutional network with self-attention mechanism, spatial attention mechanism and channel attention mechanism is used as model five, and the model of this application is used as model six. Facial expressions are recognized respectively. The recognition result accuracy can be referred to Table 1:

[0149] Table 1 Facial expression recognition accuracy results of different models

[0150] Emoticon Tags Model 1 Model 2 Model 3 Model 4 Model 5 Model 6 angry 66.51% 67.01% 69.12% 70.32% 71.84% 72.32% disgust 65.37% 66.23% 67.91% 68.72% 70.69% 71.08% fear 60.85% 66.06% 67.63% 69.71% 71.85% 72.89% happy 63.45% 64.89% 66.68% 69.28% 70.35% 72.94% sad 67.35% 68.15% 69.36% 71.15% 72.68% 74.06% surprise 68.16% 69.48% 70.06% 71.26% 72.68% 73.48%

[0151] It can be seen from Table 1 that the accuracy of the model in this embodiment is the highest.

[0152] It also uses deep learning algorithms to analyze the input voice or text content and automatically generate corresponding digital human expressions;

[0153] Expression library and animation curve unit 32: used to build an expression library, including various common expression movements and micro-expressions, so as to quickly call them when needed. The animation curve is also adjusted through the cosine annealing strategy, and the cosine annealing process is performed based on the comprehensive effect of the expression change speed, amplitude and transition effect to make the expression more natural and smooth.

[0154] Rendering engine module 4: used for rendering digital human video;

[0155] The rendering engine module 4 includes:

[0156] Real-time rendering unit 41: used to render the digital human model using advanced real-time rendering engines such as Unreal Engine, Unity, etc. to generate realistic visual effects; it also supports the adjustment and optimization of rendering effects such as ray tracing, shadows, materials, textures, etc. to improve the quality of digital human videos.

[0157] Rendering output and optimization unit 42: used to output the rendered digital human video into video files of different formats such as MP4, AVI, etc. and resolutions such as 720p, 1080p, 4K, etc., and compress and encode the rendered output video to reduce the file size, improve the transmission and storage efficiency, and ensure the video quality;

[0158] Interaction control module 5: used for data interaction with users and third parties;

[0159] The interactive control module 5 includes:

[0160] User interface and operation control unit 51: Provides a simple and intuitive user interface for users through the touch control screen, making it convenient for users to operate and set parameters of the digital human video synthesis system, such as selecting a digital human model, inputting text content, adjusting voice and expression parameters, etc.

[0161] Data management and communication interface unit 52: used to manage and store digital human models, voice data, expression data, animation data, etc., support data import, export and backup, and provide communication interfaces with other systems or applications to achieve data sharing and interaction, such as integration with video editing software, live broadcast platforms, etc.

[0162] Designing the touch feedback screen into a curved or bendable shape can adapt to different application scenarios and device forms. For example, designing a wraparound flexible touch feedback screen for a wearable device can better fit the curve of the human body and provide a more natural and comfortable interactive experience. Or designing a curved touch feedback screen for a car dashboard can improve the driver's visibility and operating convenience.

[0163] The touch feedback screen is also divided into multiple functional areas or a modular design is adopted. Different areas can have different touch functions and feedback methods, and users can customize the functions of each area according to their needs. For example, on the touch feedback screen of the game controller, the screen is divided into a direction control area, an action button area, and a function setting area, etc. Each area can provide different tactile feedback effects to enhance the immersion of the game.

[0164] Conventional touch feedback screens usually have only one sensing layer. This embodiment increases the number of sensing layers and is specifically designed as a three-layer sensing structure. Figure 4, the three-layer sensing structure from top to bottom corresponds to pressure sensing, position sensing and gesture sensing, respectively, to achieve more accurate and rich touch input recognition. For example, when the user touches the screen lightly, the first pressure sensing layer detects the pressure; the second position sensing layer determines the coordinates of the touch; the third gesture sensing layer recognizes finger gestures such as sliding and zooming. The sensing structure also uses nanomaterials such as carbon nanotubes and nano silver wires to make touch sensing circuits. These nanomaterials have excellent conductivity and flexibility, which can improve the sensitivity and bendability of the touch feedback screen. For example, using carbon nanotube film as the sensing layer, due to the high conductivity and small size of carbon nanotubes, faster signal response and higher spatial resolution can be achieved.

[0165] Reference Figure 5 , the surface of the touch feedback screen is constructed with a microlens array, which can focus light, enhance the display brightness and contrast of the screen, and can also be used to achieve 3D touch effects. When the user touches the screen, the position and strength of the touch are detected through the optical changes of the microlens array, and a three-dimensional feedback effect can be visually presented according to the touch operation. For example, in the game, the user presses the screen at different positions and strengths, and sees a 3D visual effect of objects with different degrees of protrusion or depression. A microfluidic channel is arranged under the microlens array, which is filled with different liquids with electrical or optical characteristics. For example, when the user touches the screen, the pressure will cause the liquid in the microfluidic channel to flow, thereby changing the local resistance or capacitance to achieve touch detection; or by changing the distribution of the liquid, the local light transmittance or color of the screen is affected to provide visual feedback. For example, in an e-book reading application, when the user touches the screen to turn pages, there will be a colorful liquid flow effect at the edge of the screen as feedback. The liquid pressure and flow state in the microfluidic channel dynamically adjust the curvature, spacing or direction of the microlens array; conversely, the external light, touch position and other information sensed by the microlens array can also be fed back to the microfluidic system to dynamically adjust the flow and distribution of the liquid, and realize the adaptive adjustment of screen display and touch feedback. For example, in a strong light environment outdoors, the microlens array detects an increase in light intensity, triggers the flow of liquid in the microfluidic channel, and changes the parameters of the microlens to enhance the display brightness and contrast of the screen; at the same time, when the user performs a touch operation, the microlens array transmits the touch information to the microfluidic system, causing the liquid to produce corresponding flow and tactile feedback.

[0166] By using technologies such as magnetic fields, electric fields or sound waves, the touch feedback screen can detect the movements of fingers or styluses suspended above the screen within a certain distance, thus achieving non-contact touch operations and feedback. For example, by swiping a finger across the top of the screen in the air, operations such as page scrolling and content selection can be achieved, while providing tactile feedback through air vibrations or slight vibrations of the device. A touch feedback screen with three-dimensional spatial perception can also be constructed in this way. By arranging multiple sensors inside or around the screen, the position, direction and movement of objects in three-dimensional space can be detected. For example, in virtual reality VR or augmented reality AR devices, users can directly interact with virtual objects in three-dimensional space by touching, grabbing, rotating, and other operations, and provide realistic tactile feedback through device vibrations, force feedback gloves, and other methods.

[0167] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can make equivalent replacements or changes according to the technical scheme and inventive concept of the present invention within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention.

Claims

1. A digital human video synthesis system, characterized in that: include: Model building module (1): used to model digital humans based on real people; Speech synthesis module (2): used to convert the input text content into speech audio through a speech synthesis engine, and adjust and optimize the synthesized speech; Expression generation module (3): used to generate the expression of the digital human according to the facial expression of the real person; Rendering engine module (4): used for rendering digital human video; Interaction control module (5): used for data interaction with users and third parties; The model building module (1) comprises: Character modeling unit (11): used to create an appearance model based on 3D modeling technology, and to scan a real person based on scanning technology to build a digital human model; Skeleton binding and animation unit (12): used to establish a skeleton system for the digital human model, determine the position and range of motion of the joints, and perform action recognition and animation production through skeleton binding; The expression generation module (3) comprises: Expression capture and mapping unit (31): used to capture the facial expression data of real people through facial capture equipment and map it to the digital human model to achieve real-time driving of expressions; and also use deep learning algorithms to analyze the input voice or text content to automatically generate corresponding digital human expressions; Expression library and animation curve unit (32): used to establish an expression library and control the expression change curve, amplitude and transition effect by adjusting the animation curve; The rendering engine module (4) comprises: A real-time rendering unit (41): used to render the digital human model using a real-time rendering engine; Rendering output and optimization unit (42): used to output the rendered digital human video into video files of different formats and resolutions, and compress and encode the rendered output video; The interactive control module (5) comprises: User interface and operation control unit (51): used to provide a user interface for the user through a touch control screen to operate the system and set parameters; Data management and communication interface unit (52): used to manage and store system data and provide a communication interface with other systems or applications.

2. The digital human video synthesis system according to claim 1, characterized in that: The character modeling unit (11) creates appearance models of different parts of a digital human based on 3D modeling technology and stores the models in a database.

3. The digital human video synthesis system according to claim 2, characterized in that: The skeleton binding and animation unit (12) detects the 3D human body posture and compares it with the real human body joints, and analyzes and calculates the coordinate displacement of the vth joint point of the 33 joint points of the human body based on the video frame, as follows: Based on the calculated 3D coordinate offset d of the j-1th frame data and the jth frame data j-1,j : Among them, (x j-1 ,y j-1 ,z j-1 ) is the joint position coordinate of the j-1th frame; (x j ,y j ,z j ) is the joint position coordinate of the jth frame; Offset the 3D coordinate by d j-1,j Calculate the ratio with the current real person's height and update the 3D coordinate offset: Among them, d' j-1,j is the updated 3D coordinate offset; Will d' j-1,j By taking the time difference Δt between two adjacent frames as the quotient, at the jth frame of the u action video, each joint point v generates a corresponding rate change value V ujv ; In a P-frame video, the average velocity of each joint point v for: Different actions are generated into corresponding action sets according to different action amplitudes, where each joint point corresponds to a different speed average, and the speed threshold V of other actions in the set is generated based on this. v : Where U is the number of actions in the action set; The action set is divided based on the cosine annealing strategy, and cosine annealing is performed with the action recognition judgment error value corresponding to the division of the action set as the maximum judgment error value.

4. The digital human video synthesis system according to claim 3, characterized in that: The expression capture and mapping unit (31) captures and recognizes labels based on a facial expression recognition network, wherein the facial expression recognition network includes an input module, a Transformer module fused with a self-attention mechanism, a multi-scale feature extraction module, and a fusion recognition module; The input module collects real person's facial image data X0∈R H×W×C , where H and W are the height and width of the image data, and C is the number of image channels; the collected facial expression image data is divided into equal distances based on the numerical common divisor λ of the height and width of the image data and serialized to obtain X', each image block is position-encoded, each image block is linearly projected and the position encoding vector corresponding to each image is added to obtain the real input X; The Transformer module integrating the self-attention mechanism includes a multi-head attention layer, a feedforward neural network layer and a layer normalization, wherein a residual connection module is further added between the input and output of the multi-head attention layer and the feedforward neural network layer; The multi-scale feature extraction module divides the facial expressions of real people into several regions, each region corresponds to a square matrix with a dimension of the number of nodes in the region. The output of the multi-scale feature extraction module is integrated into the image block position parameter feature α and the node number parameter feature β, and is integrated into the self-attention mechanism. The output of the value vector is as follows: The value vector matrix of the node e m After incorporating the number of nodes and the self-attention mechanism and extracting features, a new value vector e' is obtained n : Among them, f q is the dimension of the matching vector q; m and n are the total horizontal and vertical amounts of nodes in the image block, respectively; The output Q of the multi-scale feature extraction module is as follows: Among them, A, B, and E are the query vector matrix, matching vector matrix, and value vector matrix composed of a, b, and e respectively; g α and g β is the calculation matrix of the corresponding position parameter features α and β; T is the transpose of the vector; G l Corresponding to different facial areas of real people; The fusion recognition module concatenates the outputs of the Transformer module fusion self-attention mechanism and the multi-scale feature extraction module to obtain feature X R,Q , input into the spatial attention module and the channel attention module in sequence, in the spatial attention module, The input features are processed along the channel dimension through the pooling operation: Where c∈[1,C] is the number of channels; M avg (X R,Q )、M max (X R,Q ) are the average pooling features and the maximum pooling features respectively; M avg (X R,Q ) and M max (X R,Q ) are concatenated, and then a convolution layer is used for convolution operation to obtain the output spatial attention feature Y1: Y1=σ(f conv,e (M avg (X R,Q );M max (X R,Q ))) The channel attention mechanism is as follows: The input features are processed along the spatial dimensions through the pooling operation: in, is the image block height and width; avg (X R,Q ), O max (X R,Q ) are the average pooling features and the maximum pooling features respectively; O avg (X R,Q ) and O max (X R,Q ) are concatenated, and then a convolution layer is used for convolution operation to obtain the output channel attention Y1: Y2=σ(MLP(O avg (X R,Q );O max (X R,Q ))) The output of the spatial attention module and the output of the channel attention module are fused to obtain the final fused output Z: Z=δY1+(1-δ)Y2 Among them, δ is the fusion weight of the spatial attention module and the channel attention module; The fusion weight δ is obtained by optimizing based on the cosine annealing strategy; the cosine annealing strategy adds a warm up strategy for initial transition to optimize the fusion weight, as follows: The model recognition error value corresponding to the initial fusion weight is used as the cosine annealing of the maximum recognition error value. The recognition error value gradually decreases during the training process and reaches the minimum value at the end. The details are as follows: Among them, γ t is the fusion weight value of the tth iteration; is the minimum fusion weight; is the highest fusion weight; τ is the number of cycles currently being learned; N k is the total number of cycles in the current operating environment; Finally, the Softmax layer outputs the recognition results of the facial expression recognition network on the image data.

5. The digital human video synthesis system according to claim 4, characterized in that: The expression library and animation curve unit (32) also adjusts the animation curve based on the cosine annealing strategy, and performs cosine annealing processing based on the comprehensive effect of the expression change speed, amplitude and transition effect.

6. The digital human video synthesis system according to claim 1, characterized in that: The touch feedback screen is designed to be a curved or bendable shape to adapt to different application scenarios and device forms; the touch feedback screen is also spliced ​​together by several independent small touch feedback screen modules to form a complete screen, which is divided into several functional partitions according to user customized requirements, and each functional partition has different touch functions and feedback methods.

7. The digital human video synthesis system according to claim 6, characterized in that: The touch feedback screen is designed with a three-layer sensing structure, which corresponds to pressure sensing, position sensing and gesture sensing respectively; nano materials are also used to make the touch sensing circuit of the sensing structure.

8. The digital human video synthesis system according to claim 7, characterized in that: A microlens array is constructed on the surface of the touch feedback screen, and a microfluidic channel is arranged below the microlens array, which is filled with different liquids with electrical or optical characteristics. The liquid pressure and flow state in the microfluidic channel dynamically adjust the curvature, spacing or direction of the microlens array; the information of the microlens array is also fed back to the microfluidic channel to dynamically adjust the flow and distribution of the liquid.

9. The digital human video synthesis system according to claim 8, characterized in that: The touch feedback screen also uses sensing technology to enable the touch feedback screen to detect the movement of a finger or stylus suspended within a certain distance above the screen, thereby constructing a touch feedback screen with three-dimensional space sensing capabilities.

Citation Information

Patent Citations

  • Method of controlling sliding of display interface and touch terminal

    CN108021327A

  • Game control method and device, computer equipment and storage medium

    CN113867610A

  • Image target detection method based on double attention mechanism

    CN114863236A

  • Virtual human expression personalized generation method based on multi-modal interaction information

    CN116311456A

  • Digital human virtual interaction system and method based on display screen

    CN117519477A