Interaction method and system based on digital human

Abstract

The invention relates to the technical field of digital human interaction, and aims to solve the problem of poor user experience caused by difficulty in high-naturalness and high-immersion interaction between a user and a digital human in the prior art. In order to achieve the purpose, the invention provides an interaction method and system based on the digital human, and the method comprises the steps: obtaining voice information inputted by a user, and capturing the motion information of the user, the motion information comprising at least one of head motion information, face motion information and hand motion information; determining a voice interaction mode of the digital person based on voice information input by the user; and determining an action interaction mode and a text interface display mode of the digital human based on the voice information input by the user and the captured action information of the user. According to the invention, highly natural and smooth interactive experience between the digital human and the user can be realized, an immersive feeling which is almost interactive with real human beings is created, and the interactive experience of the user is improved.

Description

Technical Field

[0001] The present invention relates to the field of digital human interaction technology, and in particular to an interaction method and system based on digital human. Background Art

[0002] With the rapid development of computer graphics, artificial intelligence, and speech recognition and synthesis technology, virtual digital human technology, as an important representative of the new generation of human-computer interaction, is gradually penetrating into entertainment, education, medical care, business and other fields, showing great application potential and market value. Digital human, as a highly realistic virtual image, can not only simulate the appearance, movements and expressions of real humans, but also achieve natural and smooth communication with users through advanced interactive technology, greatly enriching the user's digital experience.

[0003] In the traditional human-computer interaction mode, users mainly input commands to computers through physical devices such as keyboards and mice. Although this interaction mode is efficient, it is obviously insufficient in immersion and naturalness. With the advancement of technology, people have begun to explore more intuitive and humanized interaction modes, such as touch screen technology, gesture recognition, facial expression recognition, etc. The application of these technologies makes human-computer interaction closer to the natural communication between people.

[0004] In the field of digital human interaction, early attempts mainly focused on simple voice interaction and preset animation display. Users controlled the basic behavior of digital humans through voice commands, and the feedback of digital humans was mostly limited to preset animations and voice responses, lacking the ability to dynamically adjust according to the user's real-time status. With the continuous optimization of algorithms such as deep learning and machine learning, and the popularization of high-precision sensors and cameras, it has become possible to capture and analyze users' voices and movements in real time, which provides a solid foundation for building a more intelligent and interactive digital human interaction system.

[0005] Especially in the entertainment industry, such as virtual idols, game characters, virtual teachers in online education and other application scenarios, users have higher requirements for the interactive experience of digital humans. Users not only hope that digital humans can accurately understand and respond to their voice commands, but also expect digital humans to respond accordingly based on users' actions, expressions and other non-verbal information, so as to achieve a deeper level of emotional communication and interactive experience.

[0006] In view of this, the art needs an interactive method and system based on digital humans to solve the above problems. Summary of the invention

[0007] In order to solve the above technical problem, that is, to solve the problem in the prior art that it is difficult for users to interact with digital humans with high naturalness and immersion, resulting in poor user experience.

[0008] In a first aspect, the present invention provides an interactive method based on a digital human, the interactive method comprising:

[0009] S1: Acquire voice information input by a user and capture the user's motion information, wherein the motion information includes at least one of head motion information, facial motion information and hand motion information;

[0010] S2: Determine the voice interaction mode of the digital human based on the voice information input by the user;

[0011] S3: Determine the action interaction mode of the digital human and the text interface display mode based on the voice information input by the user and the captured action information of the user.

[0012] In some preferred embodiments, step S2 specifically includes:

[0013] S21: Acquire language type information, speech speed information, intonation information, and voice content information input by the user;

[0014] S22: inputting the language type information, the speech speed information, the intonation information and the voice content information input by the user into a voice interaction matching model;

[0015] S23: Determine the language type, speaking speed, intonation, emotion, background and environmental sound effects of the digital human based on the voice interaction matching model.

[0016] In some preferred embodiments, the voice interaction matching model is:

[0017]

[0018] in, is the output digital human voice interaction feature vector, F is the voice interaction matching model function, is the function for preprocessing the language type information l, ψ(s) is the function for preprocessing the speech rate information s, χ(t) is the function for preprocessing the intonation information t, ω p (C p ) is the voice content information The dimensions of C p The function for preprocessing, x n is the weight of the language type information l in the power operation, y n is the weight of the speech rate information s in the power operation, z n is the weight of intonation information t in the power operation, ω np Voice content information The dimensions of C p The weight in the exponentiation operation, ω npIn matrix form, n and p are the level and dimension of the matrix respectively, ∈ n is the weight of the product operation results at different levels in the subsequent linear combination, δ m To adjust the weights in the linear combination of the mth level, γ k To adjust the weights in the linear combination of the kth level, β j To adjust the weights in the linear combination of the jth level, α i To adjust the weights in the linear combination of the i-th level, σ, τ, ω and ρ are nonlinear activation functions, and I is the weight parameter α i The number of layers that linearly combine and nonlinearly transform the internal results with the activation function σ, and J is the weight parameter β j The number of layers that linearly combine and nonlinearly transform the internal results with the activation function τ, and K is the weight parameter γ k The number of layers that linearly combine and nonlinearly transform the internal results with the activation function ω, M is the weight parameter δ m The number of layers that linearly combine and nonlinearly transform the internal results with the activation function ρ, N is the number of layers that are transformed by the weight parameter ∈ n The number of layers for linear combination of the product results of each input information, ∈ is the bias term.

[0019] In some preferred embodiments, step S3 specifically includes:

[0020] S31: determining whether the interaction method of the digital human includes a text interface;

[0021] S321: If the interaction mode of the digital human includes a text interface, further determining whether the display space of the text interface is sufficient;

[0022] S331: If the display space of the text interface is sufficient, no adjustment is made to the digital human;

[0023] S332: If the display space of the text interface is insufficient, the digital human is scaled and / or the display position of the digital human is adjusted so that the display space of the text interface is sufficient and the digital human has enough space for action changes.

[0024] In some preferred embodiments, step S332 specifically includes:

[0025] S3321: Determine the action change space of the digital human based on the voice information input by the user and the captured action information of the user;

[0026] S3322: If the display space of the text interface is insufficient, determine whether to scale the digital human alone, adjust the display position of the digital human alone, or scale and adjust the display position of the digital human at the same time based on the display space required by the text interface and the movement change space required by the digital human.

[0027] In some preferred implementations, the following formula is used to determine the motion variation space of the digital human:

[0028]

[0029] Among them, S action is the action variation space required by the calculated digital human, n d is the number of speech features, m d is the number of head motion features, k d is the number of facial action features, l d is the number of hand motion features, v i is the i-th speech feature, a hj is the jth head action feature, a hk is the kth head action feature, a fi is the i-th facial action feature, a fk is the kth facial action feature, a gj is the jth hand action feature, a gk is the kth hand action feature, p vhj is the association parameter vector of the jth voice information and head movement information, p vfi is the parameter vector of the association between the ith voice information and facial action information, p vgj is the correlation parameter vector of the jth speech information and hand movement information, q ahj is the j-th head motion information autocorrelation parameter vector, q afi is the autocorrelation parameter vector of the i-th facial action information, q agj is the j-th hand motion information autocorrelation parameter vector, r hfi is the interaction parameter vector between the i-th head action information and facial action information, r hgk is the interaction parameter vector of the kth head motion information and hand motion information, r fgk The interaction parameter vector of the kth facial action information and hand action information, C is a constant adjustment factor.

[0030] In certain preferred embodiments, the speech features include intonation features, speaking speed features and volume features; the head movement features include horizontal head rotation angle features, vertical head rotation angle features and head movement speed features; the facial movement features include eye movement features, facial expression intensity features and facial expression change frequency features; the hand movement features include finger gesture features, hand movement amplitude features and hand movement frequency features.

[0031] In some preferred implementations, step S3322 specifically includes:

[0032] S33221: If the display space of the text interface is insufficient, the space occupied by the scaled digital human is calculated according to the following formula:

[0033]

[0034] S33222: Calculate the space occupied by the digital human after adjusting the display position according to the following formula:

[0035]

[0036] S33223: Calculate the adjustment value of comprehensive influencing factors according to the following formula:

[0037]

[0038] S33224: According to S scaled , S adjusted , S influence , S action and S text Determine whether to scale the digital human alone, adjust the display position of the digital human alone, or scale the digital human and adjust the display position at the same time;

[0039] Among them, S scaled The scaled digital human occupies space, S adjusted To adjust the space occupied by the digital person after displaying the position, S original is the space occupied by the digital human in the initial state, S influence is the adjustment value of comprehensive influencing factors, S text is the display space required for the text interface, r d is the number of dimensions of the scaling ratio, k 1i is the i-th parameter in the digital human scaling feature vector, f 1i is the i-th parameter in the regularization parameter vector of the scaling factor, p k1i is the i-th parameter in the associated parameter vector of the scaling factor, k 2i The i-th parameter in the feature vector adjusted for the digital human display position, f 2iis the i-th parameter in the regularization parameter vector showing the position adjustment, p k2i is the i-th parameter in the associated parameter vector of the displayed position adjustment, s d is the number of directions of display position adjustment, x, y and z are the three dimensions of the coordinate vector of the initial display position of the digital human, Δx, Δy and Δz are the three dimensions of the adjustment vector of the display position of the digital human, C′ is a constant adjustment factor, t d is the total number of features of other influencing factors, g k is the kth parameter in the characteristic vector of other influencing factors, q gk1i is the i-th parameter in the interaction parameter vector of other influencing factors and scaling ratio, q gk2j is the jth parameter in the interaction parameter vector between other influencing factors and the display position adjustment amount.

[0040] In some preferred implementations, the other influencing factors include at least one of a user preference factor and an interface layout constraint factor; step S33224 specifically includes:

[0041] If S scaled +S influence ≥S text +S action And S adjusted +S influence ≥S text +S action , then further determine whether to scale the digital human alone or adjust the display position of the digital human alone according to the preset factors;

[0042] If S scaled +S influence ≥S text +S action And S adjusted +S influence <S text +S action , then scale the digital human alone;

[0043] If S scaled +S influence text +S action And S adjusted +S influence ≥S text +S action , then adjust the display position of the digital person separately;

[0044] If S scaled +S influence text +S action And S adjusted +S influence ​​<S text +S action , the digital human will be scaled and the display position will be adjusted at the same time.

[0045] In a second aspect, the present invention further provides an interactive system based on a digital human, the interactive system comprising:

[0046] An acquisition module, which is used to acquire voice information input by a user and capture the user's motion information, wherein the motion information includes at least one of head motion information, facial motion information and hand motion information;

[0047] A first determination module, which is used to determine the voice interaction mode of the digital human based on the voice information input by the user;

[0048] The second determination module is used to determine the action interaction mode of the digital human and the text interface display mode based on the voice information input by the user and the captured action information of the user.

[0049] It can be seen from the above that the digital human-based interaction method and system provided by the present invention have the following beneficial technical effects:

[0050] The present invention can achieve a highly natural and smooth interactive experience between the digital human and the user by deeply integrating the voice information input by the user and the user action information captured in real time, greatly promoting the leap of human-computer interaction to a higher level. At the same time, the present invention not only enables the digital human to accurately understand and instantly respond to the user's complex voice commands, but also can intelligently adjust its interactive behavior, action performance and text interface display according to the user's subtle action changes, facial expressions and other non-verbal clues, thereby creating an immersive feeling of interaction with a nearly real human being and enhancing the user's interactive experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] The features and advantages of the present invention will be more clearly understood by referring to the accompanying drawings, which are schematic and should not be construed as limiting the present invention in any way. In the accompanying drawings:

[0052] Figure 1 A flowchart of the digital human-based interaction method of the present invention;

[0053] Figure 2 It is a structural schematic diagram of the interactive system based on digital human of the present invention. DETAILED DESCRIPTION

[0054] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0055] Based on the problem pointed out in the background art that it is difficult for users to interact with digital humans with high naturalness and immersion in the prior art, resulting in poor user experience. The present invention provides an interaction method and system based on digital humans, aiming to achieve a highly natural and smooth interaction experience between digital humans and users, create an immersive feeling of interacting with nearly real humans, and enhance the user's interaction experience.

[0056] like Figure 1 As shown, the digital human-based interaction method of the present invention includes:

[0057] S1: Acquire voice information input by a user and capture the user's motion information, wherein the motion information includes at least one of head motion information, facial motion information and hand motion information, and preferably the motion information includes head motion information, facial motion information and hand motion information at the same time;

[0058] S2: Determine the voice interaction mode of the digital human based on the voice information input by the user;

[0059] S3: Determine the action interaction mode of the digital human and the text interface display mode based on the voice information input by the user and the captured action information of the user.

[0060] In a preferred embodiment, the above step S2 specifically includes:

[0061] S21: Obtain the language type information, speaking speed information, intonation information and voice content information input by the user; S22: Input the language type information, speaking speed information, intonation information and voice content information input by the user into the voice interaction matching model; S23: Determine the language type, speaking speed, intonation, emotion, background and environmental sound effects of the digital person based on the voice interaction matching model. Through such settings, that is, by obtaining multi-dimensional information such as language type, speaking speed, intonation and voice content input by the user and inputting it into the voice interaction matching model, the naturalness and personalization of the interaction are greatly improved. The digital human can respond according to the user's specific voice style, such as selecting the corresponding language type for interaction, adjusting its own speaking speed, intonation and emotional state according to different speaking speeds and intonations, etc., to enhance the immersive experience; not only considering the voice content, but also paying attention to factors such as language type, speaking speed, intonation, and the determination of background and environmental sound effects, so that the interaction between the digital human and the user is no longer limited to simple language responses, but also includes background sound effects and environmental settings that match the voice style, enhancing the immersion and richness of the interaction; in addition, the voice interaction matching model can quickly and accurately analyze the user's voice characteristics, determine the appropriate interaction method for the digital human, improve the efficiency and accuracy of the interaction, avoid misunderstandings due to language differences or different expression styles, and at the same time give more accurate responses through in-depth analysis of the voice content, adapt to diverse user needs, and meet the habits of different users and different scene needs.

[0062] Further preferably, the voice interaction matching model is:

[0063]

[0064] in, is the output digital human voice interaction feature vector, F is the voice interaction matching model function, is the function for preprocessing the language type information l, ψ(s) is the function for preprocessing the speech rate information s, χ(t) is the function for preprocessing the intonation information t, ω p (C p ) is the voice content information The dimensions of C p The function for preprocessing, x n is the weight of the language type information l in the power operation, y n is the weight of the speech rate information s in the power operation, z n is the weight of intonation information t in the power operation, ω np Voice content information The dimensions of C p The weight in the exponentiation operation, ω np In matrix form, n and p are the level and dimension of the matrix respectively, ∈ nis the weight of the product operation results at different levels in the subsequent linear combination, δ m To adjust the weights in the linear combination of the mth level, γ k To adjust the weights in the linear combination of the kth level, β j To adjust the weights in the linear combination of the jth level, α i To adjust the weights in the linear combination of the i-th level, σ, τ, ω and ρ are nonlinear activation functions, and I is the weight parameter α i The number of layers that linearly combine and nonlinearly transform the internal results with the activation function σ, and J is the weight parameter β j The number of layers that linearly combine and nonlinearly transform the internal results with the activation function τ, and K is the weight parameter γ k The number of layers that linearly combine and nonlinearly transform the internal results with the activation function ω, M is the weight parameter δ m The number of layers that linearly combine and nonlinearly transform the internal results with the activation function ρ, N is the number of layers that are transformed by the weight parameter ∈ n The number of layers for linear combination of the product results of each input information, ∈ is the bias term.

[0065] In the above, the output digital human voice interaction feature vector The various dimensions of the vector represent different characteristics of the digital human's language type, speaking speed, intonation, emotion, background and environmental sound effects. By setting specific mapping rules, the values ​​of the various dimensions of the vector can be converted into specific voice interaction parameters. For example, the values ​​of certain dimensions can correspond to the language type code of the digital human, and the specific language type can be determined according to the preset mapping table; the values ​​of other dimensions can control the digital human's speaking speed, intonation and other characteristics. The vector value is mapped to the corresponding parameter range through the conversion formula. In this way, the voice interaction mode of the digital human can be accurately determined according to the voice information input by the user, so as to achieve a more natural and personalized interaction effect. In a possible scenario, the user is using an online learning software with a digital human interaction function. The user asks the digital human questions about a math course. The voice has a certain degree of urgency, the speech speed is fast, the language is Chinese, and the content of the question is relatively complex. After step S2, the digital human responds, the language type continues to use Chinese to answer, the speech speed is accelerated to match the user's fast speech input, the tone slightly increases the urgency of the tone to respond to the user's urgency, and the emotion shows a positive and focused emotion, so that the user feels that the digital human takes the problem seriously. The background and environmental sound effects can choose a simple learning background and add some slight prompt sound effects to enhance the atmosphere of interaction. For example, the digital human quickly answers the user's question in Chinese and appropriately emphasizes the key content in the answering process, so that the user can understand the answer more clearly. At the same time, the setting of the background and environmental sound effects can allow the user to focus more on the learning interaction process. The language type can include Mandarin Chinese, Chinese dialects, English, French, German, etc.

[0066] In a preferred implementation, the aforementioned step S3 specifically includes: S31: determining whether the interaction mode of the digital human includes a text interface; S321: if the interaction mode of the digital human includes a text interface, further determining whether the display space of the text interface is sufficient; S331: if the display space of the text interface is sufficient, then no adjustment is made to the digital human; S332: if the display space of the text interface is insufficient, then scaling the digital human and / or adjusting the display position of the digital human so that the space for displaying the text interface is sufficient and the digital human has enough space for action changes; S322: if the interaction mode of the digital human does not include a text interface, then determining the action interaction mode of the digital human based on the voice information input by the user and the captured action information of the user. That is, by judging whether the digital human interaction method includes a text interface and evaluating the text interface display space, the interactive interface layout can be intelligently optimized. When the text interface display space is sufficient, it remains stable. When it is insufficient, the digital human is scaled and the display position is adjusted, which ensures that the text interface fully presents information and that the digital human has enough space for action changes, thus achieving reasonable planning and efficient use of the interactive interface. This greatly improves the smoothness of the user experience, avoids affecting the user's understanding of the interactive content due to incomplete text interface display, and ensures the action display of the digital human, allowing users to obtain information and interact more smoothly. In addition, the adaptability and flexibility of the interaction are enhanced, and the layout can be dynamically adjusted according to different interaction scenarios and device screen sizes. It is widely applicable to various terminal devices, providing users with a convenient and efficient interactive experience.

[0067] Further preferably, the above step S332 specifically includes: S3321: determining the motion change space of the digital human based on the voice information input by the user and the captured motion information of the user; S3322: if the display space of the text interface is insufficient, determining whether to scale the digital human alone, adjust the display position of the digital human alone, or scale and adjust the display position of the digital human at the same time based on the display space required by the text interface and the motion change space required by the digital human.

[0068] More preferably, in the above step S3321, the following formula is used to determine the action change space of the digital human:

[0069]

[0070] Among them, S action is the action variation space required by the calculated digital human, n d is the number of speech features, m d is the number of head motion features, k d is the number of facial action features, l d is the number of hand motion features, v i is the i-th speech feature, a hjis the jth head action feature, a hk is the kth head action feature, a fi is the i-th facial action feature, a fk is the kth facial action feature, a gj is the jth hand action feature, a gk is the kth hand action feature, p vhj is the association parameter vector of the jth voice information and head movement information (used to reflect the influence of voice features on head movement features), is the association parameter vector of the ith voice information and facial action information (used to reflect the influence of voice features on facial action features), p vgj is the association parameter vector of the jth voice information and hand movement information (used to reflect the influence of voice features on hand movement features), q jha is the jth head motion information autocorrelation parameter vector (used to reflect the mutual influence between head motion features), q afi is the autocorrelation parameter vector of the i-th facial action information (used to reflect the mutual influence between facial action features), q agj is the jth hand motion information autocorrelation parameter vector (used to reflect the mutual influence between hand motion features), r hfi is the interaction parameter vector of the i-th head action information and facial action information (used to reflect the mutual influence between head features and facial features), r hgk is the interaction parameter vector of the kth head motion information and hand motion information (used to reflect the mutual influence between head features and hand features), r fgk The kth interaction parameter vector of facial movement information and hand movement information (used to reflect the mutual influence between facial features and hand features), C is a constant adjustment factor; wherein, speech features include intonation features, speech speed features and volume features; head movement features include horizontal head rotation angle features, vertical head rotation angle features and head movement speed features; facial movement features include eye movement features, facial expression intensity features and facial expression change frequency features; hand movement features include finger gesture features, hand movement amplitude features and hand movement frequency features.

[0071] More preferably, the above step S3322 specifically includes:

[0072] S33221: If the display space of the text interface is insufficient, the space occupied by the scaled digital human is calculated according to the following formula:

[0073]

[0074] S33222: Calculate the space occupied by the digital human after adjusting the display position according to the following formula:

[0075]

[0076] S33223: Calculate the adjustment value of comprehensive influencing factors according to the following formula:

[0077]

[0078] S33224: According to S scaled , S adjusted , S influence , S action and S text Determine whether to scale the digital human alone, adjust the display position of the digital human alone, or scale the digital human and adjust the display position at the same time;

[0079] Among them, S scaled The scaled digital human occupies space, S adjusted To adjust the space occupied by the digital person after displaying the position, S original is the space occupied by the digital human in the initial state, S influence is the adjustment value of comprehensive influencing factors, S text is the display space required for the text interface, r d is the number of dimensions of the scaling ratio, k 1i is the i-th parameter in the digital human scaling feature vector, f 1i is the i-th parameter in the regularization parameter vector of the scaling factor, p k1i is the i-th parameter in the associated parameter vector of the scaling factor, k 2i The i-th parameter in the feature vector adjusted for the digital human display position, f 2i is the i-th parameter in the regularization parameter vector showing the position adjustment, p k2i is the i-th parameter in the associated parameter vector of the displayed position adjustment, s d is the number of directions of display position adjustment, x, y and z are the three dimensions of the coordinate vector of the initial display position of the digital human, Δx, Δy and Δz are the three dimensions of the adjustment vector of the display position of the digital human, C′ is a constant adjustment factor, t d is the total number of features of other influencing factors, g k is the kth parameter in the characteristic vector of other influencing factors, q gk1i is the i-th parameter in the interaction parameter vector of other influencing factors and scaling ratio, q gk2j is the jth parameter in the interaction parameter vector of other influencing factors and the display position adjustment amount, the other influencing factors include at least one of user preference factors and interface layout constraint factors, and preferably the other influencing factors include both user preference factors and interface layout constraint factors.

[0080] In the above, step S33224 specifically includes:

[0081] The first case: If S scaled +S influence ≥S text +S action And S adjusted +S influence ≥S text +S action , then according to the preset factors, it is further determined whether to scale the digital human alone or adjust the display position of the digital human alone; when this condition is met, it means that whether the digital human is scaled alone or the display position of the digital human is adjusted alone, the text interface can have sufficient display space, and the digital human also has enough space for action changes. In this case, it is necessary to further consider other factors to determine the specific operation method. For example, the aesthetic factor can be considered, which method can make the overall presentation of the digital human and the text interface more coordinated and beautiful.

[0082] The second case: If S scaled +S influence ≥S text +S action And S adjusted +S influence <S text +S action , then scale the digital human alone; when this condition is met, it means that scaling the digital human alone can meet the display space requirements of the text interface and the space requirements for the digital human's action changes, but adjusting the display position of the digital human alone cannot meet them. So in this case, choose to scale the digital human alone. For example, the scaling operation can more effectively adjust the space occupied by the digital human to meet the display requirements of the text interface, while ensuring that the digital human has enough space for action changes.

[0083] The third case: If S scaled +S influence text +S action And S adjusted +S influence ≥S text +S action , then adjust the display position of the digital human alone; when this condition is met, it means that adjusting the display position of the digital human alone can meet the requirements, while scaling alone cannot. In this case, choose to adjust the display position of the digital human alone. For example, by adjusting the display position of the digital human, the spatial relationship between the text interface and the digital human can be better coordinated, while scaling operations can cause the digital human to deform or lose some details, affecting the user experience.

[0084] ​The fourth case: If S scaled +S influence text +S action And S adjusted +S influence <S text +S action , the digital human is scaled and the display position is adjusted at the same time; when this condition is met, it means that scaling or adjusting the display position alone cannot simultaneously meet the display space requirements of the text interface and the movement change space requirements of the digital human. Therefore, it is necessary to scale the digital human and adjust the display position at the same time to achieve the best spatial coordination effect.

[0085] Preferably, in the first case above, the preset factor is an aesthetic factor, and “further determining whether to scale the digital human alone or adjust the display position of the digital human alone according to the preset factor” specifically includes:

[0086] Calculate the aesthetic value of scaling the digital human alone according to the following digital human scaling aesthetic formula;

[0087]

[0088] Among them, B 1s To calculate the aesthetic value of scaling the digital human alone, B 0s is the beauty of the original digital human, S s is the scaling ratio of the digital human, C s is the color richness of the digital human, D s is the detail complexity of the digital human, α s , β s , γ s and δ s are weight coefficients respectively;

[0089] The aesthetic value of the digital human display position is calculated according to the following digital human display position adjustment aesthetic formula:

[0090]

[0091] Among them, B 2s The aesthetic value of the calculated adjustment of the display position of the digital person, X s Y is the horizontal position adjustment of the digital human. s is the vertical position adjustment of the digital human, F s is the integration degree of the digital person with the background, L s is the light and shadow adaptation of the digital human, ∈ s , s , η s and θ s are weight coefficients respectively; ​

[0092] B 1s and B 2s Make comparisons;

[0093] If B 1s ≥B 2s , then the digital human is scaled separately;

[0094] If B 1s <B 2s , the display position of the digital person is adjusted separately.

[0095] It should be noted that the method of the present invention can be performed by a single device, such as a computer or a server. The method of the present invention can also be applied in a distributed scenario and completed by multiple devices cooperating with each other. In the case of such a distributed scenario, one of the multiple devices can only perform one or more steps in the method of the present invention, and the multiple devices will interact with each other to complete the described method.

[0096] It should be noted that the above specific embodiments of the present invention are described. Other embodiments are within the scope of the present invention. In some cases, the actions or steps recorded in the present invention can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the process depicted in the accompanying drawings does not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0097] Based on the same purpose, corresponding to any of the above-mentioned embodiments and methods, the embodiment of the present invention also provides an interactive system based on a digital human, such as Figure 2 As shown, the interactive system includes:

[0098] An acquisition module 100, which is used to acquire voice information input by a user and capture the user's motion information, wherein the motion information includes at least one of head motion information, facial motion information and hand motion information;

[0099] A first determination module 200, which is used to determine the voice interaction mode of the digital human based on the voice information input by the user;

[0100] The second determination module 300 is used to determine the action interaction mode of the digital human and the text interface display mode based on the voice information input by the user and the captured action information of the user.

[0101] The system of the above embodiment is used to implement the corresponding method in the above embodiment, and has the beneficial effects of the corresponding method embodiment, which will not be described in detail here.

[0102] For the convenience of description, the above system is described by dividing it into various units according to its functions. Of course, when implementing the present application, the functions of each unit can be implemented in the same or multiple software and / or hardware.

[0103] Those skilled in the art will appreciate that the embodiments of the present invention may be provided as methods, systems, etc. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects.

[0104] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0105] Each embodiment of the present invention is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0106] Those skilled in the art should understand that the discussion of any of the above embodiments is merely illustrative and is not intended to imply that the scope of the present invention is limited to these examples. Under the concept of the present invention, the technical features in the above embodiments or different embodiments may be combined, the steps may be implemented in any order, and there are many other variations of different aspects of one or more embodiments of the present invention as described above, which are not provided in detail for the sake of simplicity.

[0107] While the invention has been described in conjunction with specific embodiments thereof, many alternatives, modifications and variations of these embodiments will be apparent to those skilled in the art in light of the foregoing description.

[0108] One or more embodiments of the present invention are intended to cover all such replacements, modifications and variations that fall within the broad scope of the present invention. Therefore, any omissions, modifications, equivalent replacements, improvements, etc. made within the spirit and principles of one or more embodiments of the present invention should be included in the protection scope of the present invention.

Claims

1. An interactive method based on digital human, characterized in that: The interaction method comprises: S1: Acquire voice information input by a user and capture the user's motion information, wherein the motion information includes at least one of head motion information, facial motion information and hand motion information; S2: Determine the voice interaction mode of the digital human based on the voice information input by the user; S3: Determine the action interaction mode of the digital human and the text interface display mode based on the voice information input by the user and the captured action information of the user.

2. The digital human-based interaction method according to claim 1, characterized in that: Step S2 specifically includes: S21: Acquire language type information, speech speed information, intonation information, and voice content information input by the user; S22: inputting the language type information, the speech speed information, the intonation information and the voice content information input by the user into a voice interaction matching model; S23: Determine the language type, speaking speed, intonation, emotion, background and environmental sound effects of the digital human based on the voice interaction matching model.

3. The digital human-based interaction method according to claim 2, characterized in that: The voice interaction matching model is: in, is the output digital human voice interaction feature vector, F is the voice interaction matching model function, is the function for preprocessing the language type information l, ψ(s) is the function for preprocessing the speech rate information s, χ(t) is the function for preprocessing the intonation information t, ω p (C p ) is the voice content information The dimensions of C p The function for preprocessing, x n is the weight of the language type information l in the power operation, y n is the weight of the speech rate information s in the power operation, z n is the weight of intonation information t in the power operation, ω np Voice content information The dimensions of C p The weight in the exponentiation operation, ω np is in matrix form, n and p are the level and dimension of the matrix respectively, n is the weight of the product operation results at different levels in the subsequent linear combination, δ m To adjust the weights in the linear combination of the mth level, γ k To adjust the weights in the linear combination of the kth level, β j To adjust the weights in the linear combination of the jth level, α i To adjust the weights in the linear combination of the i-th level, σ, τ, ω and ρ are nonlinear activation functions, and I is the weight parameter α i The number of layers that linearly combine and nonlinearly transform the internal results with the activation function σ, and J is the weight parameter β j The number of layers that linearly combine and nonlinearly transform the internal results with the activation function τ, and K is the weight parameter γ k The number of layers that linearly combine and nonlinearly transform the internal results with the activation function ω, M is the weight parameter δ m The number of layers where the internal results are linearly combined and nonlinearly transformed by the activation function ρ, N is the number of layers where the weight parameter n The number of layers for linearly combining the product results of each input information is the bias term.

4. The digital human-based interaction method according to claim 1, characterized in that: Step S3 specifically includes: S31: determining whether the interaction method of the digital human includes a text interface; S321: If the interaction mode of the digital human includes a text interface, further determining whether the display space of the text interface is sufficient; S331: If the display space of the text interface is sufficient, no adjustment is made to the digital human; S332: If the display space of the text interface is insufficient, the digital human is scaled and / or the display position of the digital human is adjusted so that the display space of the text interface is sufficient and the digital human has enough space for action changes.

5. The digital human-based interaction method according to claim 4, characterized in that: Step S332 specifically includes: S3321: Determine the action change space of the digital human based on the voice information input by the user and the captured action information of the user; S3322: If the display space of the text interface is insufficient, determine whether to scale the digital human alone, adjust the display position of the digital human alone, or scale and adjust the display position of the digital human at the same time based on the display space required by the text interface and the movement change space required by the digital human.

6. The digital human-based interaction method according to claim 5, characterized in that: In step S3321, the following formula is used to determine the action change space of the digital human: Among them, S action is the action variation space required by the calculated digital human, n d is the number of speech features, m d is the number of head motion features, k d is the number of facial action features, l d is the number of hand motion features, v i is the i-th speech feature, a hj is the jth head action feature, a hk is the kth head action feature, a fi is the i-th facial action feature, a fk is the kth facial action feature, a gj is the jth hand action feature, a gk is the kth hand action feature, p vhj is the association parameter vector of the jth voice information and head movement information, p vfi is the correlation parameter vector between the ith voice information and facial action information, p vgj is the correlation parameter vector of the jth speech information and hand movement information, q ahj is the j-th head motion information autocorrelation parameter vector, q afi is the autocorrelation parameter vector of the i-th facial action information, q agj is the j-th hand motion information autocorrelation parameter vector, r hfi is the interaction parameter vector between the i-th head action information and facial action information, r hgk is the interaction parameter vector of the kth head motion information and hand motion information, r fgk The interaction parameter vector between the kth facial action information and the hand action information, C is a constant adjustment factor.

7. The digital human-based interaction method according to claim 6, characterized in that: The speech features include intonation features, speaking speed features and volume features; the head movement features include horizontal head rotation angle features, vertical head rotation angle features and head movement speed features; the facial movement features include eye movement features, facial expression intensity features and facial expression change frequency features; the hand movement features include finger gesture features, hand movement amplitude features and hand movement frequency features.

8. The digital human-based interaction method according to claim 6, characterized in that: Step S3322 specifically includes: S33221: If the display space of the text interface is insufficient, the space occupied by the scaled digital human is calculated according to the following formula: S33222: Calculate the space occupied by the digital human after adjusting the display position according to the following formula: S33223: Calculate the adjustment value of comprehensive influencing factors according to the following formula: S33224: According to S scaled , S adjusted , S influence , S action and S text Determine whether to scale the digital human alone, adjust the display position of the digital human alone, or scale the digital human and adjust the display position at the same time; Among them, S scaled The scaled digital human occupies space, S adjusted To adjust the space occupied by the digital person after displaying the position, S original is the space occupied by the digital human in the initial state, S influence is the adjustment value of comprehensive influencing factors, S text is the display space required for the text interface, r d is the number of dimensions of the scaling ratio, k 1i is the i-th parameter in the digital human scaling feature vector, f 1i is the i-th parameter in the regularization parameter vector of the scaling factor, p k1i is the i-th parameter in the associated parameter vector of the scaling factor, k 2i The i-th parameter in the feature vector adjusted for the digital human display position, f 2i is the i-th parameter in the regularization parameter vector showing the position adjustment, p k2i is the i-th parameter in the associated parameter vector of the displayed position adjustment, s d is the number of directions of display position adjustment, x, y and z are the three dimensions of the coordinate vector of the initial display position of the digital human, Δx, Δy and Δz are the three dimensions of the adjustment vector of the display position of the digital human, C′ is a constant adjustment factor, t d is the total number of features of other influencing factors, g k is the kth parameter in the characteristic vector of other influencing factors, q gk1i is the i-th parameter in the interaction parameter vector of other influencing factors and scaling ratio, q gk2j is the jth parameter in the interaction parameter vector between other influencing factors and the display position adjustment amount.

9. The digital human-based interaction method according to claim 8, characterized in that: The other influencing factors include at least one of a user preference factor and an interface layout constraint factor; step S33224 specifically includes: If S scaled +S influence ≥S text +S action And S adjusted +S influence ≥S text +S action , then further determine whether to scale the digital human alone or adjust the display position of the digital human alone according to the preset factors; If S scaled +S influence ≥S text +S action And S adjusted +S influence <S text +S action , then scale the digital human alone; If S scaled +S influence text +S action And S adjusted +S influence ≥S text +S action , then adjust the display position of the digital person separately;​ If S scaled +S influence text +S action And S adjusted +S influence <S text +S action , the digital human will be scaled and the display position will be adjusted at the same time.​ 10. An interactive system based on digital human, characterized in that: The interactive system comprises: An acquisition module, which is used to acquire voice information input by a user and capture the user's motion information, wherein the motion information includes at least one of head motion information, facial motion information and hand motion information; A first determination module, which is used to determine the voice interaction mode of the digital human based on the voice information input by the user; The second determination module is used to determine the action interaction mode of the digital human and the text interface display mode based on the voice information input by the user and the captured action information of the user.

Images