Digital human interaction action generation method and device, electronic equipment and storage medium

Abstract

The invention provides a digital human interaction action generation method and device, electronic equipment and a storage medium, and relates to the technical field of data processing, and the method comprises the steps: inputting multi-modal data of interaction between a target user and a digital human into a feature extraction module of an interaction model, and obtaining multi-modal features; wherein the multi-modal features comprise a visual focus feature, a voice feature and a text emotion feature corresponding to the voice feature; the multi-modal features are input into a fusion module of the interaction model, the fusion module adjusts the initial attention score of the fusion module based on the time sequence change features of the visual focus features, and the adjusted attention score is obtained; and the fusion module performs fusion processing on the multi-modal features according to the adjusted attention score to obtain multi-modal fusion features, and generates control parameters for driving at least one body part of the digital human to move according to the multi-modal fusion features.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data processing, and in particular to a digital human interactive action generation method and device, electronic equipment and storage medium. BACKGROUND

[0002] With the rapid development of artificial intelligence technology, digital humans, as a new type of human-computer interaction interface, have been widely used in customer service, education and training, virtual anchors, intelligent assistants and other fields. Digital humans can provide users with a more natural and friendly interactive experience by simulating the appearance and behavior of real humans.

[0003] In existing digital human interaction systems, the generation of digital human actions usually adopts a preset action library or a rule-based method. These methods mainly rely on the recognition and understanding of user voice content, and trigger corresponding predefined actions by analyzing the semantic information of the voice. The generated digital human actions do not match the user's real intentions and state well, and the interaction behavior appears rigid and mechanical, lacking humanized intelligent response.

[0004] Therefore, how to obtain more lively digital human interaction actions has become a problem to be solved in the industry. SUMMARY

[0005] The present application provides a digital human interactive action generation method, device, electronic equipment and storage medium to solve the problem of how to obtain more lively digital human interaction actions in the prior art.

[0006] The present application provides a digital human interactive action generation method, comprising the following steps.

[0007] The multi-modal data of the target user interacting with the digital human is input into the feature extraction module of the interaction model to obtain multi-modal features; wherein the multi-modal features include visual focus features, voice features and text sentiment features corresponding to the voice features; The multi-modal features are input into the fusion module of the interaction model, and the initial attention score of the fusion module is adjusted based on the time sequence change feature of the visual focus features to obtain an adjusted attention score; The fusion module performs fusion processing on the multi-modal features according to the adjusted attention score to obtain multi-modal fusion features, and generates control parameters for driving the movement of at least one body part of the digital human according to the multi-modal fusion features.

[0008] According to the digital human interactive action generation method provided by the present application, the fusion module adjusts the initial attention score of the fusion module based on the time sequence change feature of the visual focus features to obtain an adjusted attention score, which comprises: determine a focus change degree multi-dimensional feature and a focus change pattern feature of the visual focus feature according to the time sequence change feature of the visual focus feature; generate an attention bias parameter according to the focus change degree multi-dimensional feature, and generate an attention scaling parameter according to the focus change pattern feature; adjust the attention score of the multi-modal data according to the attention bias parameter and the attention scaling parameter, to obtain an adjusted attention score.

[0009] According to the digital human interactive action generation method provided by the application, the focus change degree multi-dimensional feature of the visual focus feature comprises: a visual focus instantaneous speed feature, obtained by calculating the change rate of the visual focus position between adjacent time frames; a visual focus acceleration feature, obtained by calculating the change rate of the visual focus instantaneous speed; a visual focus change persistence feature, obtained by calculating the total movement distance of the visual focus within a preset time window; a visual focus change amplitude feature, obtained by calculating the variance of the visual focus position within a preset time window.

[0010] According to the digital human interactive action generation method provided by the application, the determination method of the focus change pattern feature comprises: input the time sequence change feature of the visual focus feature into a pattern classifier in the fusion module, and the pattern classifier outputs a category label of the focus change pattern feature, the category label comprises at least one of a stable fixation pattern, a smooth tracking pattern, a rapid saccade pattern, an exploratory movement pattern and a regression pattern.

[0011] According to the digital human interactive action generation method provided by the application, the attention scaling parameter is generated according to the focus change pattern feature, comprising: when the focus change pattern feature is a stable fixation pattern, a first attention scaling parameter is generated, the first attention scaling parameter is used to increase the attention weight corresponding to the visual focus feature, and to reduce the attention weight corresponding to the speech feature and the text emotion feature; when the focus change pattern feature is a rapid saccade pattern, a second attention scaling parameter is generated, the second attention scaling parameter is used to increase the attention weight corresponding to the text emotion feature; when the focus change pattern feature is a smooth tracking pattern, a third attention scaling parameter is generated, the third attention scaling parameter is used to evenly distribute the attention weights corresponding to the visual focus feature, the speech feature and the text emotion feature.

[0012] According to the digital human interactive action generation method provided by the application, the attention score of the multi-modal data is adjusted according to the attention bias parameter and the attention scaling parameter, and an adjusted attention score is obtained, comprising: Elementally adding the initial attention score and the attention bias parameter to obtain an attention score after bias processing; Multiplying the attention score after bias processing and the attention scaling parameter to obtain an adjusted attention score.

[0013] According to the digital human interactive action generation method provided by the application, the visual focus feature comprises: A focus position coordinate sequence representing the gaze point position of the target user's line of sight in the screen or three-dimensional space; A focus dwell time sequence representing the dwell time of the target user at each gaze point; A focus area distribution feature representing the distribution of the target user's line of sight in the predefined region of interest; A pupil diameter change feature reflecting the cognitive load and emotional state of the target user.

[0014] According to the digital human interactive action generation method provided by the application, the interactive model further comprises an output module; The output module is used to output the interactive video of the digital human and the target user according to the control parameter corresponding to the multi-modal fusion feature, the voice feature and the three-dimensional data of the digital human; Wherein, the control parameter comprises at least one of the following: eye rotation angle control parameter, head rotation angle control parameter, and trunk rotation angle control parameter.

[0015] According to the digital human interactive action generation method provided by the application, before the step of inputting the multi-modal data of the target user interacting with the digital human into the feature extraction module of the interactive model, the method further comprises: The multi-modal data sample carrying the text emotional label, the focus change degree multi-dimensional feature label, the focus change mode feature label and the control parameter label is used as a training sample to obtain a plurality of training samples; Based on the plurality of training samples, the feature extraction module and the fusion module of the interactive model are trained, and the training is stopped when the preset training target is reached, and a trained interactive model is obtained.

[0016] The application also provides a digital human interactive action generation device, comprising: The input module is used for inputting multi-modal data of the target user interacting with the digital human into a feature extraction module of an interaction model to obtain multi-modal features, wherein the multi-modal features include visual focus features, voice features and text emotion features corresponding to the voice features. The adjusting module is used for inputting the multi-modal features into a fusion module of the interaction model, and adjusting initial attention scores of the fusion module based on time sequence change features of the visual focus features to obtain adjusted attention scores. The fusion module is used for performing fusion processing on the multi-modal features according to the adjusted attention scores to obtain multi-modal fusion features, and generating control parameters for driving at least one body part of the digital human to move according to the multi-modal fusion features.

[0017] The application further provides an electronic device including a memory, a processor and a computer program stored in the memory and executable on the processor, and the processor implements the digital human interaction action generation method according to any one of the above when executing the computer program.

[0018] The application further provides a non-transitory computer readable storage medium having a computer program stored thereon, and the computer program is executable on a processor to implement the digital human interaction action generation method according to any one of the above.

[0019] The application further provides a computer program product including a computer program, and the computer program is executable on a processor to implement the digital human interaction action generation method according to any one of the above.

[0020] The digital human interaction action generation method, device, electronic device and storage medium provided by the application can dynamically adjust the initial attention scores by using the time sequence change features of the visual focus features through the fusion module, and this adjustment mechanism can re-allocate the importance weights of different modal features according to the real-time change mode of the user's visual line, and this dynamic adjustment ensures the adaptability of the multi-modal feature fusion. The multi-modal fusion features generated according to the adjusted attention scores can more accurately reflect the real intention, interest focus and cognitive state of the user in the current interaction process. The digital human body part actions driven by the control parameters generated based on the multi-modal fusion features are no longer simple, single semantic mechanical responses, but natural behaviors highly synchronized and coordinated with the changes of the user's visual attention, voice emotion and the like, so that a more natural and coordinated human-computer interaction effect is realized, and the expressiveness and user experience of the digital human in a complex interaction scene are significantly improved. BRIEF DESCRIPTION OF DRAWINGS

[0021] In order to more clearly illustrate the technical solutions in the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or prior art description. Obviously, the drawings in the following description are some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative effort.

[0022] Figure 1 is a flowchart of the digital human interactive action generation method provided by the present application. Figure 2 is a structural schematic diagram of the digital human interactive action generation device provided by the present application. Figure 3 is a structural schematic diagram of the electronic device provided by the present application. DETAILED DESCRIPTION

[0023] In order to make the objects, technical solutions and advantages of the present application clearer, the technical solutions in the present application will be described clearly and completely below in combination with the drawings in the present application. Obviously, the described embodiments are some embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative effort are within the protection scope of the present application.

[0024] Figure 1 is a flowchart of the digital human interactive action generation method provided by the present application, as shown in Figure 1 , the method comprises the following: Step 110, inputting the multi-modal data of the target user interacting with the digital human into a feature extraction module of an interaction model to obtain multi-modal features; wherein the multi-modal features comprise visual focus features, voice features and text emotion features corresponding to the voice features. In the present application, the digital human refers to a virtual character image generated by computer graphics technology, which has appearance characteristics similar to real humans, including face, head, torso and other body parts, and can be driven by control parameters to move each body part to realize visual interaction with the user. The digital human can be a two-dimensional virtual image or a three-dimensional modeled solid figure, and its forms of expression include but are not limited to virtual customer service, virtual anchor, virtual teacher, virtual assistant, etc.

[0025] The target user refers to a real user who interacts with the digital human in real time. The user exchanges information and communicates emotions with the digital human through voice input, visual fixation, etc. The target user can be a single user or a main interactor in multiple users.

[0026] Multi-modal data refers to various types of raw data generated during the interaction between the user and the digital human, including but not limited to: user visual focus data collected through eye tracking devices or depth cameras, user voice data collected through microphones, text data converted from voice through natural language processing technology, etc. These multi-modal data reflect the user's interaction state and intention from different dimensions.

[0027] Multi-modal features refer to high-dimensional feature representations extracted from multi-modal data, which can better represent the user's interaction state. Specifically, it includes three types of features: visual focus features, voice features, and text sentiment features.

[0028] Visual focus features are features extracted from user visual focus data, reflecting the allocation of user visual attention, which can include position information, dwell time information, and movement trajectory information of visual focus.

[0029] Voice features are acoustic features extracted from user voice signals, including but not limited to Mel-frequency cepstral coefficients, fundamental frequency, energy, speech rate, and other acoustic parameters. Acoustic features reflect the acoustic characteristics of user speech.

[0030] Text sentiment features are features obtained by performing sentiment analysis on the text content obtained through voice recognition, reflecting the emotional tendency of the user's expression content, such as positive, negative, neutral, and other emotional states, as well as specific emotional categories such as happiness, sadness, anger, surprise, etc.

[0031] More specifically, the facial and eye images of the target user are captured in real time through a camera or an eye tracker, and their voice is recorded through a microphone.

[0032] The feature extraction module receives these raw data. For visual data, the module extracts visual focus features through eye tracking algorithms (such as corneal reflection-based or deep learning-based eye feature point detection algorithms).

[0033] For voice data, on the one hand, the module extracts its acoustic voice features; on the other hand, the voice data is sent to a voice-to-text sub-module to obtain text, and then the text is sent to a text sentiment analysis sub-module to obtain text sentiment features, obtaining multi-modal features including visual focus features, voice features, and text sentiment features.

[0034] Step 120, inputting the multi-modal features into the fusion module of the interaction model, adjusting the initial attention score of the fusion module based on the time sequence variation characteristics of the visual focus features to obtain an adjusted attention score; In this application, the fusion module is the core component in the interaction model, responsible for effectively fusing features of different modalities. This module uses an attention mechanism, which can dynamically adjust the importance weight of different features.

[0035] The time sequence change feature refers to the dynamic characteristics of the visual focus feature changing over time, including the position change, speed change, acceleration change, and other time dimension information of the visual focus between consecutive time frames.

[0036] The initial attention score is the original weight score given to each modality feature by the fusion module before dynamic adjustment. These scores are usually obtained through standard attention calculation methods.

[0037] In this application, the fusion module first calculates the initial attention score of each modality feature, which can be achieved through standard self-attention mechanism or cross-attention mechanism.

[0038] Then, the fusion module analyzes the time sequence change feature of the visual focus feature, such as calculating the change amount and change speed of the focus position between consecutive frames. Based on these time sequence change features, the fusion module generates adjustment parameters to dynamically adjust the initial attention score.

[0039] For example, when the user's gaze is detected to move quickly, it indicates that the user may be looking for information or in an unstable state, in which case the weight of the visual focus feature can be reduced, and the weights of the speech and emotion features can be increased; when the user's gaze is stable and fixed on a certain area, it indicates that the user is interested in the content of that area, in which case the weight of the visual focus feature can be increased.

[0040] Step 130, the fusion module fuses the multi-modal features according to the adjusted attention scores to obtain multi-modal fusion features, and generates control parameters for driving the motion of at least one body part of the digital person based on the multi-modal fusion features.

[0041] In this application, the control parameter is a numerical parameter used to drive the motion of the digital person's body parts. These parameters can be directly input into the digital person rendering engine to control the digital person's action performance.

[0042] The fusion module uses the adjusted attention scores to weight and fuse the multi-modal features to obtain comprehensive multi-modal fusion features. This fusion feature is a high-dimensional vector that contains information from multiple dimensions of vision, speech, and emotion. Subsequently, the multi-modal fusion features are converted into control parameters through a decoding network.

[0043] The control parameters can include various types, such as parameters for controlling eye rotation, such as horizontal rotation angle, vertical rotation angle, parameters for controlling head movement, such as nodding, shaking, tilt angle, parameters for controlling body posture, such as forward inclination, backward inclination angle, etc. These control parameters can be continuous numerical values or discrete action labels.

[0044] Through the method of the embodiment, digital human interactive action generation based on multi-modal feature fusion is realized, and in particular, the attention score is dynamically adjusted through the time sequence change of the visual focus feature, so that the fusion process can adaptively respond to changes in user state, and the generated control parameters can drive the digital human to produce natural actions that match the user interaction state, significantly improving the interaction realism of the digital human and the user experience.

[0045] In an optional embodiment, when multiple users want to interact with a digital human, through multi-user management, the digital human can orderly and naturally interact with multiple users.

[0046] First, through multiple sensors, an eye tracker, a depth camera or a camera simultaneously collects data in the interactive environment, detects and identifies all users who want to interact with the digital human, each user is assigned a unique user identifier, and an independent data collection channel is established for each detected user to collect multi-modal data of each user.

[0047] Specifically, different users can be distinguished through face recognition technology, different voice sources can be distinguished through voiceprint recognition technology, and for visual focus features, multiple eye tracking devices or image processing technology can be used to track the gaze of each user.

[0048] The detected multiple users are added to the interaction queue according to preset queue management rules, which can be first-come-first-served rules, priority sorting rules, attention intensity rules, or combinations thereof.

[0049] The first-come-first-served rule queues according to the time sequence of the user's interaction request, the priority sorting rule assigns priority according to the user's specific attributes, the attention intensity rule calculates the user's attention intensity to the digital human by analyzing the visual focus feature of each user, and the mixed rule combines multiple factors for comprehensive sorting.

[0050] In another optional embodiment, the preset queue management rules can also calculate the semantic relevance of the waiting user to the current interaction theme by analyzing the voice content or initial interaction intention of the waiting user, and users with high semantic relevance can obtain higher sorting priority.

[0051] The calculation of semantic correlation can be realized in various ways, including extracting semantic vectors using a pre-trained language model and calculating cosine similarity, determining theme relevance through keyword matching, analyzing the correlation distance between concepts using a knowledge graph, etc.

[0052] For example, in an educational scenario, a virtual teacher faces multiple students, and students who raise their hands will be given higher priority. At the same time, the system will also consider the waiting time of each student to avoid some students waiting for too long. In the virtual customer service scenario in the bank hall, VIP customers may have higher basic priority, but the priority of ordinary customers will gradually increase after waiting for a certain period of time.

[0053] Then, a user at the head of the user queue can be selected as the current interactive user, and the state identifier of the user is updated to be in interaction, and the state identifiers of other users remain or are updated to be in waiting.

[0054] After determining the current interactive user, an interaction switching signal is generated to trigger the digital person to perform a corresponding turning action. If the new interactive user is located on the left side of the digital person, the digital person will perform a left turning action of the head and torso, and the eyeball will also turn to the user to establish visual contact.

[0055] The multi-modal data of the current interactive user is input into the feature extraction module of the interaction model to extract the visual focus feature, speech feature and text emotion feature of the user. The fusion module adjusts the attention score based on the time sequence change feature of the visual focus feature of the current user, performs multi-modal fusion, generates control parameters for the user, and the control parameters drive the digital person to perform corresponding interaction actions.

[0056] In the embodiments of the present application, when processing the data of the current user, the data of other waiting users can be cached at the same time for preprocessing and feature extraction to reduce the processing delay when switching users subsequently.

[0057] The progress state of the current interaction is continuously monitored to determine whether the interaction is completed through a time threshold, interaction content or task completion.

[0058] In task-oriented interaction, when the specific needs of the user are met, the interaction is determined to be completed. For example, in an exhibition hall scenario, when a visitor asks about the information of a certain exhibit and expresses thanks, it is determined that the user interaction is completed, and the digital person will make a goodbye action of smiling and nodding to the visitor, and then return to a neutral posture, and then turn to the next waiting visitor, establish eye contact with him and make a welcoming expression, neither appearing hurried nor making the waiting person feel unhappy.

[0059] When it is determined that the current user interaction is completed, switching transition action control parameters can be generated, which control the digital human to perform natural transition actions, including a farewell action to the current user, a neutral reset action, and a greeting action to the next user. The design of the transition action takes into account the continuity and naturalness of the interaction.

[0060] The state identifier of the current user is updated to indicate that the interaction is completed, and the current user is removed from the queue or moved to the end of the queue. The next user is selected from the queue as a new current interaction user.

[0061] At the same time, on the other hand, for users in the waiting state, the digital human also generates appropriate feedback to maintain the user's sense of participation.

[0062] When the waiting user gazes at the digital human for a long time, the digital human can occasionally make brief eye contact with the user or make a slight nod to indicate that the user has been noticed. The digital human can inform the user of their position in the queue and the estimated waiting time through the digital human's voice or interface display. When detecting that the waiting user appears impatient, the digital human can make a soothing gesture or expression.

[0063] For example, in a hospital guidance scenario, when multiple patients simultaneously consult a virtual guidance officer, the waiting patients will see the digital human occasionally looking at them with concerned eyes and hear: Please wait a moment, I will serve you soon. This feedback mechanism makes the waiting user feel noticed and reduces anxiety during the waiting process.

[0064] In the present application, the digital human can orderly and efficiently interact in a multi-user scenario, significantly improving the practicality of the digital human in public places and multi-person environments, and is particularly suitable for application scenarios such as exhibition halls, banks, schools, and other scenarios that require service to multiple users.

[0065] Optionally, the fusion module adjusts the initial attention score of the fusion module based on the time sequence variation feature of the visual focus feature to obtain an adjusted attention score, including: According to the time sequence variation feature of the visual focus feature, a focus variation degree multi-dimensional feature and a focus variation pattern feature of the visual focus feature are determined; According to the focus variation degree multi-dimensional feature, an attention bias parameter is generated, and according to the focus variation pattern feature, an attention scaling parameter is generated; According to the attention bias parameter and the attention scaling parameter, the attention score of the multi-modal data is adjusted to obtain an adjusted attention score.

[0066] In this application, the focus change degree multi-dimensional feature refers to a set of features quantitatively describing the degree of change of visual focus from multiple dimensions, which reflects the dynamic changes of the user's visual line from different angles.

[0067] The focus change pattern feature refers to the category feature obtained by classifying the change pattern of visual focus, which reflects the overall pattern of the user's visual line movement.

[0068] More specifically, first, according to the time sequence change feature of the visual focus feature, the focus change degree multi-dimensional feature and the focus change pattern feature of the visual focus feature are determined.

[0069] The fusion module internally contains two parallel analysis units. One unit is responsible for calculating the degree of focus change, such as calculating the speed, acceleration, etc. of the visual line movement, and outputting a set of numerical values as the focus change degree multi-dimensional feature.

[0070] Another unit is responsible for identifying the pattern of focus change, such as determining whether the user's visual line is stable fixation, rapid saccade, or smooth pursuit of an object, and outputting a category label as the focus change pattern feature.

[0071] First, the time series of the visual focus feature is analyzed, and the focus change degree multi-dimensional feature is extracted. These features can include but are not limited to: average speed, maximum speed, acceleration, movement distance, movement frequency, etc. of the focus movement. Each feature quantifies the degree of focus change from a specific angle.

[0072] At the same time, the system determines the focus change pattern feature through pattern recognition methods. Clustering algorithms, classifiers or rule matching methods can be used to classify the focus change pattern into predefined pattern categories.

[0073] In this application, the attention bias parameter is a set of parameters used to adjust the baseline value of the attention score, which changes the absolute size of the attention score through addition operation.

[0074] The attention scaling parameter is a set of parameters used to adjust the relative proportion of the attention score, which changes the relative importance between different modal features through multiplication operation.

[0075] Based on the focus change degree multi-dimensional feature, the attention bias parameter is generated through a mapping function or neural network. For example, when the focus change degree is large, a larger bias value is generated to overall increase or decrease the attention baseline of certain modal.

[0076] Based on the focus change pattern feature, the attention scaling parameter is generated through a lookup table or conditional judgment. Different patterns correspond to different scaling strategies, which are used to adjust the relative weight proportion of each modal feature.

[0077] Finally, the initial attention score is adjusted according to the attention bias parameter and the attention scaling parameter to obtain an adjusted attention score.

[0078] By the scheme disclosed in the embodiment, the time sequence change of the visual focus is decomposed into two dimensions of change degree and change mode, and is respectively used to generate bias and scaling parameters, so that double and fine regulation and control of the attention score are realized. Compared with a single adjustment strategy, this way can more accurately capture the complex intention behind the user visual behavior, so that the multi-modal fusion process is more intelligent and context adaptive.

[0079] Optionally, the focus change degree multi-dimensional feature of the visual focus feature includes: The instantaneous speed feature of the visual focus is obtained by calculating the change rate of the visual focus position between adjacent time frames. The acceleration feature of the visual focus is obtained by calculating the change rate of the instantaneous speed of the visual focus. The change persistence feature of the visual focus is obtained by calculating the total movement distance of the visual focus in a preset time window. The change amplitude feature of the visual focus is obtained by calculating the variance of the visual focus position in a preset time window.

[0080] In the present application, the instantaneous speed feature of the visual focus reflects the speed of the visual focus in a short time.

[0081] Specifically, the coordinates of the focus positions of two adjacent time frames can be obtained, the Euclidean distance between the two points is calculated, and then the instantaneous speed is obtained by dividing the time interval.

[0082] The instantaneous speed can be a scalar value, or can be decomposed into speed components in the horizontal and vertical directions.

[0083] The acceleration feature of the visual focus reflects the change rate of the speed of the visual focus. By calculating the focus positions of three or more consecutive time frames, the instantaneous speed at each time is first calculated, and then the change rate of the speed is calculated to obtain the acceleration. The acceleration feature can capture the mutation of the visual line movement.

[0084] The change persistence feature of the visual focus reflects the continuous movement of the visual focus in a certain time.

[0085] A time window (such as 1 second or 2 seconds) is set, the total distance is calculated, and the total movement distance is obtained. This feature can distinguish between short jumps and continuous saccades.

[0086] The variation amplitude feature of the visual focus reflects the discrete degree of the visual focus position. In a preset time window, the variance of all focus positions relative to the center position is calculated, or the standard deviation of the focus distribution is calculated. This feature can represent the concentration or dispersion degree of the visual line.

[0087] In the embodiments of the present application, the variation degree of the visual focus is quantified from multiple physical dimensions such as speed, acceleration, persistence and amplitude, which can more comprehensively and meticulously depict the dynamic characteristics of the user's visual attention, provide rich and reliable basis for generating accurate attention bias parameters, and thus improve the judgment accuracy of the model on the user's cognitive state.

[0088] Optionally, the determination method of the focus variation mode feature comprises: inputting the time sequence variation feature of the visual focus feature into a mode classifier in the fusion module, and the mode classifier outputs a class label of the focus variation mode feature, The class label comprises at least one of a stable fixation mode, a smooth tracking mode, a rapid saccade mode, an exploratory movement mode and a regression mode.

[0089] In the present application, the mode classifier is a subcomponent in the fusion module, which can be a classification model based on machine learning, such as a support vector machine, a decision tree, a neural network classifier, etc., and is used to identify the movement mode of the visual focus.

[0090] In the present application, the time sequence variation feature of the visual focus feature is input into the mode classifier, which can be a pre-trained model, and learns the feature representation of different modes through a large amount of labeled data.

[0091] The mode classifier analyzes the input time sequence feature and classifies it into the most matching one of the above modes, and outputs the corresponding class label.

[0092] The class label output by the classifier can be a single class or a probability distribution of multiple classes.

[0093] In the present application, the stable fixation mode refers to that the user's visual line remains in a certain fixed area or target for a long time, and the focus position changes little. This mode usually occurs when the user is observing or thinking seriously.

[0094] The smooth tracking mode refers to that the user's visual line smoothly follows a moving target, and the focus trajectory presents continuous and smooth characteristics. This mode is commonly seen when the user tracks dynamic content.

[0095] The rapid saccade mode refers to that the user's visual line rapidly jumps between different positions, and the focus trajectory presents discontinuous and jumping characteristics. This mode usually occurs when the user searches for information or browses content.

[0096] Exploratory movement mode refers to the user's gaze moving erratically within a certain area, which is neither a stable gaze nor a purposeful scan.

[0097] The regression pattern refers to the user's gaze returning to its original position after moving, forming a back-and-forth motion trajectory.

[0098] The pattern classification method in this embodiment can categorize complex eye movement patterns into a limited number of typical patterns, which facilitates the adoption of corresponding attention adjustment strategies based on different patterns.

[0099] Optionally, attention scaling parameters are generated based on the focus change pattern features, including: When the focus change pattern feature is a stable gaze pattern, a first attention scaling parameter is generated. The first attention scaling parameter is used to increase the attention weight corresponding to the visual focus feature and decrease the attention weight corresponding to the speech feature and the text sentiment feature. When the focus change pattern feature is a fast scanning pattern, a second attention scaling parameter is generated. The second attention scaling parameter is used to increase the attention weight corresponding to the text sentiment feature. When the focus change pattern feature is a smooth tracking mode, a third attention scaling parameter is generated. The third attention scaling parameter is used to evenly distribute the attention weights corresponding to the visual focus feature, speech feature and text sentiment feature.

[0100] In this application, when the focus change pattern feature is identified as a stable gaze pattern, the system generates a first attention scaling parameter.

[0101] The first attention scaling parameter is characterized by assigning a larger scaling factor (such as 1.5 or 2.0) to the dimension corresponding to the visual focus feature, and a smaller scaling factor (such as 0.5 or 0.7) to the dimension corresponding to the speech feature and the text sentiment feature.

[0102] When a user is looking steadily, their visual focus information best reflects their attention level and should be given higher weight during fusion.

[0103] When the focus change pattern feature is identified as a rapid saccade pattern, a second attention scaling parameter is generated.

[0104] The second attention scaling parameter increases the scaling factor of the text sentiment features (e.g., 1.8), because rapid scanning often indicates that the user is looking for information, at which point the sentiment information of the language content is more important.

[0105] Because visual focal features change too rapidly, their instantaneous position reference value decreases, so their scaling factor can be appropriately reduced.

[0106] When the focus change mode feature is identified as the smooth tracking mode, a third attention scaling parameter is generated.

[0107] The third attention scaling parameter gives similar scaling coefficients to the three kinds of modal features (such as 1.0 or in the range of 0.9-1.1), realizing balanced feature fusion. This is because smooth tracking indicates that the user is in a normal interaction state, and various modal information is of reference value.

[0108] The attention scaling parameter can be a vector whose dimension corresponds to the feature dimension, or a matrix to realize more complex feature interaction adjustment.

[0109] More specifically, when the visual focus is quickly saccadic, attention is more focused on visual signals, and the main data source in the training set label is a video clip of driving the eyeball and head to quickly follow, a small turning gait close to the interlocutor, and other large-range, large-scale, and long-time stable movements. The specific performance is that the change amplitude in the eye rotation, head rotation, and trunk rotation sequence is larger.

[0110] When the visual focus is stable, attention is more focused on speech and emotional signals, and the main data in the training set label comes from eyelid blinking, eyeball micro-tremor, listening-type head micro-motion, unconscious micro-shaking of the head with the accent in the speech signal, chest breathing fluctuation, and emotionalized trunk posture, etc. Small-range, small-scale, and instantaneous details of the video. The specific performance is that the change amplitude in the eye rotation, head rotation, and trunk rotation sequence is smaller.

[0111] The fusion process realizes dynamic adjustment of the context within the entire window length, which is the core of avoiding the mechanical and fragmented feeling of the action.

[0112] Through the mode-dependent scaling parameter generation method of the embodiment, the function of adaptively adjusting the multi-modal feature weight according to the user's visual motion mode is realized, so that the fusion result is more in line with the characteristics of different interaction scenarios.

[0113] Optionally, according to the attention bias parameter and the attention scaling parameter, the attention score of the multi-modal data is adjusted to obtain an adjusted attention score, including: Element-wise addition of the initial attention score and the attention bias parameter is performed to obtain an attention score after bias processing; The attention score after bias processing is multiplied by the attention scaling parameter to obtain the adjusted attention score.

[0114] In the present application, the element addition operation refers to the addition operation of elements at corresponding positions of two vectors or matrices of the same dimension. In the present embodiment, the initial attention score and the attention bias parameter have the same dimension structure, and the bias adjustment is realized by element addition.

[0115] The element multiplication operation refers to the multiplication operation of elements at corresponding positions of two vectors or matrices of the same dimension.

[0116] More specifically, in the present embodiment, the fusion module adopts the Conformer architecture to perform self-attention calculation on the multi-modal feature FeatureCombined. The specific attention adjustment process is as follows: The FeatureCombined is transformed by linear transformation to generate Query (Q) and Key (K) matrices, and the basic attention score is obtained by standard dot product attention calculation: The gaze variability feature (GazeVariability) is generated by the projection layer to generate the attention bias AttentionBias; The gaze pattern feature (GazePattern) is generated by the projection layer to generate the attention scale AttentionScale; The value range of AttentionScale is between [0, 1], which is constrained by the Sigmoid activation function; the final attention score is calculated by applying the adjustment formula The present embodiment adopts the following adjustment formula: AttentionScore = (BaseScore + AttentionBias) × (1 - AttentionScale) The adjusted AttentionScore is normalized by Softmax: AttentionWeights = Softmax(AttentionScore) The normalized weights are used to weight and sum the Value matrix V to obtain the final multi-modal fusion feature: FeatureAttention = AttentionWeights × V; In the present application, the scheme disclosed in the present embodiment utilizes a specific formula combining addition bias and multiplication suppression to realize fine and dynamic adjustment of the attention score, so that the multi-modal fusion process can more intelligently adapt to different visual behavior patterns of users.

[0117] Optionally, the visual focus feature comprises: a sequence of focus position coordinates representing the gaze point position of the target user's visual line in the screen or three-dimensional space; a focus dwell time sequence representing the target user's dwell time at each fixation point; a focus region distribution feature representing the distribution of the target user's gaze in the predefined region of interest; a pupil diameter change feature reflecting the cognitive load and emotional state of the target user.

[0118] In this application, the focus position coordinate sequence is the core component of the visual focus feature, which records the spatial position information of the user's gaze in the time dimension.

[0119] In a two-dimensional screen scenario, the coordinate sequence is represented as a sequence of (x, y, t), where x and y are screen coordinates, and t is a timestamp; in a three-dimensional space scenario, the coordinate sequence is represented as a sequence of (x, y, z, t). The coordinates can be absolute coordinates or relative coordinates relative to a specific reference point.

[0120] The focus dwell time sequence records the user's dwell time at each fixation point. When the movement range of the focus in a certain region is less than a preset threshold, it is considered as a fixation, and the time from entering to leaving the region is calculated as the dwell time. The dwell time can reflect the user's attention to different content.

[0121] The focus region distribution feature describes the distribution of the user's gaze in the predefined region of interest.

[0122] The interactive interface can be divided into multiple regions, such as the digital human face region, the background region, the text display region, etc., and the number of visits, the dwell time ratio, etc. of the focus in each region are counted.

[0123] The pupil diameter change feature obtains the time series data of the user's pupil diameter through eye tracking equipment. The change of the pupil diameter is closely related to the user's cognitive load and emotional arousal. Pupil dilation may indicate interest or surprise, and pupil constriction may indicate fatigue or boredom.

[0124] The visual focus feature comprehensively describes the user's visual attention state from different angles, providing rich visual information for subsequent multi-modal fusion.

[0125] Through the multi-dimensional visual focus feature provided by this embodiment, the user's visual attention pattern can be fully captured, providing an important basis for generating digital human actions that coordinate with the user's gaze interaction.

[0126] Optionally, the interaction model further includes an output module; The output module is configured to output an interaction video of the digital human and the target user according to the control parameters corresponding to the multi-modal fusion feature, the voice feature, and the three-dimensional data of the digital human. The control parameters include at least one of the following: eye rotation angle control parameters, head rotation angle control parameters, and torso rotation angle control parameters.

[0127] In this application, the interaction model can further include an output module in addition to the feature extraction module and the fusion module. This module is located at the end of the entire processing flow and is responsible for converting abstract fusion features into the final interactive product that can be viewed.

[0128] The three-dimensional data of the digital human includes three-dimensional model data of the digital human, such as mesh models, skeleton binding information, and texture maps. These data define the appearance and movable range of the digital human.

[0129] The output module receives control parameters corresponding to the multi-modal fusion features. These control parameters include: Eye rotation angle control parameters, including horizontal rotation angle (yaw angle) and vertical rotation angle (pitch angle), control the rotation of the digital human's eyes to achieve eye contact with the user. The angle range is usually ±30 degrees horizontally and ±20 degrees vertically.

[0130] Head rotation angle control parameters, including the rotation angles of the three degrees of freedom of the head's yaw, pitch, and roll, control the posture of the digital human's head. These parameters enable the digital human to make nodding, shaking, and tilting actions.

[0131] Torso rotation angle control parameters control the inclination and rotation of the upper body of the digital human, including forward leaning, backward leaning, and left and right tilting action parameters.

[0132] The output module also receives speech features to generate the digital human's lip animation and achieve audio-visual synchronization. By analyzing the phoneme information in the speech features, the digital human's lip movement is driven.

[0133] The output module applies these control parameters to the three-dimensional data of the digital human, generates animation frames through a graphics rendering engine, and finally outputs continuous interactive videos.

[0134] The interactive video is the final output of the system, showing the dynamic picture of the digital human performing corresponding actions according to the control parameters. It can be a real-time rendered video stream or a pre-rendered video file.

[0135] Through the output module of this embodiment, the conversion from abstract control parameters to specific visual presentation is realized, enabling users to intuitively see the response actions of the digital human, completing the entire interactive loop.

[0136] Optionally, before the step of inputting the multi-modal data of the target user interacting with the digital human into the feature extraction module of the interaction model, the method further includes: The multi-modal data sample carrying the text sentiment label, the focus change degree multi-dimensional feature label, the focus change mode feature label and the control parameter label is taken as a training sample, and a plurality of training samples are obtained; Based on the plurality of training samples, the feature extraction module and the fusion module of the interaction model are trained, and in the case of reaching a preset training target, the training is stopped, and a trained interaction model is obtained.

[0137] In the present application, the training sample is a basic data unit for model training, each training sample contains a set of synchronously collected multi-modal input data and a plurality of labels corresponding thereto, wherein the text sentiment label labels the emotional category contained in the user voice content, the focus change degree multi-dimensional feature label labels the quantitative index of the user visual focus change, the focus change mode feature label classifies the overall motion mode of the user visual focus, and the control parameter label represents the quantitative parameter of the body action response of the digital human under the specific user input.

[0138] In order to obtain high-quality training data, the present embodiment adopts a two-person interactive motion capture scheme for data acquisition. In the experiment, participant A plays the role of the target user, wears a high-frequency eye movement tracking device to record visual focus data, and collects voice data through a high-fidelity microphone. Participant B plays the role of a natural reaction real model, wearing a professional optical motion capture device on the whole body or upper body. The system presents a preset interactive task to participant A, such as describing a complex picture, answering an open question, or watching a short film with fluctuating emotions. When participant A performs the task, his gaze movement and voice are recorded in real time. Participant B makes natural non-verbal body reactions according to the observed real-time state of participant A, including following nodding, doubt tilting, understanding body leaning forward, and eye contact, etc. These natural reactions are accurately recorded by the motion capture system as time series data containing three-dimensional space coordinates or joint rotation angles of head, torso and eyeball movement. All data streams are marked with synchronous time stamps to ensure that the multi-modal data can be accurately aligned.

[0139] The collected raw data needs to be processed offline to construct training samples. For each time slice, the eye movement data and speech data of participant A are taken as model inputs, the motion capture data of participant B is processed into control parameter labels matching the digital human skeletal model through inverse kinematics or coordinate system conversion, these control parameters include time series of eyeball rotation angle, head rotation angle, and trunk rotation angle, representing the learning goal of digital human action, at the same time, the speech of participant A is converted into text through automatic speech recognition technology, and then the text emotion label is generated by manual annotation or pre-trained emotion classification model, the eye movement data of participant A is analyzed, the focus change degree multi-dimensional feature label is obtained by calculating the change rate and variance of the gaze position, and the focus change mode feature label is generated by classifying the gaze movement mode into stable fixation, rapid saccade, etc. through pattern recognition algorithm or manual annotation.

[0140] Based on the constructed large-scale training sample set, the interaction model is trained, the model receives the multi-modal data of the user and tries to predict the control parameters as close as possible to the control parameter labels, the intermediate modules inside the model use the text emotion label, the focus change degree multi-dimensional feature label and the focus change mode feature label for multi-task learning or auxiliary supervision, which helps the model better understand the internal meaning of different modal features and accelerate convergence to improve performance.

[0141] The training adopts supervised learning, and the internal parameters of the feature extraction module and the fusion module are constantly adjusted through the back propagation algorithm and the optimizer to minimize the loss function between the predicted value and the label value, when the preset training conditions are met, such as the training reaches the specified number of rounds, the validation set loss no longer decreases, or the generated action reaches the preset standard, the training is stopped and the model parameters are saved, and finally the trained interaction model capable of generating natural interactive action is obtained.

[0142] Through the data collection, labeling and training methods described in this embodiment, the interaction model can learn the subtle multi-modal interaction patterns between people in the real world, so that the generated digital human action is truly coordinated with the real-time state of the user, forming a lively natural response, greatly improving the immersion and realism of human-computer interaction.

[0143] The digital human interactive action generation device provided by the present application is described below, and the digital human interactive action generation device described below can be correspondingly referred to the digital human interactive action generation method described above.

[0144] Figure 2 The structure diagram of the digital human interactive action generation device provided by the present application is shown in Figure 2 , which includes: The input module 210 is configured to input multi-modal data of interaction between a target user and a digital human into a feature extraction module of an interaction model, to obtain multi-modal features; wherein the multi-modal features comprise: visual focus features, speech features, and text sentiment features corresponding to the speech features; The adjustment module 220 is configured to input the multi-modal features into a fusion module of the interaction model, and adjust an initial attention score of the fusion module based on a time sequence change feature of the visual focus features, to obtain an adjusted attention score; The fusion module 230 is configured to perform fusion processing on the multi-modal features according to the adjusted attention score, to obtain multi-modal fusion features, and generate control parameters for driving at least one body part of the digital human to move, according to the multi-modal fusion features.

[0145] Optionally, the apparatus is further configured to: determine a focus change degree multi-dimensional feature and a focus change pattern feature of the visual focus features according to the time sequence change feature of the visual focus features; generate an attention bias parameter according to the focus change degree multi-dimensional feature, and generate an attention scaling parameter according to the focus change pattern feature; adjust the attention score of the multi-modal data according to the attention bias parameter and the attention scaling parameter, to obtain an adjusted attention score.

[0146] Optionally, the apparatus is further configured to: a momentary speed feature of the visual focus is obtained by calculating a change rate of the visual focus position between adjacent time frames; an acceleration feature of the visual focus is obtained by calculating a change rate of the momentary speed of the visual focus; a change persistence feature of the visual focus is obtained by calculating a total moving distance of the visual focus within a preset time window; a change amplitude feature of the visual focus is obtained by calculating a variance of the visual focus position within a preset time window.

[0147] Optionally, the apparatus is further configured to: input the time sequence change feature of the visual focus features into a pattern classifier in the fusion module, and the pattern classifier outputs a category label of the focus change pattern feature, the category label comprises at least one of a stable fixation pattern, a smooth tracking pattern, a rapid saccade pattern, an exploratory movement pattern, and a regressive pattern.

[0148] Optionally, the apparatus is further configured to: when the focus change mode feature is the stable fixation mode, a first attention scaling parameter is generated, the first attention scaling parameter is used to increase the attention weight corresponding to the visual focus feature, and to reduce the attention weight corresponding to the speech feature and the text emotion feature; when the focus change mode feature is the rapid saccade mode, a second attention scaling parameter is generated, the second attention scaling parameter is used to increase the attention weight corresponding to the text emotion feature; when the focus change mode feature is the smooth pursuit mode, a third attention scaling parameter is generated, the third attention scaling parameter is used to evenly distribute the attention weights corresponding to the visual focus feature, the speech feature and the text emotion feature.

[0149] Optionally, the device is also used for: elementally adding the initial attention score and the attention bias parameter to obtain a bias-processed attention score; multiplying the bias-processed attention score and the attention scaling parameter to obtain an adjusted attention score.

[0150] The interaction model further includes an output module; The output module is used to output an interaction video of the digital human and the target user according to the control parameter corresponding to the multi-modal fusion feature, the speech feature and the three-dimensional data of the digital human; The control parameter includes at least one of the following: an eyeball rotation angle control parameter, a head rotation angle control parameter, and a trunk rotation angle control parameter.

[0151] The device is also used for: obtaining a plurality of training samples by taking multi-modal data samples carrying a text emotion label, a focus change degree multi-dimensional feature label, a focus change mode feature label and a control parameter label as a training sample; training the feature extraction module and the fusion module of the interaction model based on the plurality of training samples, stopping the training when a preset training target is reached, and obtaining a trained interaction model.

[0152] In this application, the initial attention score is dynamically adjusted using the temporal variation characteristics of visual focus features through a fusion module. This adjustment mechanism can reallocate the importance weights of different modal features based on the real-time changes in the user's gaze patterns, ensuring the adaptability of multimodal feature fusion. The resulting multimodal fusion feature, generated based on the adjusted attention score, more accurately reflects the user's true intent, focus of interest, and cognitive state during the current interaction. The digital human's body movements, driven by control parameters generated from this multimodal fusion feature, are no longer simple, semantically-based mechanical responses, but rather natural behaviors highly synchronized and coordinated with the user's visual attention changes and emotional fluctuations in speech. This achieves a more natural and coordinated human-computer interaction effect, significantly improving the digital human's performance and user experience in complex interaction scenarios.

[0153] Figure 3 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 3 As shown, the electronic device may include: a processor 310, a communications interface 320, a memory 330, and a communication bus 340, wherein the processor 310, the communications interface 320, and the memory 330 communicate with each other through the communication bus 340. The processor 310 can call logical instructions in the memory 330 to execute a digital human interaction action generation method, which includes: inputting multimodal data of the target user interacting with the digital human into a feature extraction module of the interaction model to obtain multimodal features; wherein the multimodal features include: visual focus features, speech features, and text sentiment features corresponding to the speech features; The multimodal features are input into the fusion module of the interaction model. The fusion module adjusts the initial attention score of the fusion module based on the temporal change features of the visual focus features to obtain the adjusted attention score. The fusion module performs fusion processing on the multimodal features based on the adjusted attention score to obtain multimodal fusion features, and generates control parameters for driving the movement of at least one body part of the digital human based on the multimodal fusion features.

[0154] Moreover, the logic instructions in the memory 330 described above can be implemented in the form of software functional units and sold or used as independent products, and can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the present application essentially or the parts that contribute to the prior art or parts of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium, and includes several instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, and various media that can store program codes.

[0155] In another aspect, the present application also provides a computer program product, which comprises a computer program, the computer program can be stored on a non-transitory computer readable storage medium, and the computer program can be executed by a processor to enable a computer to execute the digital human interactive action generation method provided by the above-mentioned methods. The method comprises: inputting multi-modal data of a target user interacting with a digital human into a feature extraction module of an interaction model to obtain multi-modal features; wherein the multi-modal features comprise: visual focus features, speech features, and text sentiment features corresponding to the speech features; inputting the multi-modal features into a fusion module of the interaction model, and adjusting initial attention scores of the fusion module based on time sequence change characteristics of the visual focus features to obtain adjusted attention scores; The fusion module performs fusion processing on the multi-modal features according to the adjusted attention scores to obtain multi-modal fusion features, and generates control parameters for driving at least one body part of the digital human to move according to the multi-modal fusion features.

[0156] In yet another aspect, the present application also provides a non-transitory computer readable storage medium having a computer program stored thereon, the computer program being executed by a processor to implement the digital human interactive action generation method provided by the above-mentioned methods. The method comprises: inputting multi-modal data of a target user interacting with a digital human into a feature extraction module of an interaction model to obtain multi-modal features; wherein the multi-modal features comprise: visual focus features, speech features, and text sentiment features corresponding to the speech features; inputting the multi-modal features into a fusion module of the interaction model, and adjusting initial attention scores of the fusion module based on time sequence change characteristics of the visual focus features to obtain adjusted attention scores; The fusion module fuses the multi-modal features according to the adjusted attention scores to obtain multi-modal fusion features, and generates control parameters for driving the digital person to move at least one body part according to the multi-modal fusion features.

[0157] The device embodiments described above are only schematic, wherein the units shown as separate components can or can not be physically separate, and the components shown as units can or can not be physical units, i.e., can be located in one place or distributed on multiple network units. Part or all of the modules can be selected according to actual needs to achieve the purpose of the present embodiment scheme. Those skilled in the art can understand and implement without creative labor.

[0158] From the above description of the embodiments, those skilled in the art can clearly understand that the embodiments can be realized by means of software plus necessary general hardware platforms, and of course can also be realized by hardware. Based on such understanding, the above technical solutions can be embodied in the form of software products, which can be stored in a computer readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and include a number of instructions to make a computer device (which can be a personal computer, a server, or a network device, etc.) execute the methods described in each embodiment or some parts of the embodiments.

[0159] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: it can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement to some technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A digital human interactive action generation method, characterized by, The method comprises the following steps: multimodal data of a target user interacting with a digital human, inputting the multimodal data into a feature extraction module of an interaction model to obtain multimodal features, wherein the multimodal features comprise visual focus features, speech features, and text sentiment features corresponding to the speech features; inputting the multimodal features into a fusion module of the interaction model, adjusting initial attention scores of the fusion module based on time sequence variation characteristics of the visual focus features to obtain adjusted attention scores; the fusion module fuses the multimodal features according to the adjusted attention scores to obtain multimodal fusion features, and generates control parameters for driving at least one body part of the digital human to move according to the multimodal fusion features.

2. The digital human interactive action generation method of claim 1, wherein, The fusion module adjusts the initial attention scores of the fusion module based on the time sequence variation characteristics of the visual focus features to obtain adjusted attention scores, comprising: determining focus variation degree multi-dimensional features and focus variation pattern features of the visual focus features according to the time sequence variation characteristics of the visual focus features; generating an attention bias parameter according to the focus variation degree multi-dimensional features, and generating an attention scaling parameter according to the focus variation pattern features; adjusting the attention scores of the multimodal data according to the attention bias parameter and the attention scaling parameter to obtain the adjusted attention scores.

3. The digital human interactive action generation method of claim 2, wherein, The focus variation degree multi-dimensional features of the visual focus features comprise: instantaneous speed features of the visual focus, obtained by calculating the rate of change of the position of the visual focus between adjacent time frames; acceleration features of the visual focus, obtained by calculating the rate of change of the instantaneous speed of the visual focus; change persistence features of the visual focus, obtained by calculating the total movement distance of the visual focus within a preset time window; change amplitude features of the visual focus, obtained by calculating the variance of the position of the visual focus within a preset time window.

4. The digital human interactive action generation method of claim 2, wherein, The method for determining the focus variation pattern features comprises: inputting the time sequence variation characteristics of the visual focus features into a pattern classifier in the fusion module, and the pattern classifier outputs a class label of the focus variation pattern features, The class label comprises at least one of a stable fixation pattern, a smooth tracking pattern, a rapid saccade pattern, an exploratory movement pattern, and a regression pattern.

5. The digital human interactive action generation method of claim 4, wherein, Generating an attention scaling parameter according to the focus variation pattern features comprises: when the focus variation pattern features are a stable fixation pattern, generating a first attention scaling parameter, the first attention scaling parameter is used to increase the attention weight corresponding to the visual focus features, and to reduce the attention weight corresponding to the speech features and the text sentiment features; when the focus variation pattern features are a rapid saccade pattern, generating a second attention scaling parameter, the second attention scaling parameter is used to increase the attention weight corresponding to the text sentiment features; when the focus variation pattern features are a smooth tracking pattern, generating a third attention scaling parameter, the third attention scaling parameter is used to evenly distribute the attention weights corresponding to the visual focus features, the speech features, and the text sentiment features.

6. The digital human interactive action generation method of claim 2, wherein, According to the attention bias parameter and the attention scaling parameter, the attention score of the multi-modal data is adjusted to obtain an adjusted attention score, including: Elementarily adding the initial attention score and the attention bias parameter to obtain an attention score after bias processing; Multiplying the attention score after bias processing and the attention scaling parameter to obtain an adjusted attention score.

7. The digital human interactive action generation method of claim 1, wherein, The visual focus feature includes: A focus position coordinate sequence representing a gaze point position of the target user's line of sight in a screen or three-dimensional space; A focus dwell time sequence representing the target user's dwell time at each gaze point; A focus area distribution feature representing the distribution of the target user's line of sight in a predefined region of interest; A pupil diameter change feature reflecting the cognitive load and emotional state of the target user.

8. The digital human interactive action generation method of claim 1, wherein, The interaction model further includes an output module; The output module is configured to output an interaction video between the digital human and the target user according to the control parameter corresponding to the multi-modal fusion feature, the voice feature and the three-dimensional data of the digital human; The control parameter includes at least one of the following: eye rotation angle control parameter, head rotation angle control parameter, and torso rotation angle control parameter.

9. The digital human interactive action generation method of claim 1, wherein, Before the step of inputting the multi-modal data of the target user interacting with the digital human into the feature extraction module of the interaction model, the method further includes: Obtaining a plurality of training samples by taking multi-modal data samples carrying text sentiment labels, focus change degree multi-dimensional feature labels, focus change mode feature labels and control parameter labels as a training sample; Training the feature extraction module and the fusion module of the interaction model based on the plurality of training samples, and stopping training when a preset training target is reached to obtain a trained interaction model.

10. A digital human interaction action generation apparatus, characterized by, It includes: An input module configured to input multi-modal data of a target user interacting with a digital human into a feature extraction module of an interaction model to obtain multi-modal features; wherein the multi-modal features include visual focus features, voice features, and text sentiment features corresponding to the voice features; An adjustment module configured to input the multi-modal features into a fusion module of the interaction model, and adjust an initial attention score of the fusion module based on a time sequence change feature of the visual focus features to obtain an adjusted attention score; The fusion module is configured to perform fusion processing on the multi-modal features based on the adjusted attention score to obtain multi-modal fusion features, and generate control parameters for driving at least one body part of the digital human to move based on the multi-modal fusion features.

11. An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor executes the computer program to implement the digital human interaction action generation method of any one of claims 1-9.

12. A non-transitory computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to implement the digital human interaction action generation method of any one of claims 1-9.

Images