A digital teacher personalized behavior modeling method based on multi-modal feature fusion

Abstract

The application discloses a digital teacher personalized behavior modeling method based on multi-modal feature fusion, obtains preprocessed multi-modal time series data, cuts the preprocessed multi-modal time series data into behavior segments according to semantic consistency, obtains a behavior segment representation vector, performs online time series clustering on the behavior segment representation vector, obtains an emotion-intention prototype set, outputs a target emotion-intention state after confidence calibration, generates a digital teacher action trigger intention, generates a scheduled action trigger instruction queue according to an action priority and a conflict resolution rule, eliminates non-compliant action instructions and generates a compliant action trigger instruction queue, and renders a digital teacher expression, posture, voice tone and teaching auxiliary action in an end side or an edge device. The application significantly improves the interaction naturalness, action controllability and education standardization of the digital teacher in a real teaching scene.

Description

Technical Field

[0001] This invention relates to the field of digital teacher technology, and in particular to a method for personalized behavior modeling of digital teachers based on multimodal feature fusion. Background Technology

[0002] With the deepening application of artificial intelligence and computer vision technology in educational scenarios, teacher emotion recognition based on visual and voice information and digital human teacher-driven learning have become important development directions for smart classrooms and digital education.

[0003] Frame-by-frame emotion recognition methods are extremely sensitive to occlusion, lighting changes, small-amplitude rapid movements and micro-expressions during the teaching process. They are prone to high-frequency jitter and emotion misjudgment, resulting in recognition results that are out of sync with the actual teaching rhythm. Although simple time window averaging and regular smoothing can reduce jitter to some extent, they often sacrifice the timeliness and sensitivity of emotional response and are difficult to take into account the rapid fluctuations and overall evolutionary characteristics of emotional states.

[0004] Existing digital human teacher action triggering mechanisms generally adopt static emotion-action checklists or simple threshold mapping, lacking systematic consideration of actual classroom context, teaching pace and behavior priority. This easily leads to problems such as excessive or misaligned digital human actions or conflicts with the teacher's actual explanation content. It cannot guarantee the naturalness of action expression, nor can it effectively avoid interference with teaching objectives due to excessive or inappropriate action triggering. Summary of the Invention

[0005] One objective of this invention is to propose a personalized behavior modeling method for digital teachers based on multimodal feature fusion. This invention significantly improves the naturalness of interaction, controllability of actions, and standardization of education of digital human teachers in real teaching scenarios.

[0006] A method for modeling personalized behavior of digital teachers based on multimodal feature fusion according to an embodiment of the present invention includes:

[0007] Multimodal input data streams from teaching scenarios are collected and preprocessed to obtain preprocessed multimodal time-series data;

[0008] Behavioral boundary cues are generated based on preprocessed multimodal temporal data. Temporal boundary detection is performed based on the behavioral boundary cues, and the preprocessed multimodal temporal data is segmented into behavioral segments according to semantic consistency.

[0009] Multimodal feature fusion encoding is performed on each behavior segment in the behavior segment to obtain the behavior segment representation vector. Within the sliding time window, online temporal clustering is performed on the behavior segment representation vector according to the temporal continuity constraint and semantic similarity measure to obtain the emotion-intent prototype set.

[0010] An initial emotion-intention state is generated using an emotion-intention prototype set. The initial emotion-intention state is written into the emotion-intention state cache, and the confidence of the current emotion-intention state is calibrated. The results of facial expression consistency test, posture consistency test, and teacher speech consistency test are fused together to output the target emotion-intention state after confidence calibration.

[0011] The target emotion-intention state is compared with the target emotion-intention state of the previous frame with a delay, and the action strategy mapping library is queried to generate the digital teacher action triggering intention.

[0012] The system executes trigger scheduling control on the digital teacher's action triggering intent, generates a queue of scheduled action triggering instructions according to action priority and conflict resolution rules, removes non-compliant action instructions and generates a queue of compliant action triggering instructions.

[0013] The compliant action trigger command queue is sent to the digital teacher rendering module to render the digital teacher's expressions, postures, voice tone and teaching assistance actions on the edge device.

[0014] Optionally, the preprocessed multimodal time series data includes:

[0015] The system collects multimodal input data streams from the teaching scenario, including teacher images captured by the camera, teacher voice signals captured by the microphone, and ambient lighting parameters collected by external sensors.

[0016] For each frame of the teacher image, face region detection is performed, and exposure balance adjustment, white balance correction, and face jitter removal are performed sequentially on the face region;

[0017] Skeleton keypoint extraction is performed on each frame of the teacher image to obtain the set of position coordinates of all skeleton keypoints in each frame. The position coordinates of each skeleton keypoint are composed of the horizontal and vertical coordinates. Each frame contains multiple keypoints, and all keypoints constitute the skeleton keypoint set of the corresponding frame.

[0018] The skeleton keypoint set of all frames is temporally smoothed in chronological order to obtain the smoothed skeleton keypoint set of the current frame.

[0019] The teacher's speech signal is segmented according to a fixed frame length and background noise suppression is performed to obtain a noise-suppressed speech signal.

[0020] For each frame of denoised speech signal, calculate the speech signal energy features and the speech signal fundamental frequency features;

[0021] The preprocessed teacher images, skeleton key point sets, speech signal energy features and speech signal fundamental frequency features, and original illumination parameters are summarized in chronological order to obtain preprocessed multimodal time-series data.

[0022] Optionally, the step of segmenting the preprocessed multimodal time-series data into behavioral segments based on semantic consistency includes:

[0023] Calculate the average Euclidean displacement velocity of the teacher skeleton keypoint set between two adjacent frames to obtain the teacher's motion change rate value;

[0024] The mean squared error of the facial expression action unit activation value in each frame of the teacher's image is calculated between two adjacent frames to obtain the teacher's facial expression dynamics index value.

[0025] Based on the energy characteristics and fundamental frequency characteristics of the teacher's speech signal, the peak index of speech prosody is determined;

[0026] By weighted and fused with the teacher's rate of change of action, facial dynamics index, and speech prosody peak index, behavioral boundary cues are obtained.

[0027] Perform temporal boundary detection on the behavior boundary cues to obtain a set of behavior boundary frames;

[0028] Based on the set of behavior boundary frames, preprocessed multimodal time series data is divided into semantically consistent behavior segments.

[0029] Optionally, obtaining the set of emotion-intention prototypes includes:

[0030] Calculate the average ratio of the skeletal motion change rate, the average value of the facial dynamics index, and the average ratio of the speech prosody peak index in each behavioral segment;

[0031] The average ratio of the skeleton motion change rate, the average value of the facial dynamics index, and the average ratio of the peak value of the speech prosody index are concatenated to form a multimodal fusion feature vector corresponding to the behavior segment;

[0032] Within a preset sliding time window, for several consecutive behavioral segments, the multimodal fusion feature vector of each behavioral segment is compared with the feature vectors of all prototypes in the emotion-intention prototype set. When the maximum similarity is greater than or equal to the preset similarity judgment threshold, the current behavioral segment is assigned to the most similar emotion-intention prototype, and the feature vector of the emotion-intention prototype is updated using an incremental averaging method. If the maximum similarity is less than the preset similarity judgment threshold, a new emotion-intention prototype is created based on the multimodal fusion feature vector of the current behavioral segment, and added to the emotion-intention prototype set.

[0033] Apply a temporal continuity constraint to the set of emotion-intention prototypes for consecutive behavioral segments, and output the set of emotion-intention prototypes.

[0034] Optionally, the confidence-calibrated target emotion-intent state includes:

[0035] Based on the set of emotion-intention prototypes and the emotion-intention labels corresponding to the behavior segments, an initial emotion-intention state is generated.

[0036] Write the initial emotion-intention state into the emotion-intention state cache unit in chronological order;

[0037] Confidence calibration is performed on the current initial emotion-intention state in the emotion-intention state cache unit, including facial expression consistency score, posture consistency score, and speech consistency score.

[0038] The facial expression consistency score, posture consistency score, and speech consistency score are combined and weighted according to a preset weighting coefficient to obtain the multimodal fusion confidence value.

[0039] The initial emotion-intent state of the current frame is combined with its corresponding multimodal fusion confidence value to output the target emotion-intent state after confidence calibration.

[0040] Optionally, the facial expression consistency score is calculated based on the consistency of the activation values ​​of facial expression action units in each frame within the i-th behavioral segment:

[0041] ;

[0042] in, This represents the activation value of the j-th facial expression action unit in frame t. The average value of the i-th action segment in the j-th action unit. The number of frames in the action segment. Indicates the total number of action units;

[0043] The variance of the skeleton joint angle set in each time frame is calculated, the average variance of all time frames is taken, and then the attitude consistency score is obtained by subtracting the average variance.

[0044] Speech consistency score is defined based on the variability of speech signal energy features and fundamental frequency features:

[0045] ;

[0046] in, The numbers represent the standard deviation. These represent the average speech signal energy characteristics and the average speech signal fundamental frequency characteristics within a behavioral segment, respectively. These represent the energy characteristics and fundamental frequency characteristics of the speech signal, respectively.

[0047] Optionally, the generation of digital teacher action triggering intent includes:

[0048] Calculate the magnitude of the change in state label between the target emotion-intent state in the current frame and the target emotion-intent state in the previous frame.

[0049] If the change in the state label is less than the hysteresis threshold, the emotion-intention label of the previous frame is used as the hysteresis-corrected emotion-intention state label of the current frame. If the change in the state label is greater than or equal to the hysteresis threshold, the target emotion-intention label of the current frame is used as the hysteresis-corrected emotion-intention state label of the current frame.

[0050] Write the hysteresis-corrected sentiment-intent state label and the corresponding multimodal fusion confidence value of the current frame into the sentiment-intent context memory unit;

[0051] Based on the hysteresis-corrected emotion-intention state label and the state distribution statistics contained in the emotion-intention context memory unit of the current frame, digital teacher action triggering intentions are retrieved from the action policy mapping library and generated.

[0052] Optionally, the generation of the compliance action trigger instruction queue includes:

[0053] Assign a basic priority coefficient to each digital teacher's action triggering intention, and calculate a comprehensive priority score by combining the context weight coefficient;

[0054] The set of digital teacher action trigger intentions is sorted from high to low according to the comprehensive priority score to form the initial action trigger sequence;

[0055] For the initial action trigger sequence, a minimum trigger interval constraint is applied to each digital teacher action trigger intention based on the action history execution record: the time difference between the current time and the execution time of the most recent action of this category is judged. If the time difference is less than the preset minimum trigger interval, the corresponding digital teacher action trigger intention is marked as a delayed trigger candidate and removed from the current sequence. The remaining actions form the throttled action trigger sequence.

[0056] Conflict detection is performed on the action trigger sequence after throttling to form a queue of action trigger instructions after scheduling.

[0057] For each digital teacher action triggering intent in the queue of dispatched action triggering instructions, a compliance check is performed sequentially to determine whether the digital teacher action triggering intent belongs to the security whitelist set and whether the digital teacher action triggering intent meets the posture amplitude and content security requirements. If the digital teacher action triggering intent does not belong to the security whitelist set or does not meet the posture amplitude and content security requirements, it is removed and the remaining actions are retained.

[0058] All digital teacher action trigger intentions that have passed the compliance verification will be organized into a compliant action trigger instruction queue in sequence.

[0059] Optionally, the conflict detection determines whether there is a conflict relationship between any two digital teacher action triggering intentions based on the action conflict matrix. If there is a conflict relationship and the action with the lower comprehensive priority score conflicts with the action with the higher comprehensive priority score, then the action with the lower comprehensive priority score is removed and the action with the higher comprehensive priority score is retained.

[0060] The beneficial effects of this invention are:

[0061] (1) This invention proposes a multimodal behavior fragment-level coding and boundary detection method, which significantly improves the stability and robustness of emotion recognition. By jointly preprocessing the multimodal input data of teachers, the behavior boundary cue sequence is generated using action change rate, facial expression dynamics index and speech prosody peak multi-source signal. A weighted fusion mechanism is used for temporal boundary detection, which realizes variable-length, semantically consistent behavior fragment segmentation. Behavior fragment-level modeling effectively suppresses frame-level misjudgment caused by occlusion, illumination changes and short-term noise. It can better capture the actual characteristics of slow evolution and short-term fluctuation of emotions in teaching scenarios, and greatly improve the stability and robustness of emotion-intention state discrimination.

[0062] (2) This invention constructs a temporal clustering algorithm that integrates hysteresis and context awareness to obtain a highly interpretable and low-jitter emotion-intention prototype state sequence. At the behavioral segment feature level, a segment representation vector is generated by multimodal feature fusion. An online temporal clustering and time continuity constraint mechanism within a sliding window are adopted to dynamically aggregate similar behavioral segments into stable emotion-intention prototypes. Hysteresis comparison and confidence calibration are introduced to effectively suppress high-frequency jumps and false triggers of emotional states. The prototype set can be adaptively updated according to individual differences of teachers, expression styles and context changes to ensure that the emotion-intention state sequence has both good responsiveness and sufficient coherence and interpretability.

[0063] (3) This invention introduces action strategy mapping and compliance scheduling control based on state and context memory to achieve natural restraint, stable rhythm and safety compliance of digital human teacher's actions. By setting action priority, minimum trigger interval, throttling and conflict resolution mechanism in the action triggering stage, combined with safety whitelist and scene boundary rules, the risk of digital human actions being too dense, misaligned or inappropriate is effectively avoided. The action strategy mapping not only depends on the current emotion-intention state, but also combines historical context statistical information to dynamically adjust the action type and performance intensity, which significantly improves the naturalness of interaction, controllability of actions and educational standardization of digital human teacher in real teaching scenarios. Attached Figure Description

[0064] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings:

[0065] Figure 1 This is a flowchart of a digital teacher personalized behavior modeling method based on multimodal feature fusion proposed in this invention. Detailed Implementation

[0066] Example 1: Reference Figure 1 A digital teacher personalized behavior modeling method based on multimodal feature fusion includes:

[0067] Multimodal input data streams from teaching scenarios are collected and preprocessed to obtain preprocessed multimodal time-series data;

[0068] In this embodiment, the preprocessed multimodal time series data includes:

[0069] The system collects multimodal input data streams from the teaching scenario, including teacher images captured by the camera, teacher voice signals captured by the microphone, and ambient lighting parameters collected by external sensors.

[0070] Each frame in the teacher image is acquired individually, and each frame of the teacher's voice signal corresponds one-to-one with the image of the corresponding time frame. The ambient lighting parameter is the ambient brightness parameter at the corresponding time. The total number of frames acquired is the sum of the number of frames over a period of time.

[0071] For each frame of the teacher image, face region detection is performed, and exposure balance adjustment, white balance correction, and face jitter removal are performed sequentially on the face region;

[0072] In Example 1, exposure balance adjustment is achieved by adjusting the average brightness of the current image area to a preset target average brightness; white balance correction is achieved by adjusting the average values ​​of the red, green, and blue channels of the image to preset target average values ​​for each channel; face jitter removal is achieved by calculating the Euclidean distance between the center positions of key points in the face region of two consecutive frames. If the inter-frame displacement is less than a preset jitter threshold, the corresponding frame is determined to be a stable frame and participates in subsequent processing; otherwise, it is not considered a valid frame and participates in subsequent processing.

[0073] Skeleton keypoint extraction is performed on each frame of the teacher image to obtain the set of position coordinates of all skeleton keypoints in each frame. The position coordinates of each skeleton keypoint are composed of the horizontal and vertical coordinates. Each frame contains multiple keypoints, and all keypoints constitute the skeleton keypoint set of the corresponding frame.

[0074] The skeleton keypoint set of all frames is temporally smoothed in chronological order to obtain the smoothed skeleton keypoint set of the current frame.

[0075] In Example 1, a sliding window method is used to perform a weighted average of the skeleton keypoint set of each frame. The center of the sliding window is the current frame, and the window length is a specified fixed number of frames. The average of all skeleton keypoint sets within the sliding window is calculated to obtain the smoothed skeleton keypoint set of the current frame. The temporal smoothing operation can remove short-term high-frequency jitter in the skeleton keypoints, so that continuous actions remain smooth and consistent.

[0076] The teacher's speech signal is segmented according to a fixed frame length and background noise suppression is performed to obtain a noise-suppressed speech signal.

[0077] The spectral subtraction method is used to compare the speech spectrum of the current frame with the estimated noise spectrum, and the noise spectrum is subtracted from the speech spectrum to obtain the noise-suppressed speech signal. All negative values ​​are set to zero to ensure the purity of the speech signal.

[0078] For each frame of denoised speech signal, calculate the speech signal energy features and the speech signal fundamental frequency features;

[0079] The speech signal energy features are obtained by summing the squares of all sampling points in each frame of the speech signal, and are used to measure the loudness level of the teacher's speech signal. The speech signal fundamental frequency features are obtained by estimating the fundamental frequency period of each frame of the speech signal, and are used to reflect the prosodic structure of the speech signal.

[0080] The preprocessed teacher images, skeleton key point sets, speech signal energy features and speech signal fundamental frequency features, and original illumination parameters are summarized in chronological order to obtain preprocessed multimodal time-series data.

[0081] Behavioral boundary cues are generated based on preprocessed multimodal temporal data. Temporal boundary detection is performed based on the behavioral boundary cues, and the preprocessed multimodal temporal data is segmented into behavioral segments according to semantic consistency.

[0082] In this embodiment, the preprocessed multimodal time-series data is segmented into behavioral segments based on semantic consistency, including:

[0083] Calculate the average Euclidean displacement velocity of the teacher skeleton keypoint set between two adjacent frames to obtain the teacher's motion change rate value;

[0084] In Example 1, the teacher motion change rate value is used to measure the dynamic intensity of the teacher skeleton motion in time. The calculation method is as follows: in each frame, the Euclidean distance between the position coordinates of all skeleton key points in the current frame and the previous frame is calculated, the mean Euclidean distance of all skeleton key points is calculated, and the teacher motion change rate value of the corresponding frame is obtained.

[0085] The mean squared error of the facial expression action unit activation value in each frame of the teacher's image is calculated between two adjacent frames to obtain the teacher's facial expression dynamics index value.

[0086] In Example 1, the teacher facial expression dynamics index is used to detect the dynamic changes of the teacher's facial expressions. The calculation method is as follows: in each frame, the activation values ​​of all facial expression action units are subtracted from the activation values ​​of the facial expression action units in the previous frame. The average of the squares of all the differences is then calculated to obtain the teacher facial expression dynamics index value of the current frame.

[0087] The facial expression action unit activation value is used to quantify the movement amplitude of the main facial expression muscle groups of the teacher. For each frame of the teacher's face region, the response amplitude of all preset facial expression action units on the face region of the current frame is calculated, and all response amplitudes are used as the set of facial expression action unit activation values ​​for the current frame.

[0088] Based on the energy characteristics and fundamental frequency characteristics of the teacher's speech signal, the peak index of speech prosody is determined;

[0089] In Example 1, the speech prosody peak index is used to assist in identifying semantic stress, pauses, or teaching rhythm switching. The determination method is as follows: for each frame, it is determined whether the speech signal energy feature and the speech signal fundamental frequency feature have reached local maximum values ​​within the sliding window centered on the current frame. If both are local maximum values, the current frame is marked as a speech prosody peak frame; otherwise, no marking is made.

[0090] By weighted and fused with the teacher's rate of change of action, facial dynamics index, and speech prosody peak index, behavioral boundary cues are obtained.

[0091] In Example 1, the behavioral boundary cue value is used to comprehensively measure the intensity of multimodal behavioral changes in the current frame. The weighted fusion method is to normalize the teacher's action change rate value and the teacher's facial expression dynamics index value respectively, and then add the normalized teacher's action change rate value, the normalized teacher's facial expression dynamics index value and the speech prosody peak index value according to the preset weight coefficient to obtain the behavioral boundary cue value of the current frame.

[0092] Perform temporal boundary detection on the behavior boundary cues to obtain a set of behavior boundary frames;

[0093] Behavior boundary frames are used as anchor points for behavior segment division. The determination method for temporal boundary detection is as follows: in each frame, if the behavior boundary cue value is greater than or equal to the preset behavior boundary determination threshold, the current frame is marked as a behavior boundary frame; otherwise, it is not marked.

[0094] Based on the set of behavior boundary frames, preprocessed multimodal time series data is divided into semantically consistent behavior segments.

[0095] In Example 1, after obtaining the set of behavior boundary frames, each behavior boundary frame index in the entire set of behavior boundary frames is traversed sequentially. Taking each pair of adjacent behavior boundary frame indices as the start and end points, all continuous time frames between these two behavior boundary frame indices are divided as a complete behavior segment, and so on, until the entire set of behavior boundary frames is traversed.

[0096] Specifically, assuming the action boundary frame set includes n action boundary frame indices, denoted as the 1st action boundary frame index, the 2nd action boundary frame index, and so on up to the nth action boundary frame index, all consecutive time frames between the 1st and 2nd action boundary frame indices are taken as the 1st action segment, all consecutive time frames between the 2nd and 3rd action boundary frame indices are taken as the 2nd action segment, and so on, until finally all consecutive time frames from the last action boundary frame index to the end of the time series data are taken as the last action segment. All time frames included in each action segment are derived from the preprocessed multimodal time series data. Under the constraint of the action boundary frame indices, the time frames are guaranteed to belong to the same semantic state, thereby realizing the division of the preprocessed multimodal time series data into semantically consistent action segments.

[0097] Multimodal feature fusion encoding is performed on each behavior segment in the behavior segment to obtain the behavior segment representation vector. Within the sliding time window, online temporal clustering is performed on the behavior segment representation vector according to the temporal continuity constraint and semantic similarity measure to obtain the emotion-intent prototype set.

[0098] In this embodiment, obtaining the emotion-intention prototype set includes:

[0099] Calculate the average ratio of the skeletal motion change rate, the average value of the facial dynamics index, and the average ratio of the speech prosody peak index in each behavioral segment;

[0100] In Example 1, the average skeleton motion change rate represents the overall dynamic intensity of the teacher's skeleton motion within the behavior segment. The average skeleton motion change rate is obtained by summing the teacher's motion change rate values ​​of all time frames within the behavior segment and dividing by the total number of time frames contained in the behavior segment.

[0101] The average value of the facial dynamics index represents the average amplitude of the dynamic changes in the teacher's facial expressions within the behavioral segment. The average value of the facial dynamics index is obtained by summing the teacher's facial dynamics index values ​​of all time frames within the behavioral segment and then dividing by the total number of time frames contained in the behavioral segment.

[0102] The average ratio of speech prosodic peak values ​​represents the average probability of significant peaks in speech rhythm within a behavioral segment. It is obtained by summing the speech prosodic peak values ​​of all time frames within the behavioral segment and then dividing by the total number of time frames contained in the behavioral segment.

[0103] The average ratio of the skeleton motion change rate, the average value of the facial dynamics index, and the average ratio of the peak value of the speech prosody index are concatenated to form a multimodal fusion feature vector corresponding to the behavior segment;

[0104] Within a preset sliding time window, for several consecutive behavioral segments, the multimodal fusion feature vector of each behavioral segment is compared with the feature vectors of all prototypes in the emotion-intention prototype set. When the maximum similarity is greater than or equal to the preset similarity judgment threshold, the current behavioral segment is assigned to the most similar emotion-intention prototype, and the feature vector of the emotion-intention prototype is updated using an incremental averaging method. If the maximum similarity is less than the preset similarity judgment threshold, a new emotion-intention prototype is created based on the multimodal fusion feature vector of the current behavioral segment, and added to the emotion-intention prototype set.

[0105] In Example 1, S33 specifically refers to: the emotion-intention prototype set refers to the set of multiple representative multimodal feature vectors obtained by online temporal clustering of the multimodal fusion feature vectors of all behavioral fragments, and each representative multimodal feature vector corresponds to a specific teacher emotion-intention state.

[0106] For each existing emotion-intention prototype, calculate the similarity between the multimodal fusion feature vector of the current behavior segment and the emotion-intention prototype. The similarity is the cosine similarity between the multimodal fusion feature vector and the prototype feature vector.

[0107] If the similarity between the multimodal fusion feature vector of the current behavior segment and a certain emotion-intention prototype is greater than or equal to the set semantic similarity judgment threshold, then the current behavior segment is classified into the emotion-intention prototype category, and the emotion-intention prototype is updated by incremental averaging. The multimodal fusion feature vector of the current behavior segment and the feature vector of the emotion-intention prototype are weighted by the number of samples and then updated to a new emotion-intention prototype.

[0108] If the similarity between the multimodal fusion feature vector of the current behavior segment and all existing emotion-intention prototypes is lower than the semantic similarity threshold, then a new emotion-intention prototype is added based on the multimodal fusion feature vector of the current behavior segment, and added to the emotion-intention prototype set.

[0109] Apply a temporal continuity constraint to the set of emotion-intention prototypes for consecutive behavioral segments, and output the set of emotion-intention prototypes.

[0110] The temporal continuity constraint is that when the cluster label of the current action segment is different from the label of the previous action segment and the similarity difference is less than the hysteresis threshold, the previous cluster label remains unchanged.

[0111] An initial emotion-intention state is generated using an emotion-intention prototype set. The initial emotion-intention state is written into the emotion-intention state cache, and the confidence of the current emotion-intention state is calibrated. The results of facial expression consistency test, posture consistency test, and teacher speech consistency test are fused together to output the target emotion-intention state after confidence calibration.

[0112] In this embodiment, the target emotion-intention state after confidence calibration includes:

[0113] Based on the set of emotion-intention prototypes and the emotion-intention labels corresponding to the behavior segments, an initial emotion-intention state is generated.

[0114] Each initial emotion-intention state in the initial emotion-intention state is consistent with the emotion-intention prototype category to which the corresponding behavioral segment belongs, representing the initial emotion-intention state flow of the digital teacher in the teaching process.

[0115] Write the initial emotion-intention state into the emotion-intention state cache unit in chronological order;

[0116] The emotion-intention state cache unit records the initial emotion-intention state of the current behavior segment and the changes in emotion-intention state within a recent time window. The time window length is the preset state cache sliding window length.

[0117] Confidence calibration is performed on the current initial emotion-intention state in the emotion-intention state cache unit, including facial expression consistency score, posture consistency score, and speech consistency score.

[0118] The facial expression consistency score, posture consistency score, and speech consistency score are combined and weighted according to a preset weighting coefficient to obtain the multimodal fusion confidence value.

[0119] The initial emotion-intent state of the current frame is combined with its corresponding multimodal fusion confidence value to output the target emotion-intent state after confidence calibration.

[0120] In this embodiment, the facial expression consistency score is calculated based on the consistency of the activation values ​​of facial expression action units in each frame within the i-th behavioral segment:

[0121] ;

[0122] in, This represents the activation value of the j-th facial expression action unit in frame t. The average value of the i-th action segment in the j-th action unit. The number of frames in the action segment. Indicates the total number of action units;

[0123] The facial expression consistency score is used to measure the overall consistency of facial expression action unit activation values ​​across all time frames within the current behavioral segment.

[0124] The variance of the skeleton joint angle set in each time frame is calculated, the average variance of all time frames is taken, and then the attitude consistency score is obtained by subtracting the average variance.

[0125] The posture consistency score is used to measure the consistency of overall angular changes of key points on the skeleton within the current behavioral segment.

[0126] Speech consistency score is defined based on the variability of speech signal energy features and fundamental frequency features:

[0127] ;

[0128] in, The numbers represent the standard deviation. These represent the average speech signal energy characteristics and the average speech signal fundamental frequency characteristics within a behavioral segment, respectively. These represent the energy characteristics and fundamental frequency characteristics of the speech signal, respectively.

[0129] The target emotion-intention state is compared with the target emotion-intention state of the previous frame with a delay, and the action strategy mapping library is queried to generate the digital teacher action triggering intention.

[0130] In this embodiment, generating the digital teacher's action trigger intent includes:

[0131] Calculate the magnitude of the change in state label between the target emotion-intent state in the current frame and the target emotion-intent state in the previous frame.

[0132] The feature vectors corresponding to the target emotion-intention state in the current frame and the previous frame are determined respectively. The feature vectors are normalized. The two normalized feature vectors are then multiplied by a dot product. The result of the dot product is subtracted by one to obtain the change magnitude of the state label. The change magnitude of the label is used to reflect the semantic change intensity between the feature vectors of the current frame and the previous frame.

[0133] If the change in the state label is less than the hysteresis threshold, the emotion-intention label of the previous frame is used as the hysteresis-corrected emotion-intention state label of the current frame. If the change in the state label is greater than or equal to the hysteresis threshold, the target emotion-intention label of the current frame is used as the hysteresis-corrected emotion-intention state label of the current frame.

[0134] The delayed emotion-intention state label is used to reduce high-frequency state transitions, and the delayed threshold ranges from zero to one.

[0135] Write the hysteresis-corrected sentiment-intent state label and the corresponding multimodal fusion confidence value of the current frame into the sentiment-intent context memory unit;

[0136] The emotion-intention context memory unit stores the hysteresis-corrected emotion-intention state labels and multimodal fusion confidence values ​​of the most recent frames in chronological order, with the memory window length being a preset fixed value.

[0137] Based on the hysteresis-corrected emotion-intention state label and the state distribution statistics contained in the emotion-intention context memory unit of the current frame, digital teacher action triggering intentions are retrieved from the action policy mapping library and generated.

[0138] In Example 1, the delayed-corrected emotion-intention state label and the historical state labels, historical state confidence, historical state duration, historical state switching frequency, and other statistical features in the context memory unit are input into the action strategy mapping library. The action strategy mapping library includes mapping rules for different emotion-intention state labels and their contextual statistical features. By searching for mapping rules that match the current delayed-corrected emotion-intention state label and its related contextual statistical features, the corresponding action triggering strategy is determined, and the digital teacher action triggering intention is generated. The digital teacher action triggering intention includes control instructions for the digital teacher's facial expressions, gestures, tone of voice adjustment, and teaching auxiliary behavior action types.

[0139] The system executes trigger scheduling control on the digital teacher's action triggering intent, generates a queue of scheduled action triggering instructions according to action priority and conflict resolution rules, removes non-compliant action instructions and generates a queue of compliant action triggering instructions.

[0140] In this embodiment, generating a compliance action trigger instruction queue includes:

[0141] Assign a basic priority coefficient to each digital teacher's action triggering intention, and calculate a comprehensive priority score by combining the context weight coefficient;

[0142] The basic priority coefficient is used to measure the inherent importance of different types of actions in the teaching scenario. The comprehensive priority score is obtained by adding the basic priority coefficient and the context weight coefficient. The context weight coefficient is used to reflect the dynamic adjustment of the importance of the current emotion-intention context memory to the action.

[0143] The set of digital teacher action trigger intentions is sorted from high to low according to the comprehensive priority score to form the initial action trigger sequence;

[0144] In the initial action trigger sequence, the first action trigger intention is the digital teacher action trigger intention with the highest overall priority score, and the last action trigger intention is the digital teacher action trigger intention with the lowest overall priority score.

[0145] For the initial action trigger sequence, a minimum trigger interval constraint is applied to each digital teacher action trigger intention based on the action history execution record: the time difference between the current time and the execution time of the most recent action of this category is judged. If the time difference is less than the preset minimum trigger interval, the corresponding digital teacher action trigger intention is marked as a delayed trigger candidate and removed from the current sequence. The remaining actions form the throttled action trigger sequence.

[0146] The minimum trigger interval constraint is used to limit the minimum interval between digital teacher action trigger intentions of the same category over a period of time.

[0147] Conflict detection is performed on the action trigger sequence after throttling to form a queue of action trigger instructions after scheduling.

[0148] In this embodiment, conflict detection determines whether there is a conflict relationship between any two digital teacher action triggering intentions based on the action conflict matrix. If there is a conflict relationship and the action with the lower comprehensive priority score conflicts with the action with the higher comprehensive priority score, then the action with the lower comprehensive priority score is removed and the action with the higher comprehensive priority score is retained.

[0149] For each digital teacher action triggering intent in the queue of dispatched action triggering instructions, a compliance check is performed sequentially to determine whether the digital teacher action triggering intent belongs to the security whitelist set and whether the digital teacher action triggering intent meets the posture amplitude and content security requirements. If the digital teacher action triggering intent does not belong to the security whitelist set or does not meet the posture amplitude and content security requirements, it is removed and the remaining actions are retained.

[0150] All digital teacher action trigger intentions that have passed the compliance verification will be organized into a compliant action trigger instruction queue in sequence.

[0151] The compliant action trigger command queue is sent to the digital teacher rendering module to render the digital teacher's expressions, postures, voice tone and teaching assistance actions on the edge device.

[0152] Example 2: In a smart classroom, during a teacher's lecture, the digital human teacher emotion recognition and action triggering system records and analyzes multimodal input data throughout. In a specific teaching segment, the system detected that the teacher's skeletal movement change rate averaged 0.28 over a 20-second interval, significantly higher than the usual static average of 0.08. Simultaneously, the average activation value of facial expression units fluctuated by 0.35, and the peak voice energy ranged between 0.62 and 0.71, all higher than the normal classroom range. Based on the behavior segmentation algorithm, the system automatically identified this segment as a new key teaching segment. Traditional frame-by-frame classification methods would show 11 consecutive emotion jumps in this segment, while the method of this invention only showed one state transition within this segment. The digital human's movements remained consistently smooth and abrupt throughout the segment.

[0153] In the subsequent Q&A session, the system recorded an average skeletal movement change rate of 0.13, an facial expression dynamics index of 0.18, and only sporadic peaks in speech prosody. After multimodal feature normalization, the system clustered the data into an "encouraging interaction" state. Based on this state, the digital human teacher system continuously outputted actions such as smiling and nodding, with all actions occurring at intervals exceeding the system's set minimum trigger interval of 2.5 seconds. The action scheduling module compared the historical action sequences and detected no conflicts with the previous action; all actions successfully passed compliance filtering.

[0154] During a student question segment that was inserted unexpectedly, the system detected a rapid change in the teacher's facial expression, with the maximum fluctuation in the activation value of the facial expression action unit reaching 0.63. Although the rate of action change and voice features showed no significant abnormalities, the system, based on the expression consistency confidence level dropping to 0.54, temporarily suspended the generation of high-intensity emotional actions. The digital human teacher system only output slight facial expression changes, effectively avoiding high-frequency emotion jumps and erroneous action triggers. In contrast, traditional methods showed three emotion tag jumps within the same time period, two instances of mistimed actions, and one instance of an irregular, large gesture.

[0155] As the course entered its summary phase, the teacher's skeletal movement rate decreased to 0.07, the facial dynamics index dropped to 0.13, and the peak prosody of the voice was also at a low level. The system recognized this as a "stable explanation" state; the digital human teacher maintained a neutral facial expression and naturally lowered arms, without any excessive movements. The compliance module filtered all movements item by item, and all conformed to the action whitelist and scene rules. In this phase, the traditional system misinterpreted small frame-level facial fluctuations as alternating "smiling" and "focused," causing the digital human to exhibit one unnecessary high-frequency facial expression switch.

[0156] Statistical data from a continuous 40-minute teaching session showed that the system segmented 22 effective teaching behavior segments, with an average of 0.36 action jumps per segment, compared to 2.7 jumps per segment using the traditional frame-by-frame method. The digital human teacher experienced only one instance of timing error, zero action conflicts, and zero violations of action compliance. Under the same conditions, the traditional method resulted in six timing errors, two action conflicts, and three violations of action compliance.

[0157] In sample data testing, the system used 32,000 image frames, 480,000 skeleton keypoint frames, and 80,000 speech signals as training sets. After five-fold cross-validation, the stability index (number of state transitions) for digital human emotion-intention recognition was 5.7, compared to 33.8 for the traditional method. The emotion misjudgment rate was 2.2%, compared to 10.7% for the traditional method. The action timing error rate was 1.0%, compared to 8.3% for the traditional method. The naturalness subjective scoring system averaged 9.2 points (out of 10), compared to 7.6 points for the traditional system.

[0158] In one case, the system detected a segment where the teacher transitioned from an "explaining" state to an "emphasizing key points" state. The skeleton change rate increased from 0.11 to 0.33, facial dynamics increased from 0.16 to 0.41, and the peak prosody of the voice increased from 0.18 to 0.65. The system successfully recognized the state transition and retrieved the "pointing to the blackboard, emphasizing tone" action from the action strategy mapping library. All action instructions were successfully issued to the digital human module after passing scheduling, conflict, and compliance checks. The traditional system performed two action jumps in this segment and exhibited a mismatch between gestures and facial expressions because it failed to detect contextual features such as the duration and frequency of the state.

[0159] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A method for modeling personalized behavior of digital teachers based on multimodal feature fusion, characterized in that, include: Multimodal input data streams from teaching scenarios are collected and preprocessed to obtain preprocessed multimodal time-series data; Behavioral boundary cues are generated based on preprocessed multimodal temporal data. Temporal boundary detection is performed based on the behavioral boundary cues, and the preprocessed multimodal temporal data is segmented into behavioral segments according to semantic consistency. Multimodal feature fusion encoding is performed on each behavior segment in the behavior segment to obtain the behavior segment representation vector. Within the sliding time window, online temporal clustering is performed on the behavior segment representation vector according to the temporal continuity constraint and semantic similarity measure to obtain the emotion-intent prototype set. An initial emotion-intention state is generated using an emotion-intention prototype set. The initial emotion-intention state is written into the emotion-intention state cache, and the confidence of the current emotion-intention state is calibrated. The results of facial expression consistency test, posture consistency test, and teacher speech consistency test are fused together to output the target emotion-intention state after confidence calibration. The target emotion-intention state is compared with the target emotion-intention state of the previous frame with a delay, and the action strategy mapping library is queried to generate the digital teacher action triggering intention. The system executes trigger scheduling control on the digital teacher's action triggering intent, generates a queue of scheduled action triggering instructions according to action priority and conflict resolution rules, removes non-compliant action instructions and generates a queue of compliant action triggering instructions. The compliant action trigger command queue is sent to the digital teacher rendering module to render the digital teacher's expressions, postures, voice tone and teaching assistance actions on the edge device.

2. The method for personalized behavior modeling of digital teachers based on multimodal feature fusion according to claim 1, characterized in that, The preprocessed multimodal time series data includes: The system collects multimodal input data streams from the teaching scenario, including teacher images captured by the camera, teacher voice signals captured by the microphone, and ambient lighting parameters collected by external sensors. For each frame of the teacher image, face region detection is performed, and exposure balance adjustment, white balance correction, and face jitter removal are performed sequentially on the face region; Skeleton keypoint extraction is performed on each frame of the teacher image to obtain the set of position coordinates of all skeleton keypoints in each frame. The position coordinates of each skeleton keypoint are composed of the horizontal and vertical coordinates. Each frame contains multiple keypoints, and all keypoints constitute the skeleton keypoint set of the corresponding frame. The skeleton keypoint set of all frames is temporally smoothed in chronological order to obtain the smoothed skeleton keypoint set of the current frame. The teacher's speech signal is segmented according to a fixed frame length and background noise suppression is performed to obtain a noise-suppressed speech signal. For each frame of denoised speech signal, calculate the speech signal energy features and the speech signal fundamental frequency features; The preprocessed teacher images, skeleton key point sets, speech signal energy features and speech signal fundamental frequency features, and original illumination parameters are summarized in chronological order to obtain preprocessed multimodal time-series data.

3. The method for digital teacher personalized behavior modeling based on multimodal feature fusion according to claim 2, characterized in that, The step of segmenting the preprocessed multimodal time-series data into behavioral segments based on semantic consistency includes: Calculate the average Euclidean displacement velocity of the teacher skeleton keypoint set between two adjacent frames to obtain the teacher's motion change rate value; The mean squared error of the facial expression action unit activation value in each frame of the teacher's image is calculated between two adjacent frames to obtain the teacher's facial expression dynamics index value. Based on the energy characteristics and fundamental frequency characteristics of the teacher's speech signal, the peak index of speech prosody is determined; By weighted and fused with the teacher's rate of change of action, facial dynamics index, and speech prosody peak index, behavioral boundary cues are obtained. Perform temporal boundary detection on the behavior boundary cues to obtain a set of behavior boundary frames; Based on the set of behavior boundary frames, preprocessed multimodal time series data is divided into semantically consistent behavior segments.

4. The method for personalized behavior modeling of digital teachers based on multimodal feature fusion according to claim 3, characterized in that, Obtaining the set of emotion-intention prototypes includes: Calculate the average ratio of the skeletal motion change rate, the average value of the facial dynamics index, and the average ratio of the speech prosody peak index in each behavioral segment; The average ratio of the skeleton motion change rate, the average value of the facial dynamics index, and the average ratio of the peak value of the speech prosody index are concatenated to form a multimodal fusion feature vector corresponding to the behavior segment; Within a preset sliding time window, for several consecutive behavioral segments, the multimodal fusion feature vector of each behavioral segment is compared with the feature vectors of all prototypes in the emotion-intention prototype set. When the maximum similarity is greater than or equal to the preset similarity judgment threshold, the current behavioral segment is assigned to the most similar emotion-intention prototype, and the feature vector of the emotion-intention prototype is updated using an incremental averaging method. If the maximum similarity is less than the preset similarity judgment threshold, a new emotion-intention prototype is created based on the multimodal fusion feature vector of the current behavioral segment, and added to the emotion-intention prototype set. Apply a temporal continuity constraint to the set of emotion-intention prototypes for consecutive behavioral segments, and output the set of emotion-intention prototypes.

5. The method for personalized behavior modeling of digital teachers based on multimodal feature fusion according to claim 4, characterized in that, The confidence-calibrated target emotion-intention state includes: Based on the set of emotion-intention prototypes and the emotion-intention labels corresponding to the behavior segments, an initial emotion-intention state is generated. Write the initial emotion-intention state into the emotion-intention state cache unit in chronological order; Confidence calibration is performed on the current initial emotion-intention state in the emotion-intention state cache unit, including facial expression consistency score, posture consistency score, and speech consistency score. The facial expression consistency score, posture consistency score, and speech consistency score are combined and weighted according to a preset weighting coefficient to obtain the multimodal fusion confidence value. The initial emotion-intent state of the current frame is combined with its corresponding multimodal fusion confidence value to output the target emotion-intent state after confidence calibration.

6. The method for personalized behavior modeling of digital teachers based on multimodal feature fusion according to claim 5, characterized in that, The facial expression consistency score is calculated based on the consistency of the activation values ​​of facial action units in each frame within the i-th behavioral segment: ； in, This represents the activation value of the j-th facial expression action unit in frame t. The average value of the i-th action segment in the j-th action unit. The number of frames in the action segment. Indicates the total number of action units; The variance of the skeleton joint angle set in each time frame is calculated, the average variance of all time frames is taken, and then the attitude consistency score is obtained by subtracting the average variance. Speech consistency score is defined based on the variability of speech signal energy features and fundamental frequency features: ； in, The numbers represent the standard deviation. These represent the average speech signal energy characteristics and the average speech signal fundamental frequency characteristics within a behavioral segment, respectively. These represent the energy characteristics and fundamental frequency characteristics of the speech signal, respectively.

7. The method for personalized behavior modeling of digital teachers based on multimodal feature fusion according to claim 6, characterized in that, The generated digital teacher action trigger intent includes: Calculate the magnitude of the change in state label between the target emotion-intent state in the current frame and the target emotion-intent state in the previous frame. If the change in the state label is less than the hysteresis threshold, the emotion-intention label of the previous frame is used as the hysteresis-corrected emotion-intention state label of the current frame. If the change in the state label is greater than or equal to the hysteresis threshold, the target emotion-intention label of the current frame is used as the hysteresis-corrected emotion-intention state label of the current frame. Write the hysteresis-corrected sentiment-intent state label and the corresponding multimodal fusion confidence value of the current frame into the sentiment-intent context memory unit; Based on the hysteresis-corrected emotion-intention state label and the state distribution statistics contained in the emotion-intention context memory unit of the current frame, digital teacher action triggering intentions are retrieved from the action policy mapping library and generated.

8. The method for personalized behavior modeling of digital teachers based on multimodal feature fusion according to claim 7, characterized in that, The queue for generating compliance action triggering instructions includes: Assign a basic priority coefficient to each digital teacher's action triggering intention, and calculate a comprehensive priority score by combining the context weight coefficient; The set of digital teacher action trigger intentions is sorted from high to low according to the comprehensive priority score to form the initial action trigger sequence; The minimum trigger interval constraint is used to limit the minimum interval between digital teacher action trigger intentions of the same category over a period of time. For the initial action trigger sequence, a minimum trigger interval constraint is applied to each digital teacher action trigger intention based on the action history execution record: the time difference between the current time and the execution time of the most recent action of this category is judged. If the time difference is less than the preset minimum trigger interval, the corresponding digital teacher action trigger intention is marked as a delayed trigger candidate and removed from the current sequence. The remaining actions form the throttled action trigger sequence. Conflict detection is performed on the action trigger sequence after throttling to form a queue of action trigger instructions after scheduling. For each digital teacher action triggering intent in the queue of dispatched action triggering instructions, a compliance check is performed sequentially to determine whether the digital teacher action triggering intent belongs to the security whitelist set and whether the digital teacher action triggering intent meets the posture amplitude and content security requirements. If the digital teacher action triggering intent does not belong to the security whitelist set or does not meet the posture amplitude and content security requirements, it is removed and the remaining actions are retained. All digital teacher action trigger intentions that have passed the compliance verification will be organized into a compliant action trigger instruction queue in sequence.

9. The method for modeling personalized behavior of digital teachers based on multimodal feature fusion according to claim 8, characterized in that, The conflict detection is based on the action conflict matrix to determine whether there is a conflict relationship between any two digital teacher action triggering intentions. If there is a conflict relationship and the action with the lower comprehensive priority score conflicts with the action with the higher comprehensive priority score, then the action with the lower comprehensive priority score is removed and the action with the higher comprehensive priority score is retained.

Images