Intelligent processing method and system for multi-modal data of oral video based on deep learning

Abstract

The invention discloses an intelligent processing method and system for multi-modal data of an oral broadcast video based on deep learning, and relates to the technical field of oral broadcast video data processing, and the method comprises the following steps: 1, collecting multi-modal oral broadcast data in real time, recognizing emotion types, and carrying out the data preprocessing; step 2, constructing an oral playing animation generation model, judging the synchronization degree of the mouth shape, the head action and the audio in the oral playing animation, and taking coordinated regulation measures; 3, judging the consistency of the mouth shape and the emotion intensity in the oral playing animation, and implementing an anomaly repair measure; 4, constructing a depth time sequence prediction model, evaluating the difference degree between the actual oral playing animation and the prediction result of the depth time sequence prediction model, and carrying out video optimization and model tuning; the problems that when mouth shape supplement and animation filling are carried out by AI, precise synchronization of mouth shapes, expressions, head actions and emotional fluctuations is difficult to achieve, so that visual abnormity is caused, and the reality sense and the professional degree of videos are affected are solved.

Description

Technical Field

[0001] This invention relates to the field of voice-over video data processing technology, specifically to a method and system for intelligent processing of multimodal voice-over video data based on deep learning. Background Technology

[0002] With the rapid development of digital content creation and intelligent media technologies, spoken video is widely used in various scenarios such as news broadcasting, education and training, intelligent customer service, and virtual anchors. To enhance the expressiveness and immersiveness of spoken video, the industry is constantly exploring the use of deep learning to intelligently analyze and process multimodal data such as audio, video, text, facial expressions, and emotions, achieving a more natural, fluent, and emotionally rich human-computer interaction experience. Deep learning-based intelligent processing methods and systems for multimodal data in spoken video can integrate information such as speech signals, video frames, lip movements, head posture, and emotional features to automatically perform operations such as lip-sync animation generation, facial expression motion capture, and emotion-driven synchronization.

[0003] For example, the invention patent with announcement number CN117786136A discloses an AI-based personalized story system and interactive story video generation method. This system uses AI technology to personalize the characters, content, and interactive elements of a story, making the generated digital human videos more realistic. The invention generates digital avatars of family members by acquiring user-uploaded custom images, videos, or audio, allowing family members to tell stories and increasing the sense of intimacy between the digital human and the user. By inputting prompts, the system obtains corresponding story text, question-and-answer text, or illustrations, analyzes the story's emotions to generate digital human audio intonation and expressions, drives the digital human to generate digital human actions, and combines the story audio and illustrations to generate corresponding digital human videos. This allows the story to be told by family members, improving the user experience. During interaction, the invention collects user expressions and questions, searches the story library for answers in the question-and-answer section, or calculates the corresponding answers using LLM (Local Language Management) and quickly provides feedback to the user.

[0004] For example, the invention patent with the publication number CN120234761A discloses an AI-based audio-visual fusion control method and device. The method includes: obtaining audio data and video data received by an audio-visual fusion pre-control platform; performing fusion on the audio data and video data based on a preset fusion algorithm to obtain audio-visual fusion data; in response to a first trigger operation triggered by a target system, performing corresponding control operations on the audio-visual fusion data. The audio-visual fusion pre-control platform realizes the linkage control of the audio-visual fusion data through communication with the target system, and the control operations at least include data recognition operations. It can be seen that implementing the present invention can quickly and accurately fuse audio and video data into audio-visual fusion data based on a preset fusion algorithm through the audio-visual fusion pre-control platform, and can achieve precise control of audio-visual data through the linkage between the audio-visual fusion pre-control platform and the target system, which is beneficial to improving the processing efficiency of audio-visual data.

[0005] However, when AI performs tooth filling and score filling on a voice-over video, that is, filling in the lip movement and animation for the non-open-mouth segments, there are often problems of out-of-sync lip movement with the original expression, head movement, and emotional changes, resulting in visual abnormalities such as fake lip movements, unclear speech, and lip movement drift, which affect the realism and professionalism of the video. How to achieve the consistency and natural fusion of lip movement, expression, head movement, and emotion has become a key technical problem in the intelligent processing of multi-modal voice-over videos.

[0006] Therefore, in view of the above problems, there is an urgent need for an intelligent processing method and system for multi-modal data of voice-over videos based on deep learning. Summary of the Invention

[0007] Technical Problems to be Solved

[0008] Aiming at the deficiencies of the prior art, the present invention provides an intelligent processing method and system for multi-modal data of voice-over videos based on deep learning, which solves the problem that it is difficult to achieve precise synchronization of lip movement, expression, head movement, and emotional fluctuations when AI performs lip movement supplementation and animation filling for non-open-mouth segments, resulting in visual abnormalities and affecting the realism and professionalism of the video.

[0009] Technical Solutions

[0010] To achieve the above objectives, this invention provides the following technical solution: a deep learning-based intelligent processing method for multimodal data of spoken video, comprising the following steps: Step 1, real-time acquisition of multimodal spoken data, identification of emotion categories, and preprocessing of the multimodal spoken data; Step 2, construction of a spoken animation generation model based on the multimodal spoken data, and determination of the synchronization between lip movements, head movements, and audio in the spoken animation, thereby implementing coordination and control measures; Step 3, determination of the consistency between lip movements and emotional intensity in the spoken animation, based on the emotion category and the multimodal spoken data and the spoken animation generation model, and implementation of anomaly repair measures based on the consistency between lip movements and emotional intensity; Step 4, construction of a deep temporal prediction model based on the multimodal spoken data and emotion categories, and evaluation of the difference between the actual spoken animation and the prediction results of the deep temporal prediction model, thereby optimizing the video and tuning the model.

[0011] Furthermore, the process of real-time acquisition of multimodal voice-over data and identification of emotion categories, along with the specific preprocessing of the multimodal voice-over data, is as follows: Real-time acquisition of multimodal voice-over data, including audio data, video frames, text transcription, information on the lips, eyes, and eyebrows, and three-dimensional head pose parameters including pitch, yaw, and roll; the facial feature information is extracted using facial key point detection technology and face tracking algorithms; the three-dimensional head pose parameters are obtained by extracting changes in three head angles through three-dimensional face reconstruction and key point analysis methods; the audio data is processed by frame segmentation to extract audio features, and then... The facial landmark detection algorithm extracts the trajectory of lip landmarks from video frames and combines it with an emotion recognition algorithm to label emotion categories. The collected raw audio data, video frames, and text transcription content are denoised, aligned, and synchronized. An adaptive multimodal alignment mechanism is used to dynamically correct the time axis offset of each modality. Facial images in video frames are cropped and normalized. At the same time, combined with the text transcription content, sentence segmentation, word segmentation, and alignment annotation are performed to form timestamps and semantic tags. The processed multimodal audio data is standardized, normalized, and time-series aligned, and finally written into the multimodal audio database in a unified structured format.

[0012] Furthermore, the specific process of constructing a voice-over animation generation model based on multimodal voice-over data is as follows: Based on the preprocessed multimodal voice-over data, a voice-over data feature set is constructed by combining lip key point trajectories and speech features. A sequence generation algorithm of temporal generative adversarial network is adopted, and the voice-over data feature set is used as input to construct a voice-over animation generation model. The voice-over animation generation model strengthens the dynamic consistency between audio, emotion category, and head 3D pose parameters by minimizing the difference between the predicted voice-over animation and the real labeled lip shape, and by combining a synchronization loss function. Finally, a temporally aligned voice-over animation sequence is output, and the lip key points, audio features, head 3D pose parameters, and emotion category corresponding to each frame of voice-over animation are output synchronously.

[0013] Furthermore, combining multimodal voice-over data and a voice-over animation generation model, the specific process for determining the synchronization degree between lip shape, head movements, and audio in a voice-over animation is as follows: Based on the lip keypoints of the current frame output by the voice-over animation generation model, the Euclidean distance between the upper and lower lips is calculated to obtain the lip opening amplitude; for the audio data of the current frame, the root mean square value is calculated and normalized to obtain the audio energy value; the three-dimensional head posture parameters are obtained, represented in radians, and normalized to obtain the head posture change value; historical audio features and video frames are obtained, and the head sway amplitude when syllables and stresses appear is analyzed to construct a dataset corresponding to audio rhythm and head movement; for the current video frame, the synchronization degree between lip shape, head movements, and audio energy is determined by analyzing the audio syllables and stresses. The system uses a dataset corresponding to the sound and head movements to determine the expected head movement at the current moment and obtain the expected head movement value. Based on the sliding window of the current video, it calculates the absolute value of the difference between the mouth opening and closing amplitude and the head posture change value in each frame, and then takes the average value to obtain the normalization factor of normal mouth opening and head movement amplitude. The system subtracts the audio energy value from the mouth opening and closing amplitude and takes the absolute value to obtain the mouth opening and closing gap value. The system subtracts the expected head movement value from the head posture change value and takes the absolute value to obtain the head movement gap value. The system adds the mouth opening and closing gap value and the head movement gap value to obtain the comprehensive gap value. The system subtracts the ratio of the comprehensive gap value to twice the normal mouth opening and head movement amplitude normalization factor from 1 to obtain the mouth and head movement audio synchronization value.

[0014] Furthermore, the specific process of taking coordinated control measures is as follows: when the audio synchronization value of lip shape and head movement is detected to be lower than the synchronization threshold, the current time period is marked as an uncoordinated area; the video generation parameters of the lip-sync animation generation model are dynamically adjusted; the correspondence between the lip-sync animation and the current audio energy value and head posture change value is enhanced; for the uncoordinated area, local regeneration is performed, and the changes in lip shape and head movement are smoothed by frame interpolation algorithm; if the audio synchronization value of lip shape and head movement is lower than the synchronization threshold more than three times, and there are more than three uncoordinated areas, an abnormal log is generated to prompt manual review and intervention; finally, the optimized lip shape and head movement lip-sync animation sequence is output.

[0015] Furthermore, based on emotion categories and combining multimodal spoken data and spoken animation generation models, the specific process for determining the consistency between lip movements and emotional intensity in spoken animation is as follows: For each frame of spoken animation, the lip opening and closing amplitude is obtained, and the difference between the current frame's lip opening and closing amplitude and the previous frame's is calculated to obtain the lip movement change amplitude; the emotion category of each frame is obtained, and an emotion category feature set is constructed based on a publicly available emotional speech database. This feature set is used as input to train a multi-layer deep network emotion recognition model. The last layer of the emotion recognition model uses a normalized exponential function to output the probability of each emotion category. The confidence normalization value of the emotion category is obtained by distributing the data. The confidence normalization value is used as the audio emotion intensity value. The difference between the audio emotion intensity value of the current frame and the previous frame is calculated to obtain the audio emotion intensity change value. Based on the sliding window of the current video, the absolute value of the difference between the lip shape change amplitude and the audio emotion intensity change value in each frame is calculated, and then the average value is taken to obtain the lip shape emotion change normalization factor. The lip shape emotion synchronization degree value is obtained by subtracting the audio emotion intensity change value from the lip shape change amplitude and taking the absolute value. The lip shape emotion consistency evaluation value is obtained by subtracting the ratio of the lip shape emotion synchronization degree value to the lip shape emotion change normalization factor from 1.

[0016] Furthermore, based on the consistency between lip movements and emotional intensity, the specific process for implementing anomaly repair measures is as follows: Calculate the lip-to-emotion consistency assessment value for each frame of the spoken animation; identify segments with lip-to-emotion consistency assessment values ​​below the consistency threshold and mark them as anomalous segments; adjust the video generation parameters of the spoken animation generation model for the anomalous segments; use a multimodal spoken animation database to backtrack historical similar audio and facial expression segments, simultaneously generate multiple sets of candidate spoken animations, use a discriminant network and scoring mechanism to select the spoken animation that best matches the lip movements with the audio emotion, retrieve and refer to the lip movement patterns, and perform sample transfer repair on the current anomalous segments; summarize and output all anomalous segments and timestamps.

[0017] Furthermore, based on multimodal voice-over data and emotion categories, a deep temporal prediction model is constructed. The specific process for evaluating the difference between the actual voice-over animation and the prediction results of the deep temporal prediction model, combining the multimodal voice-over data and the deep temporal prediction model, is as follows: For each frame's corresponding lip keypoints and emotion category, a multimodal voice-over feature set is constructed; based on the multimodal voice-over feature set, static facial expression features are extracted using an open-source facial expression analysis algorithm, and the static facial expression features, lip keypoints, audio features, and emotion category multimodal features are concatenated in a temporal sequence; the concatenated multimodal features are input into a temporal deep learning algorithm for training, constructing a deep temporal prediction model, and outputting the predicted values ​​of mouth opening amplitude, mouth movement, and head posture change at each moment; the actual mouth opening amplitude of each frame of the voice-over animation is obtained. The predicted values ​​for mouth opening and closing amplitude are obtained by subtracting the predicted values ​​from the actual mouth opening and closing amplitude and taking the absolute value. Simultaneously, the actual head posture change value and the predicted head posture change value for each frame of the spoken animation are obtained, and the head movement difference is obtained by subtracting the predicted head posture change value from the actual head posture change value and taking the absolute value. The lip keypoint trajectory is obtained, and lip-related movements are identified. Using an open-source facial expression analysis algorithm, the activation score of the facial action unit corresponding to each frame's lip-related movements is output as the actual mouth movement evaluation value. The predicted mouth movement value is obtained, and the mouth movement difference is obtained by subtracting the predicted mouth movement value from the actual mouth movement evaluation value and taking the absolute value. The mouth opening and closing difference, head movement difference, and mouth movement difference are added together, and the average value is calculated to obtain the multimodal comprehensive anomaly evaluation value.

[0018] Furthermore, the specific process of video optimization and model tuning is as follows: Frames with multimodal comprehensive anomaly evaluation values ​​higher than the anomaly threshold are labeled, and an interactive review interface is generated. The audio synchronization value of lip-sync and head-movement, the lip-sync emotion consistency evaluation value, and the multimodal comprehensive anomaly evaluation value are cross-validated to identify the anomaly type and cause, and to correct the coordination of lip-sync, head-movement, and facial expression in the narration animation. All review and correction operations are archived in real time, and the narration animation generation model and the deep temporal prediction model are trained and tuned through online learning and incremental fine-tuning methods. The optimized and reviewed high-fidelity video is output, unifying lip-sync, head-movement, facial expression, and emotion, and a detailed and interpretable report is generated. The multimodal comprehensive anomaly evaluation value and the corrected frames are recorded to achieve closed-loop optimization.

[0019] The second aspect of this invention provides a deep learning-based intelligent processing system for multimodal data of spoken video, comprising: a data acquisition and preprocessing module for real-time acquisition of multimodal spoken data, identification of emotion categories, and preprocessing of the multimodal spoken data; a joint modeling and synchronization judgment module for constructing a spoken animation generation model based on the multimodal spoken data, and combining the multimodal spoken data and the spoken animation generation model to determine the degree of synchronization between lip movements, head movements, and audio in the spoken animation, thereby taking coordination and control measures; a consistency detection and adaptive repair module for determining the consistency between lip movements and emotional intensity in the spoken animation based on the emotion category, combining the multimodal spoken data and the spoken animation generation model, and implementing anomaly repair measures based on the consistency between lip movements and emotional intensity; and an intelligent review and high-fidelity output module for constructing a deep temporal prediction model based on the multimodal spoken data and emotion categories, and combining the multimodal spoken data and the deep temporal prediction model to evaluate the degree of difference between the actual spoken animation and the prediction results of the deep temporal prediction model, thereby optimizing the video and tuning the model.

[0020] Beneficial effects

[0021] The present invention has the following beneficial effects:

[0022] (1) This invention achieves unified structured management of multimodal data by accurately collecting, preprocessing and standardizing the time sequence alignment of multi-source heterogeneous data such as audio, video, text, facial expressions, emotions and head movements of spoken videos, providing a data foundation for subsequent intelligent processing.

[0023] (2) This invention integrates multimodal temporal modeling and generative adversarial networks, and can dynamically generate natural and smooth mouth shape and facial expression animations that are highly consistent with the context based on the multimodal features of audio pronunciation, emotional changes and head movements, thereby improving the visual naturalness and emotional coherence of AI mouth shape completion.

[0024] (3) This invention automatically detects and accurately identifies abnormal areas where mouth shape, facial expression and head movement are out of sync by designing end-to-end multimodal consistency loss and various synchronization and anomaly evaluation indicators. It also adopts self-rescue measures such as local regeneration, parameter fine-tuning and history migration to achieve automatic optimization and efficient repair, thereby improving the naturalness and realism of the overall video.

[0025] (4) This invention automatically labels and interactively reviews abnormal segments, and the results of manual correction and review are fed back to the model in real time, driving the continuous optimization of the voice-over animation generation model and the deep time series prediction model, forming a closed-loop intelligent evolution of model, detection, review and retraining, effectively ensuring the professionalism and consistency of the output video.

[0026] Of course, any product implementing this invention does not necessarily need to achieve all of the advantages described above at the same time. Attached Figure Description

[0027] Figure 1 Flowchart of a deep learning-based intelligent processing method for multimodal data of spoken video;

[0028] Figure 2 This is a structural diagram of a deep learning-based intelligent processing system for multimodal data of voice-over video.

[0029] Figure 3 Radar chart showing the multimodal difference distribution at different times;

[0030] Figure 4 This is a trend chart of the multimodal integrated anomaly assessment value. Detailed Implementation

[0031] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. As those skilled in the art will understand, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0032] Please see Figures 1-4 This invention provides a technical solution: a method and system for intelligent processing of multimodal data of spoken video based on deep learning, comprising the following steps: Step 1, real-time acquisition of multimodal spoken data, identification of emotion categories, and preprocessing of the multimodal spoken data; Step 2, construction of a spoken animation generation model based on the multimodal spoken data, and determination of the synchronization between lip movements, head movements, and audio in the spoken animation based on the multimodal spoken data and the spoken animation generation model, thereby taking coordination and control measures; Step 3, determination of the consistency between lip movements and emotional intensity in the spoken animation based on the emotion category, combined with the multimodal spoken data and the spoken animation generation model, and implementation of anomaly repair measures based on the consistency between lip movements and emotional intensity; Step 4, construction of a deep temporal prediction model based on the multimodal spoken data and the emotion category, evaluation of the difference between the actual spoken animation and the prediction results of the deep temporal prediction model based on the multimodal spoken data and the deep temporal prediction model, thereby performing video optimization and model tuning.

[0033] Specifically, the process of real-time acquisition of multimodal speech data and identification of emotion categories, along with data preprocessing, involves: real-time acquisition of multimodal speech data from the original speech video. This multimodal speech data includes: audio data, video frames, text transcription, information on the lips, eyes, and eyebrows, and three-dimensional head posture parameters including pitch, yaw, and roll. The text transcription can be automatically extracted using speech recognition. The facial landmark information is extracted using facial key point detection technology and face tracking algorithms. The three-dimensional head posture parameters are obtained by extracting changes in three head angles using three-dimensional face reconstruction and key point analysis methods. The audio data is processed by frame segmentation to extract audio features such as MFCC, phonemes, pitch, and speech rate, thereby improving the processing capabilities of subsequent speech video generation models and deep temporal prediction models for pronunciation details and speech rhythm. Through human... The facial landmark detection algorithm extracts the trajectory of lip landmarks from video frames and combines it with an emotion recognition algorithm to label emotion categories, such as happiness, calmness, and tension, to achieve accurate quantification of emotional information in the spoken video. The collected raw audio data, video frames, and transcribed text are denoised, aligned, and synchronized to eliminate noise and latency between multimodal data and improve temporal consistency. An adaptive multimodal alignment mechanism dynamically corrects temporal axis offsets for each modality. Facial images in video frames are cropped and normalized to standardize facial feature representations under different shooting conditions, ensuring uniformity of feature distribution. Simultaneously, combined with transcribed text, sentence segmentation, word segmentation, and alignment annotation are performed to form timestamps and semantic tags. The processed multimodal spoken data undergoes standardization, normalization, and temporal alignment processing before being written into the multimodal spoken database in a unified structured format.

[0034] This implementation scheme achieves precise fusion and temporal alignment of audio, video, text, and facial information in spoken video through real-time acquisition, emotion recognition, and high-quality preprocessing of multimodal spoken video data. This not only significantly improves data consistency and structure but also provides a solid data foundation for subsequent models to deeply model pronunciation details, speech rhythm, facial expressions, and emotional states. It significantly enhances the spoken video generation and prediction model's ability to understand and express complex multimodal features, effectively ensuring the robustness and professionalism of subsequent stages, thereby improving the naturalness, emotional richness, and intelligent processing level of spoken video.

[0035] Specifically, the process of constructing a voice-over animation generation model based on multimodal voice-over data is as follows: Based on the preprocessed multimodal voice-over data, a voice-over data feature set is constructed by combining lip key point trajectories and speech features. The sequence generation algorithm of temporal generative adversarial network is preferred. The voice-over data feature set is used as input to construct a voice-over animation generation model. During the training process, the generator network is responsible for outputting mouth shape, facial expression and head movement parameters corresponding to the temporal sequence of real voice-over animation. The discriminator network is used to judge the consistency between the generated result and the manually annotated animation. The voice-over animation generation model minimizes the difference between the predicted voice-over animation and the actual annotated lip movements. By combining a synchronization loss function, it focuses on constraining the dynamic coupling relationship between audio and lip movements, emotions and lip movements, and head movements and audio. It strengthens the dynamic consistency between audio, emotion categories, and head 3D pose parameters, achieving high-precision linkage and synchronous modeling between different modal signals. Through alternating optimization of the generator and discriminator, it finally outputs a time-aligned voice-over animation sequence, including key points of body parts and facial expression parameters. It also synchronously outputs the lip key points, audio features, head 3D pose parameters, and emotion categories corresponding to each frame of voice-over animation, forming a high-quality data stream that can be directly used for voice-over video animation completion.

[0036] This implementation scheme achieves deep feature fusion and dynamic temporal modeling of multimodal voice-over data, enhancing the voice-over animation generation model's comprehensive understanding and collaborative modeling capabilities of multi-source signals including speech, facial expressions, and head movements. Through advanced algorithms using generative adversarial networks and diffusion models, it can output highly temporally aligned, natural, and synchronized voice-over animation sequences, ensuring high consistency of lip movements, facial expressions, and emotional information across frames. This provides high-quality, practical data support for voice-over video animation completion, effectively improving the professionalism and realism of voice-over videos.

[0037] Specifically, the process of determining the synchronization between lip shape, head movements, and audio in a spoken animation, combining multimodal spoken data and a spoken animation generation model, is as follows: Based on the lip keypoints of the current frame output by the spoken animation generation model, the Euclidean distance between the upper and lower lips is calculated to obtain the lip opening amplitude, used to quantify the actual opening and closing state of the mouth; for the audio data of the current frame, the root mean square value is calculated and normalized to obtain the audio energy value, accurately reflecting the speech intensity and providing a basis for judging the synchronization between lip shape and pronunciation; three-dimensional head posture parameters are obtained, represented in radians, and normalized to obtain head posture change values, comprehensively reflecting the head movement state; historical audio features and video frames are obtained, and the head sway amplitude when syllables and stresses appear is analyzed to construct a dataset corresponding to audio rhythm and head movement; for the current video frame, the synchronization of the current audio rhythm and head movement is determined using this dataset. The expected head movements are obtained at all times. Based on the sliding window of the current video, the absolute value of the difference between the mouth opening and closing amplitude and the head posture change value in each frame is calculated, and then the average value is taken to obtain the normalization factor of normal mouth shape and head movement amplitude, so as to achieve dynamic normalization processing between different segments. The mouth opening and closing amplitude is subtracted from the audio energy value and the absolute value is taken to obtain the mouth opening and closing gap value, which is used to measure the consistency between mouth shape and audio production. The head posture change value is subtracted from the expected head movement value and the absolute value is taken to obtain the head movement gap value, which is used to measure the synchronization between the actual head movement and the ideal state. The mouth opening and closing gap value and the head movement gap value are added to obtain the comprehensive gap value. The ratio of the comprehensive gap value to twice the normal mouth shape and head movement amplitude normalization factor is subtracted from 1 to obtain the mouth shape, head movement and audio synchronization value, which accurately evaluates the dynamic coordination of mouth shape, head movement and audio in the current frame of the voice-over animation.

[0038] The specific formula for the lip-sync value is as follows:

[0039] ;

[0040] In the formula, The audio synchronization value of lip shape and head movement is used to measure the degree of synchronization between lip shape, head movement and audio signal in a certain frame from the perspective of physical synchronization. The closer the audio synchronization value of lip shape and head movement is to 1, the more synchronized and natural the lip shape, head movement and vocalization are. The closer the audio synchronization value of lip shape and head movement is to 0, the more asynchronous and unnatural there is between different modalities, and the more likely fake lip shape phenomenon will appear. It indicates the range of mouth opening and closing, reflecting the actual opening and closing state of the lips at that moment, and is one of the core indicators for evaluating the naturalness of spoken animation; It represents the audio energy value. When the audio energy is high, the mouth opens wider, which can reflect the dynamics of the mouth shape during speech and is the basis for judging the synchronization of mouth shape and speech. This represents the difference between the actual and expected opening and closing of the mouth in the animated speech; This indicates changes in head posture. Natural head movements, such as nodding or head shaking while speaking, should be coordinated with mouth shape and audio. Excessive inconsistency may lead to stiff facial expressions and fake mouth shapes. It represents the expected value of head movement, providing an ideal head movement reference corresponding to the rhythm for each audio frame, making it easier to determine whether the current head movement is synchronized with the vocalization; This represents the head movement difference value, which measures the difference between changes in head posture and the expected head movement for the audio rhythm; This represents the normalization factor between normal mouth shape and head movement amplitude, ensuring the rationality of the denominator value and stabilizing the audio synchronization value of mouth shape and head movement between 0 and 1, which facilitates comparison of different samples and different time periods. To standardize the data, the differences in mouth opening and closing and head movement were normalized to ensure that the synchronization was actually comparable.

[0041] This implementation scheme precisely quantifies the synchronization relationship between lip movements, head posture changes, and audio energy in spoken animation by jointly analyzing and dynamically normalizing multimodal data of lip opening and closing amplitude, head posture changes, and audio energy. This effectively improves the spoken animation generation model's ability to recognize and model the coordination and consistency of pronunciation dynamics, facial movements, and head movements, providing a scientific quantitative basis for timely detection and correction of multimodal asynchrony issues, thereby significantly enhancing the naturalness, realism, and professional expressiveness of spoken videos.

[0042] Specifically, the process of taking coordinated control measures is as follows: when the audio synchronization value of lip-sync and head-movement is detected to be lower than the synchronization threshold, the current time period is marked as an incoordination region; the video generation parameters of the lip-sync animation generation model are dynamically adjusted, including the time window size and the weight of the synchronization loss function, to adaptively optimize the lip-sync animation generation model's response capability to incoordination regions; the correspondence between the lip-sync animation and the current audio energy value and head posture change value is enhanced, thereby improving the fitting ability of lip-sync with audio and head-movement; for incoordination regions, local regeneration is performed, that is, local resampling and generation of lip-sync animation sequences, and the changes in lip-sync and head-movement are smoothed by frame interpolation algorithms; if the audio synchronization value of lip-sync and head-movement is lower than the synchronization threshold more than three times and there are more than three incoordination regions, an abnormal log is generated, prompting manual review and intervention to ensure the safety of complex and extreme abnormal scenarios; finally, the optimized lip-sync and head-movement lip-sync animation sequence is output.

[0043] This implementation scheme achieves dynamic detection and intelligent control of the multimodal synchronization of lip movements, head movements, and audio in narrated animations. For areas of inconsistency, it adaptively adjusts model parameters and automatically performs local optimization, regeneration, and temporal smoothing, effectively improving the coordination and naturalness between lip movements, audio, and head movements. Simultaneously, through anomaly logging and manual review mechanisms, it further enhances its fallback capabilities, ensuring the professionalism, stability, and high-quality performance of the final output video.

[0044] Specifically, based on emotion categories and combining multimodal spoken data and spoken animation generation models, the process for determining the consistency between lip movements and emotional intensity in spoken animation is as follows: For each frame of spoken animation, the amplitude of lip opening and closing is obtained, and the difference between the current frame's lip opening and closing amplitude and the previous frame's is calculated to obtain the amplitude of lip movement change, thereby quantifying the temporal trend of lip movement dynamic features; the emotion category of each frame is obtained, and an emotion category feature set is constructed based on a publicly available emotional speech database. This feature set is then used as input to train a multi-layer deep network emotion recognition model. The last layer of the emotion recognition model uses a normalized exponential function, i.e., the Softmax function. The model outputs the probability distribution of each emotion category to obtain the normalized confidence score of each emotion category. The emotion recognition model employs an appropriate deep learning structure based on different types of input data. For audio and speech signals, an acoustic feature extraction and emotion classification model based on convolutional neural networks and recurrent neural networks is used. For text data, a pre-trained language model with a Transformer structure is used for text sentiment analysis. In multimodal scenarios, audio features, text features, and facial expression features from video frames can be concatenated through a feature fusion module and input into a multimodal fusion network, such as a multimodal Transformer network, for emotion recognition and classification. The model outputs the probability distribution of each emotion category, such as happiness, sadness, and tension, and obtains the confidence score for each emotion category after normalization using the Softmax function, achieving quantitative analysis and output of emotional states. The confidence level normalized value is used as the audio emotional intensity value, which ranges from 0 to 1. This effectively represents the emotional expression intensity of the current frame's audio. The difference between the current frame's audio emotional intensity value and the previous frame's value is calculated to obtain the audio emotional intensity change value, enabling quantitative tracking of the dynamic fluctuations in speech emotion. Based on the sliding window of the current video, the absolute value of the difference between the lip-shape change amplitude and the audio emotional intensity change value in each frame is calculated, and then the average value is taken to obtain the lip-shape emotional change normalization factor. The lip-shape emotional synchronization degree value is obtained by subtracting the audio emotional intensity change value from the lip-shape change amplitude and taking the absolute value, reflecting the instantaneous coordination between lip-shape movement and emotional fluctuation. The ratio of the lip-shape emotional synchronization degree value to the lip-shape emotional change normalization factor is subtracted from 1 to obtain the lip-shape emotional consistency evaluation value, thereby quantifying the consistency between lip-shape and emotional expression in each frame of the spoken animation.

[0045] The specific formula for the lip-shape emotion consistency assessment value is as follows:

[0046] ;

[0047] In the formula, The lip-sync value is used to measure the consistency between the lip-sync animation and the changes in the emotional intensity of the audio in each frame. The closer the lip-sync value is to 1, the more synchronized the dynamic changes of the lip shape are with the emotional changes, and the lip expression and emotional expression are coordinated and natural. The closer the lip-sync value is to 0, the more discrepancy there is between the lip movement and the emotional expression, and the more likely there will be fake expressions and lip-sync issues. It indicates the range of mouth shape changes, reflecting the instantaneous movement range of the voice-over animation, that is, the speed and direction of the mouth shape changes at the current moment; It represents the change value of audio emotional intensity, reflecting the current speaker's emotional fluctuations in the audio, such as the speed of change in the intensity of emotions like happiness, excitement, and sadness; This value represents the degree of synchronization between lip movements and emotional expression, measuring the synchronicity between the amplitude of lip movement changes and the amplitude of emotional changes; the smaller the difference between the two, the more coordinated the movement of the lip movements and the emotional expression. This represents a normalization factor for changes in lip-shape emotion, ensuring a stable denominator and normalizing the result to the range of 0 to 1. This facilitates comparison and analysis between different behaviors and videos, and avoids distortion of the consistency assessment value of lip-shape emotion due to different scales.

[0048] This implementation scheme enables precise quantification of the consistency between lip-sync dynamics and emotional intensity changes in narrated animations. Through a multi-layered deep network emotion recognition model and a sliding window normalization mechanism, it tracks and analyzes the synchronization relationship between lip-sync and audio emotion in each frame in real time, promptly identifying and evaluating segments where lip-sync and emotional expression are inconsistent. This improves the coordination and naturalness of lip-sync and emotional expression in narrated videos, thereby enhancing the emotional expressiveness and viewing experience of the final video.

[0049] Specifically, based on the consistency between lip movements and emotional intensity, the specific process for implementing anomaly repair measures is as follows: Calculate the lip-sync emotional consistency evaluation value for each frame of the spoken animation; identify segments with lip-sync emotional consistency evaluation values ​​below the consistency threshold and mark them as anomalous segments; adjust the video generation parameters of the spoken animation generation model for the anomalous segments, including: increasing the weight of the emotional synchronization loss function for the segments, so that the model pays more attention to the coordination between lip movements and audio emotions when generating interval animations; narrowing the animation generation window, increasing the frame sampling density, enhancing the ability to model local details, and improving the dynamic smoothness and naturalness of anomalous segments; and achieving adaptive optimization of the model to improve the ability to repair anomalous areas. By leveraging a multimodal lip-sync database to trace back similar audio and facial expression clips, multiple candidate lip-sync animations are generated simultaneously. A discriminant network and scoring mechanism are used to select the lip-sync animation that best matches the audio emotion. The discriminant network and scoring mechanism automatically judge and score the generated and repaired lip-sync animations in terms of naturalness and multimodal consistency through deep learning models and quantitative evaluation methods, thereby intelligently selecting the optimal and most coordinated animation result. By retrieving and referring to the lip movement patterns, sample transfer repair is performed on the current abnormal segments to achieve efficient self-healing in abnormal scenarios. All abnormal segments and timestamps are summarized and output for subsequent review, manual intervention, and system performance traceability.

[0050] This implementation scheme achieves real-time detection and intelligent repair of the consistency between lip movements and emotional expression in spoken animations. It not only promptly identifies and accurately marks abnormal areas but also generates high-quality animation results that best match the audio's emotional tone through model adaptive optimization, multimodal historical sample transfer, and automatic filtering mechanisms. The deep integration of the discriminant network and scoring mechanism significantly improves the automation and intelligence of anomaly repair, effectively ensuring the naturalness, emotional expressiveness, and professionalism of the final spoken video. Simultaneously, it provides complete traceability and data support for anomaly management and subsequent manual review.

[0051] Specifically, based on multimodal voice-over data and emotion categories, a deep temporal prediction model is constructed. The process of evaluating the difference between the actual voice-over animation and the prediction results of the deep temporal prediction model, combining the multimodal voice-over data and the deep temporal prediction model, is as follows: Based on the voice-over animation sequence output by the voice-over animation generation model, a multimodal voice-over feature set is constructed for each frame's corresponding lip keypoints and emotion category. Based on the multimodal voice-over feature set, an open-source facial expression analysis algorithm based on CNN-based ResNet-50 is used to extract static facial expression features. Simultaneously, lip keypoints, audio features, and emotion category information corresponding to each frame are extracted, and the static facial expression features, lip keypoints, audio features, and emotion category multimodal features are concatenated in a temporal sequence. In the specific implementation, the extraction of static facial expression features uses a facial expression analysis algorithm to process each frame of static image independently, obtaining the facial action unit activation score for each frame. Lip keypoints and audio features are obtained synchronously with the expression features. The features of each modality are aligned frame by frame and concatenated into an input sequence, which serves as the input to the temporal deep learning algorithm. The concatenated multimodal features are input into a temporal deep learning algorithm for training. In the specific training process, a CNN model is first used to extract static facial expression features on static frames. Then, the static facial expression features are concatenated with lip key points, audio features, and emotion category features in the temporal dimension at the frame level to form multimodal sequence features, which are then input into the temporal network for end-to-end modeling and training.The loss function of the temporal network can include the mean squared error between the mouth opening amplitude and the predicted value, as well as the multimodal consistency index of the posture synchronization loss. This allows for the construction of a deep temporal prediction model, outputting the predicted values ​​of mouth opening amplitude, mouth movements, and head posture changes at each time step. The model acquires the actual mouth opening amplitude and the predicted value for each frame of the animated video, subtracting the predicted value from the actual amplitude and taking the absolute value to obtain the mouth opening difference. Simultaneously, it acquires the actual head posture change value and the predicted value for each frame of the animated video, subtracting the predicted value from the actual posture change and taking the absolute value to obtain the head movement difference. Finally, it acquires the lip keypoint trajectory and identifies lip-related movements, such as upturned corners of the mouth and corners of the mouth. Pulling down, lips opening; using an open-source facial expression analysis algorithm and an internationally recognized facial motion coding system, the algorithm outputs the facial motion unit activation score corresponding to each frame of lip-related movements as the actual mouth movement evaluation value. The facial motion unit activation score refers to the quantification of the activation degree of each lip motion unit in each frame of facial expression by the facial expression recognition algorithm, represented by a continuous real number between 0 and 1, to accurately reflect the actual state of the movements of each part of the face; the predicted value of mouth movements is obtained, and the mouth movement difference is obtained by subtracting the predicted value of mouth movements from the actual mouth movement evaluation value and taking the absolute value, which measures the differences in facial expressions; the difference in mouth opening and closing, the difference in head movements, and the difference in mouth movements are added together and then averaged to obtain the multimodal comprehensive anomaly evaluation value.

[0052] The specific formula for the multimodal integrated anomaly assessment value is as follows:

[0053] ;

[0054] In the formula, The multimodal comprehensive anomaly assessment value is used to evaluate the degree of difference between the generated mouth shape, mouth expression, and head movement of a certain frame and the expected output of the depth temporal prediction model in terms of consistency with the ideal model. It measures the difference between the actual and generated results in mouth shape, mouth expression, and head movement, and comprehensively judges the anomalies between mouth shape, movement, and expression. The higher the multimodal comprehensive anomaly assessment value, the larger the deviation between the mouth shape, expression, and head movement of the current frame and the predicted value, and visual anomalies and unnatural phenomena will appear. It represents the actual range of mouth opening and closing, reflecting the actual lip opening and closing situation in the current frame, and is an important data for assessing the naturalness of mouth shape and pronunciation synchronization; It represents the predicted value of mouth opening and closing amplitude, which represents the most natural and matching mouth shape state in theory, and is a benchmark for measuring the consistency between the actual situation and the model target. It represents the actual mouth movement assessment value, reflecting the current facial expression state, and is of great significance for identifying abnormal expressions, such as fake smiles and blank expressions; This represents the predicted value of mouth movements, serving as a benchmark for comparison with the actual mouth movement evaluation value, and is used to capture the difference between mouth expressions and the ideal state; It represents the actual changes in head posture, reflecting the actual head movement in the video, and helps to identify problems such as stiff facial expressions and abnormal head movements; This represents the predicted value of head posture changes, which is used to compare with the actual detected value to identify abnormal situations where head movements do not match expectations.

[0055] Based on the differences in mouth opening and closing, head movement, and mouth movement at different times, five sets of multimodal comprehensive anomaly assessment parameter data are provided, and the multimodal comprehensive anomaly assessment value is calculated. Table 1 shows the multimodal comprehensive anomaly assessment value data.

[0056] surface Multimodal integrated anomaly assessment value data table

[0057]

[0058] like Figure 3 As shown, this is a radar chart of the multimodal difference distribution at various time points provided in the embodiments of this application. It displays the distribution characteristics of the multimodal comprehensive anomaly assessment values ​​at five time points. Each line represents a frame, focusing on three core indicators: mouth opening and closing difference, mouth movement difference, and head movement difference. Different colored lines represent different time points, and each vertex corresponds to the error magnitude of a mode. The further the vertex is from the center of the circle, the higher the degree of anomaly in that mode at the current time. According to Table 1 and... Figure 3 It can be seen that at time 5, i.e. the purple line, the differences in mouth movement and head movement are significantly higher than at other times, indicating that the facial expression and head movement in this frame deviate significantly from the prediction results, and it is the time with the greatest risk of anomaly. At time 1, i.e. the blue line, the whole is relatively close to the center of the circle, indicating that the errors are small and the generated voice-over animation is consistent with the prediction.

[0059] like Figure 4 As shown, this is a trend chart of the multimodal comprehensive anomaly evaluation value provided in an embodiment of this application, displaying the changes in the multimodal comprehensive anomaly evaluation value at five time points. Each point represents the degree of comprehensive deviation between the animation at the corresponding time point and the model prediction in modalities such as mouth shape, facial expression, and head movement; according to Table 1 and Figure 4 It can be seen that the multimodal comprehensive anomaly assessment value is the highest at time 5, while the multimodal comprehensive anomaly assessment values ​​are lower at times 1 and 4. Overall, the multimodal comprehensive anomaly assessment value first rises, then falls, and then rises sharply, reflecting the abnormal fluctuations and possible abnormal sudden points in the dynamic process.

[0060] This implementation plan comprehensively integrates multimodal voice-over data and emotional information, and uses a deep temporal prediction model to perform high-precision dynamic modeling and consistent prediction of multidimensional features of lip movements, head movements, and lip actions. It not only quantitatively evaluates the differences between each frame of actual voice-over animation and the ideal model output in terms of multimodal features of lip movements, head movements, and lip actions, but also achieves standardized judgment of animation naturalness and expression accuracy through internationally recognized facial motion coding systems and facial motion unit activation scores. The multimodal comprehensive anomaly assessment value provides a scientific basis for subsequent anomaly detection, optimization and correction, and high-fidelity review, effectively improving the overall performance of voice-over animation in terms of naturalness, emotional expression, and professional reliability.

[0061] Specifically, the process of video optimization and model tuning is as follows: Frames with multimodal comprehensive anomaly evaluation values ​​exceeding the anomaly threshold are labeled, and an interactive review interface is generated for easy manual review and correction, enabling efficient and accurate location and labeling of anomaly areas; lip-sync and head-motion audio synchronization values, lip-expression consistency evaluation values, and multimodal comprehensive anomaly evaluation values ​​are cross-validated to identify anomaly types and causes, correcting the coordination of lip-sync, head-motion, and facial expressions in the spoken animation, achieving targeted optimization and adjustment for different anomalies; all review and correction operations are archived in real time at the event, frame, and segment levels, supporting graded and weighted sample processing; and the spoken animation generation model and deep temporal prediction model are trained and tuned through online learning and incremental fine-tuning methods to continuously improve their adaptive capabilities and processing accuracy for complex scenes; high-fidelity videos that have been optimized and reviewed are output, unifying lip-sync, head-motion, facial expressions, and emotions, and generating detailed and interpretable reports, recording multimodal comprehensive anomaly evaluation values ​​and corrected frames to achieve closed-loop optimization.

[0062] This implementation plan, through automatic anomaly labeling, interactive review and correction, and a closed-loop online model learning system, enables efficient localization, intelligent identification, and precise optimization of multimodal anomalies in spoken animation. It not only improves the overall coordination and naturalness of lip movements, head movements, facial expressions, and emotions, but also enhances the adaptive handling capabilities for complex anomaly scenarios. Detailed data archiving and interpretable reports provide a scientific basis for subsequent performance tracking and continuous optimization, ensuring the output of high-fidelity, professional, and sustainably self-learning intelligent spoken video results.

[0063] Reference Figure 2As shown, the second aspect of this invention provides a deep learning-based intelligent processing system for multimodal data of spoken video, applied to the aforementioned deep learning-based intelligent processing method for multimodal data of spoken video. The system includes: a data acquisition and preprocessing module for real-time acquisition of multimodal spoken data, identification of emotion categories, and preprocessing of the multimodal spoken data; and a joint modeling and synchronization judgment module for constructing a spoken animation generation model based on the multimodal spoken data, and combining the multimodal spoken data and the spoken animation generation model to determine the degree of synchronization between lip movements, head movements, and audio in the spoken animation, thereby... Coordinated control measures are implemented; a consistency detection and adaptive repair module is used to determine the consistency between lip movements and emotional intensity in spoken animations based on emotion categories, combined with multimodal spoken data and spoken animation generation models, and to implement anomaly repair measures based on the consistency between lip movements and emotional intensity; an intelligent review and high-fidelity output module is used to construct a deep temporal prediction model based on multimodal spoken data and emotion categories, and to evaluate the degree of difference between the actual spoken animation and the prediction results of the deep temporal prediction model, thereby optimizing the video and tuning the model.

[0064] This implementation plan integrates multiple modules, including data acquisition and preprocessing, joint modeling and synchronous judgment, consistency detection and adaptive repair, intelligent review and high-fidelity output, to achieve efficient fusion and deep understanding of multimodal data in spoken video. It can dynamically evaluate and optimize the synchronization and emotional consistency of lip movements, head gestures, and audio. It possesses the ability to automatically detect and intelligently repair multimodal anomalies, supports continuous adaptive optimization and closed-loop upgrades of the model, and greatly improves the overall level of spoken video in terms of naturalness, emotional expression, and professional quality, providing strong technical support for high-fidelity, intelligent video generation and review.

[0065] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus.

[0066] The preferred embodiments of the present invention disclosed above are merely illustrative of the invention. These preferred embodiments do not exhaustively describe all details, nor do they limit the invention to the specific implementations described. As those skilled in the art will understand, many modifications and variations can be made based on the content of this specification. This specification selects and specifically describes these embodiments to better explain the principles and practical applications of the invention, thereby enabling those skilled in the art to better understand and utilize the invention. The invention is limited only by the claims and their full scope and equivalents.

Claims

1. A deep learning-based intelligent processing method for multimodal data of spoken video, characterized in that, Includes the following steps: Step 1: Collect multimodal voice-over data in real time, identify emotion categories, and preprocess the multimodal voice-over data; Step 2: Construct a voice-over animation generation model based on multimodal voice-over data, and combine the multimodal voice-over data and the voice-over animation generation model to determine the degree of synchronization between mouth shape, head movements and audio in the voice-over animation, so as to take coordination and control measures. Step 3: Based on the emotion category, combined with multimodal voice-over data and voice-over animation generation model, determine the consistency between lip movements and emotional intensity in the voice-over animation, and implement anomaly repair measures based on the consistency between lip movements and emotional intensity. Step four: Based on multimodal voice-over data and emotion categories, construct a deep temporal prediction model. Combine the multimodal voice-over data and the deep temporal prediction model to evaluate the degree of difference between the actual voice-over animation and the prediction results of the deep temporal prediction model, thereby optimizing the video and tuning the model.

2. The intelligent processing method for multimodal data of spoken video based on deep learning according to claim 1, characterized in that, The specific process of real-time acquisition of multimodal spoken data, identification of emotion categories, and data preprocessing of the multimodal spoken data is as follows: Real-time acquisition of multimodal voice-over data, including audio data, video frames, text transcription content, information on the lips, eyes, and eyebrows, and three-dimensional head pose parameters including pitch, yaw, and roll; among them, the part information is extracted through facial key point detection technology and face tracking algorithm; the three-dimensional head pose parameters are obtained by extracting the changes in the three angles of the head through three-dimensional face reconstruction and key point analysis methods. The audio data is processed by frame segmentation to extract audio features. Lip key point trajectories are extracted from video frames using a facial key point detection algorithm, and emotion categories are labeled using an emotion recognition algorithm. The collected raw audio data, video frames, and text transcription content are denoised, aligned, and synchronized. An adaptive multimodal alignment mechanism is used to dynamically correct the time axis offset of each modality. Facial images in video frames are cropped and normalized. At the same time, combined with the text transcription content, sentence segmentation, word segmentation, and alignment annotation are performed to form timestamps and semantic tags. The processed multimodal audio data is standardized, normalized, and time-series aligned, and finally written into the multimodal audio database in a unified structured format.

3. The intelligent processing method for multimodal data of spoken video based on deep learning according to claim 1, characterized in that, The specific process of constructing a voice-over animation generation model based on multimodal voice-over data is as follows: Based on preprocessed multimodal speech data, a speech data feature set is constructed by combining lip keypoint trajectories and speech features. A sequence generation algorithm using temporal generative adversarial networks is adopted, and a speech animation generation model is constructed using the speech data feature set as input. The speech animation generation model enhances the dynamic consistency between audio, emotion category, and head 3D pose parameters by minimizing the difference between the predicted speech animation and the actual labeled lip shape, and by combining a synchronization loss function. Finally, a temporally aligned speech animation sequence is output, and the lip keypoints, audio features, head 3D pose parameters, and emotion category corresponding to each frame of speech animation are output synchronously.

4. The intelligent processing method for multimodal data of spoken video based on deep learning according to claim 1, characterized in that, The specific process of determining the synchronization degree between lip movements, head actions, and audio in a spoken animation by combining multimodal spoken data and a spoken animation generation model is as follows: Based on the lip key points of the current frame output by the voice-over animation generation model, the Euclidean distance between the upper and lower lips is calculated to obtain the mouth opening and closing amplitude; for the audio data of the current frame, the root mean square value is calculated and normalized to obtain the audio energy value. Obtain the three-dimensional head pose parameters, represent them in radians, and perform normalization to obtain the head pose change values; Acquire historical audio features and video frames, analyze the head sway amplitude when syllables and stresses appear, construct a dataset corresponding to audio rhythm and head movement, and for the current video frame, determine the expected head movement at the current moment through the dataset corresponding to audio rhythm and head movement to obtain the expected value of head movement. Based on the sliding window of the current video, calculate the absolute value of the difference between the mouth opening and closing amplitude and the head posture change value in each frame, and then take the average value to obtain the normalization factor of normal mouth shape and head movement amplitude. The lip opening and closing amplitude is subtracted from the audio energy value and the absolute value is taken to obtain the lip opening and closing difference value; the head movement difference value is obtained by subtracting the expected head movement value from the head posture change value and taking the absolute value; the lip opening and closing difference value and the head movement difference value are added together to obtain the comprehensive difference value; the ratio of the comprehensive difference value to twice the normal lip and head movement amplitude normalization factor is subtracted from 1 to obtain the lip and head movement audio synchronization value.

5. The intelligent processing method for multimodal data of spoken video based on deep learning according to claim 1, characterized in that, The specific process of taking coordinated and regulatory measures is as follows: When the audio synchronization value of mouth shape and head movement is detected to be lower than the synchronization threshold, the current time period is marked as an uncoordinated region; the video generation parameters of the voice-over animation generation model are dynamically adjusted; and the correspondence between the voice-over animation and the current audio energy value and head posture change value is enhanced. For areas of inconsistency, local regeneration is performed, and frame interpolation algorithms are used to smooth changes in lip movements and head movements. If the audio synchronization value of lip movements and head movements falls below the synchronization threshold more than three times, or if there are more than three areas of inconsistency, an abnormal log is generated, prompting manual review and intervention. Finally, the optimized lip movement and head movement audio animation sequence is output.

6. The intelligent processing method for multimodal data of spoken video based on deep learning according to claim 1, characterized in that, The specific process of determining the consistency between lip movements and emotional intensity in a spoken animation based on emotion category, combined with multimodal spoken data and a spoken animation generation model, is as follows: For each frame of the voice-over animation, obtain the mouth opening and closing amplitude, and calculate the difference between the current frame's lip opening and closing amplitude and the previous frame to obtain the mouth shape change amplitude. The emotion category of each frame is obtained. Based on a publicly available emotional speech database, an emotion category feature set is constructed. The emotion category feature set is used as input, and a multi-layer deep network emotion recognition model is trained. The last layer of the emotion recognition model uses a normalized exponential function to output the probability distribution of each emotion category to obtain the normalized confidence value of the emotion category. The normalized confidence value is used as the audio emotion intensity value. The difference between the audio emotion intensity value of the current frame and the previous frame is calculated to obtain the audio emotion intensity change value. Based on the sliding window of the current video, calculate the absolute value of the difference between the lip shape change amplitude and the audio emotional intensity change value in each frame, and then take the average value to obtain the lip shape emotional change normalization factor. The lip-shape emotion synchronization value is obtained by subtracting the audio emotion intensity change value from the lip-shape change amplitude and taking the absolute value; the lip-shape emotion consistency assessment value is obtained by subtracting the ratio of the lip-shape emotion synchronization value to the lip-shape emotion change normalization factor from 1.

7. The intelligent processing method for multimodal data of spoken video based on deep learning according to claim 1, characterized in that, The specific process for implementing abnormal repair measures based on the consistency between lip shape and emotional intensity is as follows: Calculate the lip-sync consistency evaluation value for each frame of the narration animation, identify segments with lip-sync consistency evaluation values ​​below the consistency threshold, mark them as abnormal segments, and adjust the video generation parameters of the narration animation generation model for abnormal segments; By using a multimodal audio database to trace back similar audio and facial expression segments in history, multiple sets of candidate audio animations are generated simultaneously. A discriminant network and scoring mechanism are used to select the audio animation that best matches the lip movements with the audio emotions. By retrieving and referring to the lip movement patterns, sample transfer repair is performed on the current abnormal segments. All abnormal segments and timestamps are summarized and output.

8. The intelligent processing method for multimodal data of spoken video based on deep learning according to claim 1, characterized in that, The specific process of constructing a deep temporal prediction model based on multimodal voice-over data and emotion categories, and then evaluating the difference between the actual voice-over animation and the prediction results of the deep temporal prediction model by combining the multimodal voice-over data and the deep temporal prediction model, is as follows: Based on the voice-over animation sequence output by the voice-over animation generation model, a multimodal voice-over feature set is constructed for each frame corresponding to lip key points and emotion category. Based on the multimodal voice-over feature set, static facial expression features are extracted using an open-source facial expression analysis algorithm, and the static facial expression features, lip key points, audio features, and emotion category multimodal features are concatenated in a temporal sequence. The spliced ​​multimodal features are input into a temporal deep learning algorithm for training to build a deep temporal prediction model, which outputs the predicted values ​​of mouth opening and closing amplitude, mouth movement, and head posture change at each time step. Obtain the actual mouth opening amplitude and the predicted mouth opening amplitude for each frame of the voice-over animation. Subtract the predicted mouth opening amplitude from the actual mouth opening amplitude and take the absolute value to obtain the mouth opening difference. At the same time, obtain the actual head posture change value and the predicted head posture change value for each frame of the voice-over animation. Subtract the predicted head posture change value from the actual head posture change value and take the absolute value to obtain the head movement difference. The algorithm acquires the trajectory of key points of the lips, identifies lip-related movements, and outputs the activation score of facial action units corresponding to lip-related movements in each frame as the actual mouth movement evaluation value through the facial expression recognition algorithm of open-source facial expression analysis. It also obtains the mouth movement prediction value, subtracts the mouth movement prediction value from the actual mouth movement evaluation value and takes the absolute value to obtain the mouth movement difference. The multimodal comprehensive anomaly assessment value is obtained by adding the differences in mouth opening and closing, head movement, and mouth movement, and then averaging the results.

9. The intelligent processing method for multimodal data of spoken video based on deep learning according to claim 1, characterized in that, The specific process of video optimization and model tuning is as follows: Frames with multimodal comprehensive anomaly evaluation values ​​exceeding the anomaly threshold are labeled, and an interactive review interface is generated. The audio synchronization value of lip-sync and head-movement, the lip-sync emotion consistency evaluation value, and the multimodal comprehensive anomaly evaluation value are cross-validated to identify the anomaly type and cause, and to correct the coordination of lip-sync, head-movement, and facial expression in the voice-over animation. All review and correction operations are archived in real time, and the voice-over animation generation model and the deep temporal prediction model are trained and optimized through online learning and incremental fine-tuning methods. Output optimized and reviewed high-fidelity videos, unifying lip movements, head movements, facial expressions, and emotions, and generating detailed and interpretable reports. Record multimodal comprehensive anomaly assessment values ​​and correction frames to achieve closed-loop optimization.

10. A deep learning-based intelligent processing system for multimodal data of voice-over video, characterized in that: include: The data acquisition and preprocessing module is used to acquire multimodal spoken data in real time, identify emotion categories, and perform data preprocessing on the multimodal spoken data. The joint modeling and synchronization judgment module is used to build a voice-over animation generation model based on multimodal voice-over data, and combine the multimodal voice-over data and the voice-over animation generation model to judge the degree of synchronization between mouth shape, head movement and audio in the voice-over animation, so as to take coordination and control measures. The consistency detection and adaptive repair module is used to determine the consistency between lip movements and emotional intensity in spoken animations based on emotion categories, combined with multimodal spoken data and spoken animation generation models, and to implement anomaly repair measures based on the consistency between lip movements and emotional intensity. The intelligent review and high-fidelity output module is used to build a deep temporal prediction model based on multimodal voice-over data and emotion categories. By combining multimodal voice-over data and the deep temporal prediction model, it evaluates the degree of difference between the actual voice-over animation and the prediction results of the deep temporal prediction model, thereby optimizing the video and tuning the model.

Images