Method and system for processing video images in video calls

Through the combination of DenseNet algorithm and voice and face information features, automatic face-simulation video switching and recovery when users leave the camera during video calls is achieved, solving the continuity problem when users leave during video calls and improving user experience The natural fluency of video calls.

CN118381870BActive Publication Date: 2024-11-22GUANGZHOU MIMAI CULTURE MEDIA CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202410504792.X
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-04-25
Publication Date
2024-11-22
Estimated Expiration
2044-04-25

AI Technical Summary

Technical Problem

During a video call, when a user needs to leave the camera shooting area to handle other things, the prior art cannot achieve this requirement without the other party's awareness, resulting in the impact of the continuity and user experience of the video call.

Method used

DenseNet algorithm monitors whether the user's face leaves the camera area, and switches to simulate face video calls when it is detected, using voice and face information features to generate virtual face videos until the user returns to the camera range and restores the original Video call flow.

Benefits of technology

It automatically switches to simulate face video calls when the user leaves the camera, avoids blank screens or interrupts, improves the user experience and continuity of video calls, and ensures the natural smoothness and interaction of video calls.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118381870B_ABST
    Figure CN118381870B_ABST
Patent Text Reader

Abstract

The present invention relates to the field of video processing technology, and in particular to a method and system for processing video images in a video call, comprising: monitoring whether the user in a real-time video call leaves the camera shooting area through a DenseNet algorithm, and executing the next step if the user leaves. Collecting user video data, extracting voice information and facial information features, the facial information includes facial expressions and lip movements, and establishing a voice matching face video algorithm. Inputting new voice information features, generating a virtual face video, and merging and replacing it with the original video. Using a facial key point detection algorithm to monitor in real time whether the user's face has returned to its original position, and restoring the original video call process if it has returned to its original position. The present invention realizes automatic switching to a virtual face video call when the user leaves the camera, and automatically restoring the original video call process when the user returns to the camera range, thereby improving the user experience and continuity of the video call.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of video processing technology, and in particular to a method and system for processing video images in a video call. Background Art

[0002] During a video call, existing video calls use cameras to capture the user's video images in real time and provide the video images to the other party for display. However, the user may sometimes need to leave the camera shooting area to handle other things, but do not want the other party of the video call to find out. The existing video processing method cannot meet the user's need to leave the camera shooting area without the other party noticing. Summary of the invention

[0003] To solve the above problems, the present invention provides a method and system for processing video images in a video call, which can automatically switch to a simulated face video call when the user leaves the camera, and restore the original video call process when the user returns to the camera range, thereby improving the user experience and continuity of the video call.

[0004] The purpose of the present invention can be achieved through the following technical solutions:

[0005] A method for processing a video image in a video call comprises the following steps:

[0006] Step 1: Face departure detection, including:

[0007] In a real-time video call, the DenseNet algorithm is used to monitor whether the user's face leaves the camera shooting area. If the face leaves, step 2 is executed.

[0008] Step 2: Simulate a face video call, including:

[0009] Get user video: collect user video data in the current video call environment;

[0010] Feature extraction: extract voice information features and face information features from user videos. Face information features include facial expression features and lip movement features.

[0011] Establish a model: Use voice information features as input and face information features as output to establish a voice matching face video algorithm;

[0012] Video generation: acquiring new voice information, extracting new voice information features, inputting the new voice information features into the voice matching face video algorithm, and generating a virtual face video that matches the new voice;

[0013] Video fusion update: The generated virtual face video is merged with the original user video and replaced with the video screen corresponding to the user in the current video call;

[0014] Face resetting detection: Use the face key point detection algorithm to monitor in real time whether the user's face has returned to the camera shooting area. If it is detected that the user's face has returned, execute step 3;

[0015] Step 3: Return to the original video call process, including:

[0016] Restore the current video call to the original video call process and continue the real video call communication.

[0017] Furthermore, the construction steps of the DenseNet algorithm include:

[0018] Data collection: Collect video data sets containing faces leaving the camera shooting area and face image data sets at different angles and lighting conditions as training data;

[0019] Data preprocessing: preprocess the collected video data and face image data, including face detection, cropping and resizing operations;

[0020] Data partitioning: Divide the preprocessed data into training data set and test data set;

[0021] Build a DenseNet model: Build a DenseNet model for face detection;

[0022] Training model: Input the training data set into the DenseNet model for training. Through supervised learning, the model learns the characteristics of whether the face leaves the camera shooting area.

[0023] Model evaluation: Use the test data set to evaluate the performance of the trained model, including accuracy, recall, and F1 value indicators, and perform model tuning based on the evaluation results;

[0024] Deploy the model: Deploy the trained DenseNet model to the real-time video call system;

[0025] Continuous optimization: Continuously collect user feedback data to update and optimize the model.

[0026] Furthermore, the extraction of voice information features from user videos is achieved through a preset voiceprint recognition and voice emotion analysis composite model.

[0027] Furthermore, the method for constructing the composite model of voiceprint recognition and speech emotion analysis includes:

[0028] Audio collection: Collect the user videos that the user has authorized in the video call system;

[0029] Extract audio: extract the audio part from the user's video file through FFmpeg;

[0030] Sound segmentation: Use the VAD algorithm to segment the extracted audio to obtain speech fragments;

[0031] Preprocessing: Each speech segment is preprocessed, including noise removal, noise reduction, and audio enhancement;

[0032] Feature extraction: Use voiceprint recognition algorithm and speech emotion analysis algorithm to extract audio features from each speech segment, including the sound spectrum features, voiceprint features, emotion features and intonation features;

[0033] Feature representation: The extracted audio features are represented as feature matrices;

[0034] Model training: Use the SVM algorithm to build a composite model of voiceprint recognition and speech sentiment analysis, and learn the relationship between features and corresponding labels through training data sets;

[0035] Model testing and evaluation: Use the test data set to evaluate the performance of the trained model, including accuracy, recall, confusion matrix, and ROC curve indicators;

[0036] Application deployment: Deploy the trained voiceprint recognition and voice sentiment analysis models to the real-time video call system.

[0037] Further, extracting facial information features from the user video includes the following steps:

[0038] Face detection and positioning: Through the Haar cascade detector, face detection is performed in each frame of the video to locate the position of the face;

[0039] Extract facial expression features;

[0040] Extract lip movement features;

[0041] Feature fusion: The extracted facial expression features and lip movement features are fused to form an overall facial information feature vector.

[0042] Further, the extraction of facial expression features includes:

[0043] Feature point calibration: Use the Dlib algorithm to detect the face area; in the detected face area, use the Dlib algorithm to calibrate the key point positions, including eyes, mouth and eyebrows;

[0044] Construction of expression recognition model: Design an expression recognition model based on a recurrent neural network; input a facial image sequence characterized by key point positions, and learn different facial expression features through a recurrent neural network;

[0045] Feature extraction: Use the constructed expression recognition model to predict the face image and obtain the probability distribution of expression categories; extract facial expression features from the probability distribution of expression categories and convert the facial expression features into vector form.

[0046] Further, the extracting of lip movement features includes:

[0047] Mouth area extraction: Use the MTCNN algorithm to locate the mouth area;

[0048] Mouth action recognition: A mouth action recognition model is built based on the GRU model to learn and understand mouth movements, thereby inferring the pronunciation and opening and closing degree of the mouth;

[0049] Feature extraction: Use the trained mouth movement recognition model to extract mouth movement features and convert the mouth movement features into vector form.

[0050] Furthermore, the method for constructing the voice matching face video algorithm includes:

[0051] Prepare voice information and corresponding face information datasets;

[0052] Extract the voice information features and face information features of the prepared data set;

[0053] Select the CNN model, use voice information features as input and face information features as output, and build the model architecture of the voice matching face video algorithm;

[0054] Use the prepared data set to train the constructed model architecture, and continuously adjust the model parameters through the back propagation algorithm so that the model can match the corresponding face information from the voice information;

[0055] Evaluate the trained model, use the validation set or test set to evaluate the performance of the model, and adjust and optimize the model based on the evaluation results;

[0056] Deploy the trained voice matching face video algorithm to the real-time video call system.

[0057] Furthermore, the method of monitoring in real time whether the user's face returns to the camera shooting area by using a facial key point detection algorithm includes:

[0058] Capture the user's face image in real time through the camera, and input the captured face image into the pre-set Hourglass network for face position detection;

[0059] According to the results output by the Hourglass network, it is determined whether the user's face has returned to the camera shooting area based on the coordinate information of the face position.

[0060] A system for processing video images in a video call is provided, wherein the system for processing video images in a video call is used to execute the method for processing video images in a video call as described above.

[0061] The beneficial effects of the present invention are:

[0062] The present invention realizes automatic switching to a simulated face video call when the user leaves the camera, and restores the original video call process when the user returns to the camera range, thereby improving the user experience and continuity of the video call. Specifically, by automatically switching to a simulated face video call, when the user leaves the camera range, the continuity of the video call can be maintained, and a blank screen or interruption can be avoided, thereby improving the user experience. The method can automatically switch to a simulated face video call after the user leaves the camera range, and restore the original video call process when the user returns to the camera range, so that the switching of the video call is more natural and smooth, the continuity of the call process is maintained, and the user feels abrupt or uncomfortable. The DenseNet algorithm and the Hourglass network are used to perform face departure detection and face reset detection respectively, so as to realize real-time monitoring of the user status, and perform corresponding video processing according to the monitoring results, so as to ensure the timeliness and accuracy of the video call processing. By obtaining user video data, extracting voice and face information features, establishing a voice matching face video algorithm, and generating a virtual face video, smooth switching and replacement can be realized when the user leaves the camera, and the interactive sense and visual effect of the video call are enhanced. In general, the present invention combines face detection technology, feature extraction technology and virtual face generation technology, and realizes automatic processing and optimization of the video call process through intelligent algorithms, bringing users a smoother and more natural video call experience, and improving the comfort and convenience of users participating in video calls.

[0063] The present invention combines the two algorithms of voiceprint recognition and speech emotion analysis, extracts multiple audio features at the same time, and uses the SVM algorithm to establish a composite model. Through a large amount of data training and testing, it realizes accurate recognition and emotion analysis of user voice information.

[0064] The present invention combines multiple technologies such as face detection, feature extraction and fusion, making the extraction of face information features from user videos more comprehensive and accurate, which is helpful for further in-depth analysis and application of face-related information.

[0065] The present invention combines technologies such as speech recognition, face recognition and deep learning, integrates knowledge and technologies from multiple fields, and can effectively realize the construction and application of speech matching face video algorithm, providing more intelligent and convenient services for real-time video call systems. BRIEF DESCRIPTION OF THE DRAWINGS

[0066] In order to facilitate understanding by those skilled in the art, the present invention is further described below with reference to the accompanying drawings.

[0067] Figure 1 The present invention is a flowchart of a method for processing video images in a video call. DETAILED DESCRIPTION

[0068] In order to further explain the technical means and effects adopted by the present invention to achieve the predetermined invention purpose, the specific implementation methods, structures, features and effects of the present invention are described in detail below in conjunction with the accompanying drawings and preferred embodiments.

[0069] See also Figure 1 As shown, the present invention provides a method and system for processing video images in a video call, which can automatically switch to a simulated face video call when the user leaves the camera, and restore the original video call process when the user returns to the camera range, thereby improving the user experience and continuity of the video call.

[0070] The method for processing video images in a video call includes the following steps:

[0071] Step 1: Face departure detection, including:

[0072] In a real-time video call, the DenseNet algorithm is used to monitor whether the user's face leaves the camera shooting area. If the face leaves, step 2 is executed.

[0073] Step 2: Simulate a face video call, including:

[0074] Get user video: collect user video data in the current video call environment;

[0075] Feature extraction: extract voice information features and face information features from user videos. Face information features include facial expression features and lip movement features.

[0076] Establish a model: Use voice information features as input and face information features as output to establish a voice matching face video algorithm;

[0077] Video generation: acquiring new voice information, extracting new voice information features, inputting the new voice information features into the voice matching face video algorithm, and generating a virtual face video that matches the new voice;

[0078] Video fusion update: The generated virtual face video is merged with the original user video and replaced with the video screen corresponding to the user in the current video call;

[0079] Face resetting detection: Use the face key point detection algorithm to monitor in real time whether the user's face has returned to the camera shooting area. If it is detected that the user's face has returned, execute step 3;

[0080] Step 3: Return to the original video call process, including:

[0081] Restore the current video call to the original video call process and continue the real video call communication.

[0082] In the above scheme, the method for processing video images in a video call has the following beneficial effects:

[0083] 1. Improve user experience: By automatically switching to simulated face video calls, when the user leaves the camera range, the continuity of the video call can be maintained, avoiding blank screens or interruptions, thereby improving the user experience.

[0084] 2. Continuity: This method can automatically switch to a simulated face video call after the user leaves the camera range, and restore the original video call process when the user returns to the camera range, making the switching of video calls more natural and smooth, maintaining the continuity of the call process, and avoiding users feeling abrupt or uncomfortable.

[0085] 3. Real-time monitoring and processing: The DenseNet algorithm is used to detect face departure and face re-entry, realize real-time monitoring of user status, and perform corresponding video processing based on the monitoring results to ensure the timeliness and accuracy of video call processing.

[0086] 4. Virtual face generation technology: By acquiring user video data, extracting voice and face information features, establishing a voice matching face video algorithm, and generating virtual face videos, it can achieve smooth switching and replacement when the user leaves the camera, thereby enhancing the interactivity and visual effects of video calls.

[0087] Overall, this method of processing video images in video calls combines face detection technology, feature extraction technology and virtual face generation technology, and realizes automatic processing and optimization of the video call process through intelligent algorithms, bringing users a smoother and more natural video call experience, and improving the comfort and convenience of users participating in video calls.

[0088] Furthermore, the construction steps of the DenseNet algorithm include:

[0089] Data collection: Collect video data sets containing faces leaving the camera's shooting area and face image data sets under different angles and lighting conditions as training data; among them, it is necessary to ensure that the collected data sets contain sufficiently diverse video data of faces leaving the camera's shooting area and face image data under different angles and lighting conditions to improve the generalization ability of the model.

[0090] Data preprocessing: Preprocess the collected video data and face image data, including face detection, cropping, and resizing operations. When performing face detection, cropping, and resizing operations, it is necessary to ensure that the processed data can still retain key information and features to avoid negative impacts on model training due to information loss.

[0091] Data partitioning: Divide the preprocessed data into training data set and test data set;

[0092] Build a DenseNet model: Build a DenseNet model for face detection. DenseNet is a densely connected deep learning model. It is necessary to ensure that the network structure is well designed and the parameters are set appropriately to effectively learn the characteristics of the face leaving the camera.

[0093] Training model: Input the training data set into the DenseNet model for training. Through supervised learning, the model learns the characteristics of whether the face leaves the camera shooting area. When training the model, you can choose the appropriate loss function and optimizer according to actual needs, and adjust the hyperparameters in time according to the changes in indicators during the training process to improve the convergence speed and performance of the model.

[0094] Model evaluation: Use the test data set to evaluate the performance of the trained model, including accuracy, recall, and F1 value indicators, and perform model tuning based on the evaluation results;

[0095] Deploy the model: Deploy the trained DenseNet model to the real-time video call system;

[0096] Continuous optimization: Continuously collect user feedback data to update and optimize the model.

[0097] Furthermore, the extraction of voice information features from the user video is achieved through a preset voiceprint recognition and voice emotion analysis composite model. The construction process of the voiceprint recognition and voice emotion analysis composite model includes:

[0098] 1. Audio collection: Collect audio data from videos authorized by users in the video call system as the data source for subsequent processing.

[0099] 2. Extract audio: Use the FFmpeg tool to extract the audio part from the user's video file, preparing for subsequent sound segmentation and feature extraction.

[0100] 3. Sound segmentation: Use the VAD (Voice Activity Detection) algorithm to segment the extracted audio and divide it into voice segments for subsequent processing.

[0101] 4. Preprocessing: Each speech segment is preprocessed, including noise removal, noise reduction and audio enhancement, to improve the accuracy and stability of subsequent feature extraction.

[0102] 5. Feature extraction: Use voiceprint recognition algorithm and speech emotion analysis algorithm to extract audio features from each speech segment, such as sound spectrum features, voiceprint features, emotion features and intonation features.

[0103] 6. Feature representation: The extracted audio features are represented in the form of a feature matrix for subsequent model training.

[0104] 7. Model training: Use the SVM (Support Vector Machine) algorithm to establish a composite model of voiceprint recognition and speech sentiment analysis. Use the training data set to learn the relationship between features and corresponding labels to achieve recognition of voice information and sentiment analysis.

[0105] 8. Model testing and evaluation: Use the test data set to evaluate the performance of the trained model, including indicators such as accuracy, recall, confusion matrix, and ROC curve, to verify the effectiveness and reliability of the model.

[0106] 9. Application deployment: Deploy the trained voiceprint recognition and voice emotion analysis models to the real-time video call system to achieve real-time recognition and emotion analysis of user voice information, thereby improving the user experience and service quality of the system.

[0107] The advantage of this model building process is that it combines the two algorithms of voiceprint recognition and speech sentiment analysis, extracts multiple audio features at the same time, and uses the SVM algorithm to establish a composite model. Through a large amount of data training and testing, it achieves accurate recognition and sentiment analysis of user voice information.

[0108] Furthermore, extracting facial information features from user videos involves multiple key steps, which are specifically analyzed as follows:

[0109] Step M1: Face detection and positioning

[0110] Face detection and positioning is performed through the Haar cascade detector, which is a method that can efficiently recognize faces and accurately locate the face position in the video frame, laying the foundation for subsequent feature extraction.

[0111] Step M2: Extract facial expression features

[0112] 1. Feature point calibration: Use the Dlib algorithm to calibrate the positions of key points, including eyes, mouth, and eyebrows. Such calibration can provide detailed information of different areas of the face.

[0113] 2. Expression recognition model construction: The expression recognition model is designed based on the recurrent neural network, which can capture the timing information between different facial expressions and learn the facial expression characteristics more comprehensively.

[0114] 3. Feature extraction: Use the constructed expression recognition model to predict facial images, extract the probability distribution of expression categories, and convert them into vector form. The features obtained in this way can better represent the facial expression information.

[0115] Step M3: Extract lip movement features

[0116] 1. Mouth area extraction: Using the MTCNN algorithm to locate the mouth area helps to accurately locate the mouth area and extract lip shape information.

[0117] 2. Mouth movement recognition model construction: Based on the GRU model, learn and understand mouth movements, infer mouth pronunciation and opening and closing degree information. The GRU model is suitable for processing time series information and is conducive to capturing the changing patterns of mouth movements.

[0118] 3. Feature extraction: Use the trained mouth movement recognition model to extract lip movement features and convert them into vector form. The features obtained in this way can effectively represent the lip movement information.

[0119] Step M4: Feature fusion

[0120] The extracted facial expression features and lip movement features are fused to form an overall face information feature vector. By fusing different features, the facial expression and lip shape information can be comprehensively reflected, providing a more comprehensive feature description for subsequent face recognition, sentiment analysis and other tasks.

[0121] In general, this embodiment combines multiple technologies such as face detection, feature extraction and fusion, making the extraction of facial information features from user videos more comprehensive and accurate, which helps to further analyze and apply face-related information.

[0122] Furthermore, the method for constructing the voice matching face video algorithm includes:

[0123] 1. Prepare voice information and corresponding face information datasets: Collect a large amount of data with voice and face information to ensure the diversity and adequacy of the dataset.

[0124] 2. Extract the speech information features and face information features of the prepared data set: Use feature extraction technology, such as MFCC (Mel Frequency Cepstral Coefficient) to extract key features in speech information, and use face recognition technology to extract features in face information.

[0125] 3. Select the CNN model, use the voice information features as input and the face information features as output, and build the model architecture of the voice matching face video algorithm: The CNN model has a good effect in the field of image processing. By inputting the features of voice and face information into the CNN model, the matching between voice and face information can be achieved.

[0126] 4. Use the prepared data set to train the constructed model architecture, and continuously adjust the model parameters through the back propagation algorithm so that the model can match the corresponding face information from the voice information: through repeated training of a large amount of data, optimize the parameters of the CNN model and improve the matching accuracy of the model.

[0127] 5. Evaluate the trained model and use the validation set or test set to evaluate the performance of the model. Adjust and optimize the model based on the evaluation results: By evaluating the performance of the model on unknown data, verify the generalization ability of the algorithm, and adjust and optimize the model based on the evaluation results to further improve the matching accuracy.

[0128] 6. Deploy the trained voice matching face video algorithm to the real-time video call system: Deploy the trained and optimized algorithm to the actual application scenario to achieve real-time matching of voice and face information in the real-time video call system, improving user experience and data processing efficiency.

[0129] The above construction method combines technologies such as speech recognition, face recognition and deep learning, integrates knowledge and technologies from multiple fields, and can effectively realize the construction and application of speech matching face video algorithm, providing more intelligent and convenient services for real-time video call systems.

[0130] Furthermore, the real-time monitoring of whether the user's face returns to the camera shooting area by using a facial key point detection algorithm includes:

[0131] Capture the user's face image in real time through the camera, and input the captured face image into the pre-set Hourglass network for face position detection;

[0132] According to the results output by the Hourglass network, it is determined whether the user's face has returned to the camera shooting area based on the coordinate information of the face position.

[0133] In this embodiment, a camera is used to collect the user's facial image in real time to provide a data source for subsequent analysis. The captured facial image is input into the pre-set Hourglass network for processing. The location information of the face can be obtained through the Hourglass network. According to the result output by the Hourglass network, that is, the coordinate information of the face position, it is determined whether the user's face has returned to the camera shooting area. This step is achieved by analyzing and comparing the location information. Based on the real-time captured facial image, this embodiment can realize real-time monitoring of the user's facial position in combination with the fast processing capability of the Hourglass network. Using the facial key point detection algorithm, the user's facial position can be accurately located, so as to determine whether it has returned to the camera shooting area, thereby improving the accuracy of monitoring. In general, by combining the facial key point detection algorithm with the Hourglass network for facial position detection, it is possible to effectively monitor whether the user has returned to the camera shooting area.

[0134] A system for processing video images in a video call is provided, wherein the system for processing video images in a video call is used to execute the method for processing video images in a video call as described above.

[0135] The above description is only a preferred embodiment of the present invention and does not limit the present invention in any form. Although the present invention has been disclosed as a preferred embodiment as above, it is not used to limit the present invention. Any technical personnel in this field can make some changes or modify the technical contents disclosed above into equivalent embodiments without departing from the scope of the technical solution of the present invention. However, any brief modifications, equivalent changes and modifications made to the above embodiments based on the technical essence of the present invention without departing from the content of the technical solution of the present invention are still within the scope of the technical solution of the present invention.

Claims

1. A method for processing a video image in a video call, characterized in that: The following steps are involved: Step 1: During a real-time video call, the DenseNet algorithm is used to monitor whether the user's face leaves the camera shooting area. If the face leaves, step 2 is executed. Step 2: Simulate a face video call, including: Get user video: collect user video data in the current video call environment; Feature extraction: extract voice information features and face information features from user videos. Face information features include facial expression features and lip movement features. Establish a model: Use voice information features as input and face information features as output to establish a voice matching face video algorithm; Video generation: acquiring new voice information, extracting new voice information features, inputting the new voice information features into the voice matching face video algorithm, and generating a virtual face video that matches the new voice; Video fusion update: The generated virtual face video is merged with the original user video and replaced with the video screen corresponding to the user in the current video call; Face resetting detection: Use the face key point detection algorithm to monitor in real time whether the user's face has returned to the camera shooting area. If it is detected that the user's face has returned, execute step 3; Step 3: Restore the current video call to the original video call process and continue the real video call communication; in, The construction steps of the DenseNet algorithm include: Data collection: Collect video data sets containing faces leaving the camera shooting area and face image data sets at different angles and lighting conditions as training data; Data preprocessing: preprocess the collected video data and face image data, including face detection, cropping and resizing operations; Data partitioning: Divide the preprocessed data into training data set and test data set; Build a DenseNet model: Build a DenseNet model for face detection; Training model: Input the training data set into the DenseNet model for training. Through supervised learning, the model learns the characteristics of whether the face leaves the camera shooting area. Model evaluation: Use the test data set to evaluate the performance of the trained model, including accuracy, recall, and F1 value indicators, and perform model tuning based on the evaluation results; Deploy the model: Deploy the trained DenseNet model to the real-time video call system; Continuous optimization: Continuously collect user feedback data to update and optimize the model; The extraction of voice information features from user videos is achieved through a preset voiceprint recognition and voice emotion analysis composite model; the method for constructing the voiceprint recognition and voice emotion analysis composite model includes: Audio collection: Collect the user videos that the user has authorized in the video call system; Extract audio: extract the audio part from the user's video file through FFmpeg; Sound segmentation: Use the VAD algorithm to segment the extracted audio to obtain speech fragments; Preprocessing: Each speech segment is preprocessed, including noise removal, noise reduction, and audio enhancement; Feature extraction: Use voiceprint recognition algorithm and speech emotion analysis algorithm to extract audio features from each speech segment, including the sound spectrum features, voiceprint features, emotion features and intonation features; Feature representation: The extracted audio features are represented as feature matrices; Model training: Use the SVM algorithm to build a composite model of voiceprint recognition and speech sentiment analysis, and learn the relationship between features and corresponding labels through training data sets; Model testing and evaluation: Use the test data set to evaluate the performance of the trained model, including accuracy, recall, confusion matrix, and ROC curve indicators; Application deployment: Deploy the trained voiceprint recognition and voice emotion analysis models to the real-time video call system; The method for constructing the voice matching face video algorithm includes: Prepare voice information and corresponding face information datasets; Extract the voice information features and face information features of the prepared data set; Select the CNN model, use voice information features as input and face information features as output, and build the model architecture of the voice matching face video algorithm; Use the prepared data set to train the constructed model architecture, and continuously adjust the model parameters through the back propagation algorithm so that the model can match the corresponding face information from the voice information; Evaluate the trained model, use the validation set or test set to evaluate the performance of the model, and adjust and optimize the model based on the evaluation results; Deploy the trained voice-matching face video algorithm to the real-time video call system; The method of monitoring in real time whether the user's face returns to the camera shooting area by using a face key point detection algorithm includes: Capture the user's face image in real time through the camera, and input the captured face image into the pre-set Hourglass network for face position detection; According to the results output by the Hourglass network, it is determined whether the user's face has returned to the camera shooting area based on the coordinate information of the face position.

2. The method for processing video images in a video call according to claim 1, characterized in that: Extracting facial information features from user videos includes the following steps: Face detection and positioning: Through the Haar cascade detector, face detection is performed in each frame of the video to locate the position of the face; Extract facial expression features; Extract lip movement features; Feature fusion: The extracted facial expression features and lip movement features are fused to form an overall facial information feature vector.

3. The method for processing video images in a video call according to claim 2, characterized in that: The extracting of facial expression features comprises: Feature point calibration: Use the Dlib algorithm to detect the face area; in the detected face area, use the Dlib algorithm to calibrate the key point positions, including eyes, mouth and eyebrows; Construction of expression recognition model: Design an expression recognition model based on a recurrent neural network; input a facial image sequence characterized by key point positions, and learn different facial expression features through a recurrent neural network; Feature extraction: Use the constructed expression recognition model to predict the face image and obtain the probability distribution of expression categories; extract facial expression features from the probability distribution of expression categories and convert the facial expression features into vector form.

4. The method for processing video images in a video call according to claim 2, characterized in that: The extracting of lip movement features comprises: Mouth area extraction: Use the MTCNN algorithm to locate the mouth area; Mouth action recognition: A mouth action recognition model is built based on the GRU model to learn and understand mouth movements, thereby inferring the pronunciation and opening and closing degree of the mouth; Feature extraction: Use the trained mouth movement recognition model to extract mouth movement features and convert the mouth movement features into vector form.

5. A system for processing video images in a video call, characterized in that: The system for processing video images in a video call is used to execute the method for processing video images in a video call as described in any one of claims 1-4.