Singing mouth shape real-time detection method based on spatiotemporal depth feature adaptive decision

Abstract

This invention proposes a real-time lip-sync detection method based on spatiotemporal depth features and adaptive decision-making, comprising the following steps: Step S1: Acquire facial video images of the singer and annotate the images according to the actual lip movements, constructing a sample dataset for model training and testing; Step S2: Construct a YOLOv8-based real-time lip-sync target detection network to segment the lip-sync images from the facial video; Step S3: Construct a lip-sync spatial depth feature extraction model based on capsule networks, inputting a series of extracted spatial depth features into a BiLSTM network, and further extracting the temporal series features of the lip-sync images through the BiLSTM network; Step S4: Optimize the hyperparameters of the random forest algorithm using the Love Evolution algorithm, mapping the extracted temporal series features to the high-dimensional space of the random forest for adaptive decision-making and classification; This invention can detect the changes in the singer's lip movements during performance in real time, and is more objective and efficient than manual methods.

Description

Technical Field

[0001] This invention relates to the field of facial recognition technology, and in particular to a real-time lip-sync detection method for singing based on spatiotemporal depth feature adaptive decision-making. Background Technology

[0002] Singing, as an art form, relies heavily on vocal techniques and performance style for a singer's expressiveness and musical delivery. During singing, changes in lip shape directly affect resonance, timbre, and pitch. Therefore, accurate detection and evaluation of vocal lip shape helps singers better master vocal techniques and improve their performance level. Furthermore, in vocal instruction, teachers often need to observe and guide students' vocal lip movements. Vocal lip shape detection techniques can provide teachers with objective and accurate assessment data, helping them better guide students in vocal exercises. Therefore, vocal lip shape detection has significant application value and practical significance in music research and teaching.

[0003] Currently, existing methods for detecting vocalization lip movements mainly fall into three categories: manual observation, traditional image feature extraction and recognition, and instrument-assisted detection.

[0004] The manual observation method is the most traditional and direct method, which requires professional vocal teachers or judges to observe and evaluate the singer's mouth shape. Although this method can intuitively capture subtle changes, it is highly subjective, easily affected by the observer's experience and fatigue, and cannot provide quantitative data.

[0005] Traditional image feature extraction and recognition methods extract lip-shape features, such as lip shape and degree of opening, from videos or images of singers. These features are then identified and analyzed using traditional image processing methods to determine whether the singer's lip movements are correct. While this method can provide relatively objective and quantitative evaluation results and reduce the influence of human factors, it has high requirements for image quality and lighting conditions, and the accuracy and robustness of the algorithm still need further improvement.

[0006] The third method involves instrument-assisted detection, typically using specialized instruments or equipment to detect the singer's lip movements, such as optical sensors or pressure sensors, to measure lip movement and force. While these instruments can provide accurate measurement data, helping to more accurately assess the singer's lip movements, they are expensive, can be relatively complex to operate, and may have limitations in real-time detection. Summary of the Invention

[0007] This invention proposes a real-time lip-shape detection method based on spatiotemporal depth feature adaptive decision-making. It can detect changes in a singer's lip shape in real time during the performance. Compared with manual methods, it has a more objective standard and higher efficiency. It can be used as an auxiliary tool for lip-shape detection during singing practice, and can help judge whether a singer's lip shape is standard in singing teaching, thereby improving the singer's singing level and stage performance.

[0008] The present invention adopts the following technical solution.

[0009] A real-time singing lip-syncing detection method based on spatiotemporal depth feature adaptive decision-making includes the following steps;

[0010] Step S1: Collect facial video images of the singers and annotate the images according to the actual lip movements to construct a sample dataset for model training and testing.

[0011] Step S2: Construct a real-time target detection network for singing lip movements based on YOLOv8 to segment lip shape images for singing from face videos;

[0012] Step S3: Construct a lip shape spatial depth feature extraction model based on capsule network, input a series of extracted spatial depth features into BiLSTM network, and further extract time series features of lip shape image through BiLSTM network;

[0013] Step S4: Optimize the hyperparameters of the random forest algorithm using the love evolution algorithm, and map the extracted time series features to the high-dimensional space of the random forest for adaptive decision classification.

[0014] Step S5: Train the model using the training and validation sets in the sample dataset, and test and analyze the model using the test set in the sample dataset.

[0015] The specific method of step S1 is as follows: select singers of different ages and genders as the collection subjects, with no less than 300 people. In order to ensure the robustness of the model, use the same shooting equipment to collect videos in the morning, noon and evening. Extract the collected videos according to the continuous time series. Collect the videos according to the mouth shape classification: vertical mouth, round mouth, pointed mouth, open mouth, closed mouth and smiling mouth. Extract no less than 6,000 continuous time series for each mouth shape. Divide these images into training set, validation set and test set according to the ratio of 8:1:1.

[0016] In step S2, lip-shape detection is performed on the continuous time series image of the singer's face. In step S2, a real-time lip-shape target detection network based on YOLOv8 is first constructed, and then the face image of the singer is detected and segmented to effectively detect the lip shape in real time. The lip-shape image of singing is segmented from the face video as the input of the model in subsequent steps.

[0017] The network structure of YOLOv8 is as follows: Figure 1 As shown, it consists of three parts: Backbone, Neck, and Head. The Backbone is responsible for extracting high-level semantic features from the input image. It contains multiple convolutional layers and pooling layers, which build a deep feature extractor that transforms the image into a feature representation with rich semantic information.

[0018] The Neck is used to further integrate and adjust the features extracted by the Backbone. By fusing features at different levels, it enhances the network's ability to perceive targets. The Neck structure consists of a series of convolutional layers and upsampling layers, which are used to fuse low-level feature maps with high-level feature maps to provide more global and local feature information, thereby helping to capture the details and contextual relationships of targets and improve the accuracy of target detection.

[0019] The Head is the last layer of the YOLOv8 model. In object detection tasks, it uses bounding box regressors and classifiers to output the bounding box location and category information of the object.

[0020] The specific method for calculating the training loss of the YOLOv8-based real-time lip-sync image detection and segmentation network model is as follows:

[0021] YOLOv8's loss function includes the bounding box loss (rect / box_loss) and the classification loss (cls_loss), and the total loss function is the sum of the two losses with their respective weight coefficients. The calculation formula is shown in equation (1).

[0022] Loss = a × L obj +b×L rect +c×L cls (1)

[0023] Where a, b, and c are the weighting coefficients for each type of loss, and the sum of the three equals 1;

[0024] The classification loss and confidence loss were calculated using BCEWithLogitsLoss, and the bounding box loss was calculated using the CIoU_Loss function + Distribution Focal Loss. The calculation method is shown in Equation (2).

[0025]

[0026] In the formula, b and b gt ρ represents the center point of the predicted bounding box and the center point of the ground truth bounding box. 2 () represents the Euclidean distance, and c is the length of the diagonal of the smallest rectangle containing the two boxes;

[0027] If A is the ground truth bounding box and B is the predicted bounding box, then the IoU calculation formula in the equation is shown in equation (3).

[0028]

[0029] DFL loss: DFL optimizes the probability of the two positions closest to label y (left and right) in the form of cross-entropy, so that the network can focus on the target position and the distribution of the surrounding area more quickly.

[0030] The category loss is calculated by using the category score of the predicted box and the eone-hot representation of the ground truth box, as shown in Equation (4).

[0031]

[0032] In step S3, capsule networks are used to enhance the focus on the positional feature information of the mouth shape;

[0033] Capsule networks consist of convolutional layers, initial capsules, and digital capsules, with the following structure: Figure 3 As shown. Convolutional structures are used for feature extraction in capsule networks. The convolutional layers in capsule networks extract local features from the image by performing convolution operations on the input data. During the convolution process, each convolutional kernel slides at different positions in the input data and shares the same weight parameters.

[0034] The initial capsule is the initial layer in the capsule network, responsible for extracting low-level local features from the input lip-shape image or video. These features include edges, corners, and textures, which are used for subsequent lip-shape recognition. The initial capsule captures these features through a set of learnable weight and bias parameters and encodes these features into the capsule's output vector. The capsule's output vector represents the presence or absence of features, as well as the spatial information and directionality of the features, which is used to capture the shape and dynamic changes of the lips in lip-shape detection.

[0035] Digital capsules reside deeper within the capsule network and are responsible for constructing higher-level abstract representations from the low-level features extracted from the initial capsules. These representations are used to capture more complex lip-shape patterns and structures. By combining and transforming information from the initial capsules, a more comprehensive understanding of lip shapes can be formed.

[0036] In step S3, the BiLSTM network uses deep learning methods to model and extract features from lip movements. By simultaneously considering the context of the sequence, it more accurately captures the dynamic changes and contextual relationships in the lip movement time series, extracting deep-level features from the lip movement time series. These deep-level features include static information about the lip shape and information about the dynamic changes in the lip shape. Specifically, the BiLSTM network receives a series of spatial depth features extracted by the capsule network as input and converts them into time series data. Then, the network processes the sequence through two Long Short-Term Memory (LSTM) layers in both forward and backward directions. The forward LSTM layer encodes from the beginning to the end of the sequence, capturing forward lip movement information; while the backward LSTM layer encodes from the end to the beginning of the sequence, capturing backward lip movement information. The outputs of these two LSTM layers are concatenated to fully utilize the contextual information in the sequence.

[0037] In step S4, the output of BiLSTM is used as the input variable for the random forest decision tree, and the love evolution algorithm is used to optimize the number of hyperparameter subsets T and the number of features m of the random forest. Finally, the classification results of each decision tree are voted on to achieve the function of automatically determining the lip shape category. The specific implementation process includes the following steps;

[0038] Step 1: Construct the dataset and divide the data, with one part used as training data and the remaining part as test data;

[0039] Step 2: Use random forest to build a model with optimized parameters based on the love evolution algorithm, and evaluate the model's performance on the training data as happiness; the evaluation process includes evaluating accuracy and loss function value;

[0040] Step 3: Use the bootstrap resampling method to sample T times with replacement from the updated training dataset to form T training subsets;

[0041] Step 4: Use the output of BiLSTM as the input variable for the random forest decision tree. The generation of the decision tree is a process of recursively building a binary classification tree. At each node of the tree, m features are randomly selected from M input variables as candidate features. Generally, m is the root mean square value of M. Then, the optimal feature and the best binary split point are selected from these m features to split the node, thereby building T decision trees and generating a random forest. Step 5: Use the love evolution algorithm to optimize the number of hyperparameter subsets T and the number of features m of the random forest, and generate multiple pairs of couples for love evaluation.

[0042] Step 6: For spouses with an acceptance level greater than 0.5, proceed with the separation process and enter the reflection phase to continue reflecting and changing personal characteristics until the conditions are met before proceeding to the next step;

[0043] Step 7: Continue with the value and role assessment stages for spouses with low acceptance levels. If an acceptance level greater than 0.5 occurs at any stage, proceed to Step 6 to update characteristics.

[0044] Step 8: Repeat steps 6 and 7 for multiple iterations until the termination condition is met (such as reaching the maximum number of iterations or meeting the preset accuracy requirements).

[0045] Step 9: Based on the lip-shape detection and classification results of each decision tree in the optimized random forest, vote to determine the final lip-shape category result.

[0046] The detection method is used in singing instruction to help judge whether a singer's lip movements are standard.

[0047] This invention employs a YOLOv8 network for lip-sync detection, avoiding manual lip segmentation and significantly improving model efficiency. It uses a capsule network to extract spatial depth features from the input lip shape, focusing on the positional features of the lip shape, which is more consistent with human cognitive processes. Furthermore, BI-LSTM effectively extracts time-series features, enhancing the real-time accuracy of lip-sync detection during singing. The Love Evolution algorithm is used to optimize the hyperparameters of the random forest to achieve optimal lip-sync detection results. This method can detect changes in a singer's lip shape in real time during performance, offering a more objective standard and higher efficiency compared to manual methods. It can serve as an auxiliary tool for lip-sync detection during singing practice, helping to judge the standardity of a singer's lip shape in singing instruction, thereby improving the singer's performance level and stage presence. Attached Figure Description

[0048] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments:

[0049] Appendix Figure 1 This is a schematic diagram of the YOLOv8 lip detection network structure of the present invention;

[0050] Appendix Figure 2 This is a schematic diagram of the network structure of the overall model of the present invention;

[0051] Appendix Figure 3 This is a schematic diagram of the capsule network structure of the present invention;

[0052] Appendix Figure 4 This is a schematic diagram of the overall process of the method described in this invention;

[0053] Appendix Figure 5This is a schematic diagram of the training loss of the real-time segmentation network model for singing lip-syncing images based on YOLOv8 in this invention.

[0054] Appendix Figure 6 This is a schematic diagram of the training average map of the real-time segmentation network model for singing lip-syncing images based on YOLOv8 in this invention.

[0055] Appendix Figure 7 This is a schematic diagram of the lip-shaped target detection structure in this invention. Detailed Implementation

[0056] As shown in the figure, the real-time singing lip-sync detection method based on spatiotemporal depth feature adaptive decision-making includes the following steps;

[0057] Step S1: Collect facial video images of the singers and annotate the images according to the actual lip movements to construct a sample dataset for model training and testing.

[0058] Step S2: Construct a real-time target detection network for singing lip movements based on YOLOv8 to segment lip shape images for singing from face videos;

[0059] Step S3: Construct a lip shape spatial depth feature extraction model based on capsule network, input a series of extracted spatial depth features into BiLSTM network, and further extract time series features of lip shape image through BiLSTM network;

[0060] Step S4: Optimize the hyperparameters of the random forest algorithm using the love evolution algorithm, and map the extracted time series features to the high-dimensional space of the random forest for adaptive decision classification.

[0061] Step S5: Train the model using the training and validation sets in the sample dataset, and test and analyze the model using the test set in the sample dataset.

[0062] The specific method of step S1 is as follows: select singers of different ages and genders as the collection subjects, with no less than 300 people. In order to ensure the robustness of the model, use the same shooting equipment to collect videos in the morning, noon and evening. Extract the collected videos according to the continuous time series. Collect the videos according to the mouth shape classification: vertical mouth, round mouth, pointed mouth, open mouth, closed mouth and smiling mouth. Extract no less than 6,000 continuous time series for each mouth shape. Divide these images into training set, validation set and test set according to the ratio of 8:1:1.

[0063] In step S2, lip-shape detection is performed on the continuous time series image of the singer's face. In step S2, a real-time lip-shape target detection network based on YOLOv8 is first constructed, and then the face image of the singer is detected and segmented to effectively detect the lip shape in real time. The lip-shape image of singing is segmented from the face video as the input of the model in subsequent steps.

[0064] The network structure of YOLOv8 is as follows: Figure 1 As shown, it consists of three parts: Backbone, Neck, and Head. The Backbone is responsible for extracting high-level semantic features from the input image. It contains multiple convolutional layers and pooling layers, which build a deep feature extractor that transforms the image into a feature representation with rich semantic information.

[0065] The Neck is used to further integrate and adjust the features extracted by the Backbone. By fusing features at different levels, it enhances the network's ability to perceive targets. The Neck structure consists of a series of convolutional layers and upsampling layers, which are used to fuse low-level feature maps with high-level feature maps to provide more global and local feature information, thereby helping to capture the details and contextual relationships of targets and improve the accuracy of target detection.

[0066] The Head is the last layer of the YOLOv8 model. In object detection tasks, it uses bounding box regressors and classifiers to output the bounding box location and category information of the object.

[0067] The specific method for calculating the training loss of the YOLOv8-based real-time lip-sync image detection and segmentation network model is as follows:

[0068] YOLOv8's loss functions include rectangular box loss (rect / box_loss) and classification loss (cls_loss), such as... Figure 5 As shown, the total loss function is the sum of the two losses with their respective weighting coefficients. The calculation formula is shown in equation (1).

[0069] Loss = a × L obj +b×L rect +c×L cls (1)

[0070] Where a, b, and c are the weighting coefficients for each type of loss, and the sum of the three equals 1;

[0071] The classification loss and confidence loss were calculated using BCEWithLogitsLoss, and the bounding box loss was calculated using the CIoU_Loss function + Distribution Focal Loss. The calculation method is shown in Equation (2).

[0072]

[0073] In the formula, b and b gt ρ represents the center point of the predicted bounding box and the center point of the ground truth bounding box. 2 () represents the Euclidean distance, and c is the length of the diagonal of the smallest rectangle containing the two boxes;

[0074] If A is the ground truth bounding box and B is the predicted bounding box, then the IoU calculation formula in the equation is shown in equation (3).

[0075]

[0076] DFL loss: DFL optimizes the probability of the two positions closest to label y (left and right) in the form of cross-entropy, so that the network can focus on the target position and the distribution of the surrounding area more quickly.

[0077] The category loss is calculated by using the category score of the predicted box and the eone-hot representation of the ground truth box, as shown in Equation (4).

[0078]

[0079] The average map during the training process is as follows: Figure 6 As shown, mAP_0.5:0.95 represents the average precision (mAP) obtained at different Intersection over Union (IoU) thresholds (from 0.5 to 0.95, with a step size of 0.05) (0.5, 0.55, 0.6, 0.65, 0.7, 0.75, 0.8, 0.85, 0.9, 0.95), and then the average of all values ​​is taken.

[0080] In step S3, capsule networks are used to enhance the focus on the positional feature information of the mouth shape;

[0081] Capsule networks consist of convolutional layers, initial capsules, and digital capsules, with the following structure: Figure 3 As shown. Convolutional structures are used for feature extraction in capsule networks. The convolutional layers in capsule networks extract local features from the image by performing convolution operations on the input data. During the convolution process, each convolutional kernel slides at different positions in the input data and shares the same weight parameters.

[0082] The initial capsule is the initial layer in the capsule network, responsible for extracting low-level local features from the input lip-shape image or video. These features include edges, corners, and textures, which are used for subsequent lip-shape recognition. The initial capsule captures these features through a set of learnable weight and bias parameters and encodes these features into the capsule's output vector. The capsule's output vector represents the presence or absence of features, as well as the spatial information and directionality of the features, which is used to capture the shape and dynamic changes of the lips in lip-shape detection.

[0083] Digital capsules reside deeper within the capsule network and are responsible for constructing higher-level abstract representations from the low-level features extracted from the initial capsules. These representations are used to capture more complex lip-shape patterns and structures. By combining and transforming information from the initial capsules, a more comprehensive understanding of lip shapes can be formed.

[0084] In step S3, the BiLSTM network uses deep learning methods to model and extract features from lip movements. By simultaneously considering the context of the sequence, it more accurately captures the dynamic changes and contextual relationships in the lip movement time series, extracting deep-level features from the lip movement time series. These deep-level features include static information about the lip shape and information about the dynamic changes in the lip shape. Specifically, the BiLSTM network receives a series of spatial depth features extracted by the capsule network as input and converts them into time series data. Then, the network processes the sequence through two Long Short-Term Memory (LSTM) layers in both forward and backward directions. The forward LSTM layer encodes from the beginning to the end of the sequence, capturing forward lip movement information; while the backward LSTM layer encodes from the end to the beginning of the sequence, capturing backward lip movement information. The outputs of these two LSTM layers are concatenated to fully utilize the contextual information in the sequence.

[0085] In step S4 of this example, an adaptive random forest algorithm based on the love evolution algorithm is used. The Random Forest (RF) algorithm consists of a large number of decision trees, and the final result is determined by voting on the classification results of all decision trees. The training set used for each tree is resampled using Bootstrap, and samples and feature attributes are randomly selected during the training process, enhancing the diversity and uncertainty of the algorithm. The feature variables of each tree are not completely identical; even if some input features are unreliable, they will only affect a small portion of the decision trees and will not affect the final voting decision. Therefore, this algorithm has strong data adaptability, can directly process high-dimensional data, is less affected by noise and outlier data, and has high classification accuracy and strong generalization ability.

[0086] The Love Evolution Algorithm (LEA) is a novel metaheuristic optimization algorithm inspired by and improved upon the stimulus-value-role theory. In LEA, individual characteristics are abstracted as variables, and all of a person's characteristics are considered a candidate solution, with happiness as the objective function value. LEA comprises three phases: stimulus, value, and role. Both parties experience and benefit from these phases, regardless of the relationship's outcome. In the stimulus phase, both parties are stimulated by their respective appearances, behaviors, and personalities. The value phase involves using convolution to define convolution operators and determining which couples can enter the role phase. Furthermore, one of LEA's key features is its unique search operations, including convolution, crossover, and multiplication and division mutations between variables, which give it strong evolutionary capabilities, fast search speed, and high optimization ability.

[0087] In step S4, the output of BiLSTM is used as the input variable for the random forest decision tree, and the love evolution algorithm is used to optimize the number of hyperparameter subsets T and the number of features m of the random forest. Finally, the classification results of each decision tree are voted on to achieve the function of automatically determining the lip shape category. The specific implementation process includes the following steps;

[0088] Step 1: Construct the dataset and divide the data, with one part used as training data and the remaining part as test data;

[0089] Step 2: Use random forest to build a model with optimized parameters based on the love evolution algorithm, and evaluate the model's performance on the training data as happiness; the evaluation process includes evaluating accuracy and loss function value;

[0090] Step 3: Use the bootstrap resampling method to sample T times with replacement from the updated training dataset to form T training subsets;

[0091] Step 4: Use the output of BiLSTM as the input variable for the random forest decision tree. The generation of the decision tree is a process of recursively building a binary classification tree. At each node of the tree, m features are randomly selected from M input variables as candidate features. Generally, m is the root mean square value of M. Then, the optimal feature and the best binary split point are selected from these m features to split the node, thereby building T decision trees and generating a random forest. Step 5: Use the love evolution algorithm to optimize the number of hyperparameter subsets T and the number of features m of the random forest, and generate multiple pairs of couples for love evaluation.

[0092] Step 6: For spouses with an acceptance level greater than 0.5, proceed with the separation process and enter the reflection phase to continue reflecting and changing personal characteristics until the conditions are met before proceeding to the next step;

[0093] Step 7: Continue with the value and role assessment stages for spouses with low acceptance levels. If an acceptance level greater than 0.5 occurs at any stage, proceed to Step 6 to update characteristics.

[0094] Step 8: Repeat steps 6 and 7 for multiple iterations until the termination condition is met (such as reaching the maximum number of iterations or meeting the preset accuracy requirements).

[0095] Step 9: Based on the lip-shape detection and classification results of each decision tree in the optimized random forest, vote to determine the final lip-shape category result.

[0096] The detection method is used in singing instruction to help judge whether a singer's lip movements are standard.

[0097] The example of face and mouth shape detection in this case is as follows: Figure 7 As shown in (a)-(f), after inputting the facial image of the singer into the trained YOLOv8 model, the corresponding lip shape image can be automatically detected and segmented.

[0098] In this example, the process of detecting vocalization lip movements is as follows: Figure 4 As shown: First, a real-time lip-sync image segmentation network model based on YOLOv8 is constructed. The collected singing lip-sync time series is input into YOLOv8 for lip-sync detection to obtain a time series image of the lip-sync. Second, the lip-sync time series image is divided into training, validation, and test sets in an 8:1:1 ratio. Next, a series of spatial depth features are extracted from the lip-sync time series image using a capsule network and input into a BiLSTM network for extracting the time series features of the singing lip-sync. Then, a random forest with parameters optimized by the Love Evolution algorithm is used for prediction and classification. Finally, the overall model is trained using the training and validation sets, and tested and analyzed using the test set. The performance of the model can be evaluated by inputting the test set.

[0099] In this example, to meet the demand for large-scale image processing and improve training efficiency, an RTX 3080 (10GB) and a 12vCPU Intel(R) Xeon(R) Platinum 8255C CPU@2.50GHz processor are used. The Python programming language is used, with CUDA version 11.0 for all versions, and the deep learning framework used is PyTorch 1.7.0.

[0100] This patent uses facial video images of teachers and students from the Music Department of Fujian Normal University singing vowels to construct a dataset. A total of 300 teachers and students of different ages and genders were selected as the subjects. To ensure model robustness, the same camera was used to capture videos in the morning, noon, and evening. The captured videos were then extracted as continuous time series, categorized according to mouth shapes such as vertical, round, pointed, open, closed, and smiling. 6000 images were extracted for each mouth shape, totaling 36000 images across the six mouth shapes. These images were then divided into training, validation, and test sets in an 8:1:1 ratio (28800:3600:3600).

Claims

1. A real-time singing lip-syncing detection method based on spatiotemporal depth feature adaptive decision-making, characterized in that: Includes the following steps; Step S1: Collect facial video images of the singers and annotate the images according to the actual lip movements to construct a sample dataset for model training and testing. Step S2: Construct a real-time target detection network for singing lip movements based on YOLOv8 to segment lip movement images from face videos; Step S3: Construct a lip shape spatial depth feature extraction model based on capsule network, input a series of extracted spatial depth features into BiLSTM network, and further extract time series features of lip shape image through BiLSTM network; Step S4: Optimize the hyperparameters of the random forest algorithm using the love evolution algorithm, and map the extracted time series features to the high-dimensional space of the random forest for adaptive decision classification. Step S5: Train the model using the training and validation sets in the sample dataset, and test and analyze the results using the test set in the sample dataset. In step S3, the BiLSTM network uses deep learning methods to model and extract features from lip movements. By simultaneously considering the context of the sequence, it more accurately captures the dynamic changes and contextual relationships in the lip movement time series, extracting deep-level features from the lip movement time series. These deep-level features include static information about the lip shape and information about the dynamic changes in the lip shape. Specifically, the BiLSTM network receives a series of spatial depth features extracted by the capsule network as input and converts them into time series data. Then, the network processes the sequence through two Long Short-Term Memory (LSTM) layers in both forward and backward directions. The forward LSTM layer encodes from the beginning to the end of the sequence, capturing forward lip movement information; while the backward LSTM layer encodes from the end to the beginning of the sequence, capturing backward lip movement information. The outputs of these two LSTM layers are concatenated to fully utilize the contextual information in the sequence. In step S4, the output of BiLSTM is used as the input variable of the random forest decision tree, and the love evolution algorithm is used to optimize the number of hyperparameter subsets T and the number of features m of the random forest. Finally, the classification results of each decision tree are voted on to realize the function of automatically determining the lip shape category. The specific implementation process is as follows, including the following steps. Step 1: Construct the dataset and divide the data, with one part used as training data and the remaining part as test data; Step 2: Use random forest to build a model with optimized parameters based on the love evolution algorithm, and evaluate the model's performance on the training data as happiness; the evaluation process includes evaluating accuracy and loss function value; Step 3: Use the bootstrap resampling method to sample T times with replacement from the updated training dataset to form T training subsets; Step 4: Use the output of BiLSTM as the input variable of the random forest decision tree. The generation of the decision tree is the process of recursively building a binary classification tree. At each node of the tree, m features are randomly selected from M input variables as candidate features, where m takes the root mean square value of M. Then, the optimal feature and the best binary split point are selected from these m features to split the node, thereby building T decision trees and generating a random forest. Step 5: Optimize the number of hyperparameter subsets T and the number of features m of the random forest using the love evolution algorithm to generate multiple pairs of couples for love evaluation; Step 6: For spouses with an acceptance level greater than 0.5, proceed with the separation process and enter the reflection phase to continue reflecting and changing personal characteristics until the conditions are met before proceeding to the next step; Step 7: Continue with the value and role phase for spouses who are less receptive; If the acceptance rate is greater than 0.5 at a certain stage, proceed to step 6 to update the features; Step 8: Repeat steps 6 and 7 for multiple iterations until the termination condition is met; Step 9: Based on the lip-shape detection and classification results of each decision tree in the optimized random forest, vote to determine the final lip-shape category result.

2. The real-time singing lip-sync detection method based on spatiotemporal depth feature adaptive decision-making according to claim 1, characterized in that: The specific method of step S1 is as follows: select singers of different ages and genders as the collection subjects, with no less than 300 people. In order to ensure the robustness of the model, use the same shooting equipment to collect videos in the morning, noon and evening. Extract the collected videos according to the continuous time series. Collect the videos according to the mouth shape classification: vertical mouth, round mouth, pointed mouth, open mouth, closed mouth and smiling mouth. Extract no less than 6,000 continuous time series for each mouth shape. Divide these images into training set, validation set and test set according to the ratio of 8:1:

1.

3. The real-time singing lip-sync detection method based on spatiotemporal depth feature adaptive decision-making according to claim 1, characterized in that: In step S2, lip-shape detection is performed on the continuous time series image of the singer's face. In step S2, a real-time lip-shape target detection network based on YOLOv8 is first constructed, and then the face image of the singer is detected and segmented to effectively detect the lip shape in real time. The lip-shape image of singing is segmented from the face video as the input of the model in subsequent steps. The YOLOv8 network structure consists of three parts: Backbone, Neck, and Head. Backbone is responsible for extracting high-level semantic features from the input image. It contains multiple convolutional layers and pooling layers, which build a deep feature extractor that transforms the image into a feature representation with rich semantic information. The Neck is used to further integrate and adjust the features extracted by the Backbone. By fusing features at different levels, it enhances the network's ability to perceive targets. The Neck structure consists of a series of convolutional layers and upsampling layers, which are used to fuse low-level feature maps with high-level feature maps to provide more global and local feature information, thereby helping to capture the details and contextual relationships of targets and improve the accuracy of target detection. The Head is the last layer of the YOLOv8 model. In object detection tasks, it uses bounding box regressors and classifiers to output the bounding box location and category information of the object. The specific method for calculating the training loss of the YOLOv8-based real-time detection and segmentation network model for singing lip movements is as follows: YOLOv8's loss function includes the rectangular box loss (rect / box_loss) and the classification loss (cls_loss), and the total loss function is the sum of the two with their respective weight coefficients; the calculation formula is shown in equation (1): Loss=a×L obj +b×L rect +c×L cls (1) Where a, b, and c are the weighting coefficients for each type of loss, and the sum of the three equals 1; The classification loss and confidence loss were calculated using BCEWithLogitsLoss, and the bounding box loss was calculated using the CIoU_Loss function + DistributionFocal Loss; the calculation method is shown in Equation (2): In the formula, b and b gt ρ represents the center point of the predicted bounding box and the center point of the ground truth bounding box. 2 () represents the Euclidean distance, and c is the length of the diagonal of the smallest rectangle containing the two boxes; If A is the ground truth bounding box and B is the predicted bounding box, then the IoU calculation formula in the equation is as shown in equation (3): DFL loss: DFL optimizes the probability of the two positions closest to label y (left and right) in the form of cross-entropy, so that the network can focus on the target position and the distribution of the neighboring area more quickly; the class loss is calculated by the class score of the predicted box and the eone-hot representation of the ground truth box, as shown in Equation (4):

4. The real-time singing lip-syncing detection method based on spatiotemporal depth feature adaptive decision-making according to claim 3, characterized in that: In step S3, capsule networks are used to enhance the focus on the positional feature information of the mouth shape; Capsule networks consist of convolutional layers, initial capsules, and digital capsules. Convolutional structures are used for feature extraction in capsule networks. Convolutional layers in capsule networks extract local features from images by performing convolution operations on the input data. During convolution, each convolutional kernel slides at different positions in the input data and shares the same weight parameters. The initial capsule is the initial layer in the capsule network, responsible for extracting low-level local features from the input lip-shape image or video. These features include edges, corners, and textures, which are used for subsequent lip-shape recognition. The initial capsule captures these features through a set of learnable weight and bias parameters and encodes these features into the capsule's output vector. The capsule's output vector represents the presence or absence of features, as well as the spatial information and directionality of the features, which is used to capture the shape and dynamic changes of the lips in lip-shape detection. Digital capsules reside deeper within the capsule network and are responsible for constructing higher-level abstract representations from the low-level features extracted from the initial capsules. These representations are used to capture more complex lip-shape patterns and structures. By combining and transforming information from the initial capsules, a more comprehensive understanding of lip shapes can be formed.

5. The real-time singing lip-sync detection method based on spatiotemporal depth feature adaptive decision-making according to claim 1, characterized in that: The detection method is used in singing instruction to help judge whether a singer's lip movements are standard.

Images