Vocal singing emotion evaluation method based on audio-visual fusion

Through the audio-visual fusion method, combining video and audio features, the dual-modal cross attention mechanism is used to complement information, which solves the problem of facial expressions being ignored and evaluation efficiency in the existing technology, and achieves more accurate emotional evaluation, which is suitable for Mobile devices, respond to national aesthetic education policies.

CN119027765APending Publication Date: 2024-11-26CHONGQING UNIV OF POSTS & TELECOMM
View PDF 3 Cites 0 Cited by

Patent Information

Application Number
CN202410973280.8
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-19
Publication Date
2024-11-26

AI Technical Summary

Technical Problem

The prior art ignores the importance of facial expressions in vocal singing emotional evaluation, and the traditional evaluation methods are inefficient and costly, and the visual and auditory emotional scoring weights are not suitable, which cannot effectively improve the accuracy of singing emotional evaluation.

Method used

The audio-visual fusion method is adopted to capture dynamic information of macro and micro-expressions through video features, and combine it with audio features. The dual-modal cross-attention mechanism is used to complement information, learn modal weights independently, and build loss function to calculate prediction scores. result.

Benefits of technology

It significantly improves the accuracy of emotional evaluation of vocal singing, can perform emotional scoring without being restricted by venue, time, and teacher conditions, responds to the national aesthetic education policy, and improves the efficiency of music learning.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119027765A_ABST
    Figure CN119027765A_ABST
Patent Text Reader

Abstract

The invention relates to a vocal music singing emotion evaluation method based on audio-visual fusion, and belongs to the field of vocal music emotion evaluation, and the method comprises the steps: constructing a singing emotion scoring data set, and carrying out the preprocessing of the data set; a slow path and a fast path of a video feature extraction network are adopted to capture dynamic information of macro expressions and micro expressions in the video at the same time, feature information extracted from the slow path and the fast path is fused, and visual modal features are obtained; extracting auditory modal features, capable of representing the emotion of the singer, of the singing audio; performing interaction on the visual modal characteristics and the auditory modal characteristics by adopting a bimodal cross attention mechanism to obtain complementary visual modal characteristics and auditory modal characteristics, setting corresponding weight parameters capable of autonomously learning for the complementary visual modal characteristics and auditory modal characteristics, and obtaining a scoring result according to the characteristics and weights; and constructing a corresponding loss function, and calculating a prediction scoring error according to the loss function. According to the invention, the accuracy of emotion scoring in the field of vocal singing emotion intelligent evaluation is obviously improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The invention belongs to the field of vocal emotion evaluation, and relates to a vocal singing emotion evaluation method based on audio-visual fusion. Background Art

[0002] In the field of music education, related research usually only conducts emotional evaluation on singing audio, ignoring the importance of facial expressions in singing emotion evaluation.

[0003] The existing traditional evaluation methods are time-consuming, venue-limited, require a large number of teachers, are inefficient, and have high training costs. In recent years, my country has attached great importance to the development of aesthetic education and has issued many articles on the use of artificial intelligence to improve the evaluation process and revitalize teaching content.

[0004] The prior art with publication number CN115579021A proposes a vocal performance scoring method and system based on neural network and audio-visual fusion, using different neural networks to obtain expert scoring data in three dimensions, including audio scoring, emotional scoring and clothing scoring, and then input the scores of the three dimensions into the expert scoring fitting neural network to finally obtain a comprehensive score. It extracts facial features and pitch sequences for emotional scoring, but it still has the following problems: it does not take into account the amplitude and duration of facial mouth movements during singing; it does not take into account the rapid duration of muscle and eye movements; the visual and auditory emotional scoring weights are not equal, how to comprehensively obtain a more suitable weight for the two; in the traditional teacher music evaluation method, it is usually necessary to quantify the singing emotional evaluation results instead of classifying them. Therefore, in order to improve the efficiency and accuracy of singing emotional evaluation, a vocal singing emotional evaluation method based on audio-visual fusion is urgently needed to solve this problem. Summary of the invention

[0005] In view of this, the purpose of the present invention is to provide a vocal singing emotion evaluation method based on audio-visual fusion.

[0006] In order to achieve the above object, the present invention provides the following technical solutions:

[0007] A method for evaluating vocal singing emotion based on audio-visual fusion, comprising:

[0008] S1. Construct a singing emotion scoring dataset and preprocess the dataset;

[0009] S2, using the slow path and fast path of the video feature extraction network to simultaneously capture the dynamic information of macro-expressions and micro-expressions in the video, and fusing the feature information extracted on the slow path and the fast path to obtain the visual modality feature;

[0010] S3, using the Wav2Vec2.0-base model to extract the auditory modal features of the singing audio that can represent the singer's emotions;

[0011] S4. Use a bimodal cross-attention mechanism to interact with visual modal features and auditory modal features to obtain complementary visual modal features and auditory modal features, set corresponding weight parameters that can be learned autonomously, and then obtain scoring results based on the features and weights;

[0012] S5. Construct the corresponding loss function and calculate the error of the predicted score based on the loss function.

[0013] Further, in step S1, it includes:

[0014] S11, firstly recording video files of several singers singing songs, and cutting the recorded video files into sub-video files of preset length;

[0015] S12, vocal experts give comprehensive emotional scoring annotations based on the singer's facial expressions and vocal emotional expressions, and add label annotations to the sub-video file, wherein the label annotations include: video file serial number, singer's gender, song serial number, emotional scoring annotation, sub-video file serial number;

[0016] S13, using the cv2.VideoCapture function in the OpenCV library to read the video frame by frame, calculate the frame interval, and extract the frame image from the sub-video file; according to the number of frames extracted per second, the facial expression of the singer is divided into macro-expression and micro-expression; and then the facial area of ​​the face in the frame image is segmented and saved;

[0017] S14. Put the original audio directly into the processor module of Wav2Vec2.0 to preprocess the audio data and convert the audio data into audio features.

[0018] Furthermore, in step S2, a video branch network is used to capture visual modal features, wherein the video branch network includes a slow path for extracting macro-expression features and a fast path for extracting micro-expression features; several 3Dcnext networks are used on both the fast path and the slow path as backbone modules for feature extraction, and the visual features extracted on the fast path are down-sampled in the time dimension through several time-step convolution modules and then fused with the features on the slow path to finally obtain spatiotemporal fusion features.

[0019] Furthermore, in step S2, 3Dcnext includes a backbone network and a downsample part, which converts 2D convolution into 3D convolution, wherein the backbone network includes three 3D convolution layers with a step size of 1. The first convolution layer uses 3D separable convolution with a convolution kernel size of 7*7*7, followed by layer normalization. The second and third convolution layers both use 1*1*1 point convolution, and the activation function is GELU.

[0020] The downsample part uses a 3D convolution with a convolution kernel size of 2*2*2 and a step size of 2 to downsample the visual features.

[0021] Further, in step S3, Wav2Vec2.0-processor is used as a front-end processor to convert the original audio data into a format that can be understood by the Wav2Vec2.0-base model. The Wav2Vec2.0-base model will use the processed audio features to perform emotion feature extraction, and input the extracted auditory emotion features into the downstream one-dimensional convolutional network to finally obtain high-dimensional auditory modal features.

[0022] Further, in step S4, it includes:

[0023] S41. The auditory modality features and visual modality features output by the feature extraction network are represented as X a and X v ,The cross-attention mechanism is used to dynamically adjust the information flow between the two modalities and obtain the cross-attention features;

[0024] S42, adding the cross attention feature to the original feature to complement the information, and obtaining the auditory modal feature and the visual modal feature after information complementation [F a ,F v ];

[0025] S43, for the complementary auditory modality features and visual modality features [F a ,F v ] respectively set different weight parameters [W a ,W v ], optimize the weights [W by back propagation and gradient descent a ,W v ], after autonomously learning the weights of the two modalities, the final prediction score is calculated.

[0026] Further, in step S41, the calculation process of the cross attention feature is:

[0027]

[0028] in, is the learned weight matrix, d k are the dimensions of the query and key vectors.

[0029] Further, in step S42, the complementary auditory modality features and visual modality features [F a ,F v ] is calculated as:

[0030] F a =X a +CrossAttention(X a ,X v )

[0031] F v =X v +CrossAttention(X v ,X a )

[0032] The complementary auditory modal features and visual modal features retain the feature information of the original features.

[0033] Furthermore, in step S43, different weights are multiplied by the two modes, and then the features are added and input into the two-layer Linear, and finally the model prediction score Score is output:

[0034] F = Concat(W a ·F a ,W v ·F v )

[0035] Score=Linear2(Linear1(F))

[0036] Among them, F represents the features after AVCroatt fusion.

[0037] Furthermore, the Smooth L1Loss loss function is used to calculate the error between the predicted score and the expert's actual score, which is expressed as:

[0038]

[0039] Where: x n Score the prediction, y n is the expert score, β is a hyperparameter used to control the transition between L1 loss and L2 loss, and n represents the total number of samples in the dataset for training, validation, or testing.

[0040] The beneficial effects of the present invention are:

[0041] The present invention introduces visual information and designs an algorithm based on the evaluation weight rules of the facial features of the singer and the real scene, filling the gap in the market for emotional evaluation based only on the singing audio, and significantly improving the accuracy of emotional scoring in the field of vocal singing emotional intelligence evaluation. The algorithm model can be deployed to mobile devices, which can achieve singing emotional scoring without being restricted by conditions such as venue, time, and teachers, improving the efficiency of music learning and responding to the country's aesthetic education policy.

[0042] Other advantages, objectives and features of the present invention will be described in the following description to some extent, and to some extent, will be obvious to those skilled in the art based on the following examination and study, or can be taught from the practice of the present invention. The objectives and other advantages of the present invention can be realized and obtained through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] In order to make the purpose, technical solutions and advantages of the present invention more clear, the present invention will be described in detail below in conjunction with the accompanying drawings, wherein:

[0044] Figure 1 Schematic diagram of the dataset constructed for the present invention;

[0045] Figure 2 It is a schematic diagram of the visual data preprocessing process of the present invention;

[0046] Figure 3 It is a schematic diagram of the overall process of the present invention;

[0047] Figure 4 It is a structural schematic diagram of a backbone module of a video branch network of the present invention;

[0048] Figure 5 A schematic diagram of the flow of the cross-attention mechanism of the audio-visual fusion branch network of the present invention;

[0049] Figure 6 It is a schematic diagram of the loss results of the present invention. DETAILED DESCRIPTION

[0050] The following describes the embodiments of the present invention by specific examples, and those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the illustrations provided in the following embodiments only illustrate the basic concept of the present invention in a schematic manner, and the following embodiments and features in the embodiments can be combined with each other without conflict.

[0051] Among them, the drawings are only used for illustrative explanations, and they only represent schematic diagrams rather than actual pictures, and should not be understood as limitations on the present invention. In order to better illustrate the embodiments of the present invention, some parts of the drawings may be omitted, enlarged or reduced, and do not represent the size of actual products. For those skilled in the art, it is understandable that some well-known structures and their descriptions in the drawings may be omitted.

[0052] The same or similar numbers in the drawings of the embodiments of the present invention correspond to the same or similar parts; in the description of the present invention, it should be understood that if the terms "upper", "lower", "left", "right", "front", "rear", etc. indicate the orientation or position relationship, they are based on the orientation or position relationship shown in the drawings, which is only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operate in a specific orientation. Therefore, the terms describing the position relationship in the drawings are only used for illustrative purposes and cannot be understood as limiting the present invention. For ordinary technicians in this field, the specific meanings of the above terms can be understood according to specific circumstances.

[0053] See also Figure 1 to Figure 6 , which is a vocal singing emotion evaluation method based on audio-visual fusion. In general, the vocal singing emotion evaluation method based on audio-visual fusion of the present invention first classifies singing expressions into macro-expressions and micro-expressions, and proposes the idea of ​​fast-slow path for visual modality, which is responsible for the emotion calculation of micro-expressions and macro-expressions respectively, and uses 3Dcnext to fully extract the temporal features of micro-expressions and the spatial features of macro-expressions, and then uses lateral connection to connect the two paths to complete the interaction of spatiotemporal information.

[0054] Secondly, the wav2vec2.0-base audio model is used as the backbone to fully extract the audio features of the singing. The extracted auditory emotion features are input into the downstream one-dimensional convolutional network, and finally high-dimensional audio features are obtained to match the input feature dimension of the fusion module.

[0055] Finally, the present invention proposes the AVCroatt module, which simulates the teacher's scoring method in real teaching, further analyzes the emotional contribution of each modality according to the students' external expression changes and vocal emotional characteristics, and more objectively evaluates the students' emotional investment and expression in the vocal performance process, making up for the lack of quantitative evaluation of the singer's emotional communication ability in the existing intelligent vocal evaluation field.

[0056] Specifically, it includes the following steps:

[0057] S1. Construct a singing emotion scoring dataset and preprocess the dataset;

[0058] S2, using the slow path and fast path of the video feature extraction network to simultaneously capture the dynamic information of macro expressions and micro expressions in the video, and fusing the feature information extracted on the slow path and the fast path;

[0059] S3, using Wav2Vec2.0-base model to extract emotional features of singing audio;

[0060] S4. Use a bimodal cross-attention mechanism to interact with visual modal features and auditory modal features to obtain complementary visual modal features and auditory modal features, set corresponding weight parameters that can be learned autonomously, and then obtain scoring results based on the features and weights;

[0061] S5. Construct the corresponding loss function and calculate the error of the predicted score based on the loss function.

[0062] Example

[0063] In step S1 of this embodiment, a singing emotion scoring dataset (Sing Video Audio Emotion, SVAE) is constructed to address the problem that there is currently no public dataset proposed for vocal singing emotion evaluation. The dataset was recorded by 26 sophomores and juniors majoring in vocal music using mobile phones. The recorded content is the climax of 5 songs: "My Motherland and I", "Nanshan South", "Chrysanthemum Terrace", "Chengdu", and "Yellow River Cantata". The recording location is the piano room of Chongqing University of Humanities, Science and Technology. The recording results are 546 long video files with a duration of 25-40s, and the video resolution is 1024×576 pixels. Considering the easy-to-change characteristics of singing tasks, the inconsistent emotions before and after a long video may cause a large error in the scoring result, so the above-mentioned long video files are intercepted into sub-video files (.MP4) with a duration of 4s.

[0064] The data labels are annotated by vocal experts based on the singer's facial expressions and vocal emotions. The sample population is music learners with a certain foundation, so the score annotations are in the range of [60, 100]. The label annotations include: "long video number_gender_song number_emotional score annotation_the number of short videos captured from the long video". The final constructed dataset is as follows: Figure 1 shown.

[0065] Preprocess the visual and auditory data in the dataset. Figure 2As shown in the figure, for visual data, the extraction of video frames uses the cv2.VideoCapture function in the OpenCV library to read the video frame by frame, calculate the frame interval, and extract frame images from each video with a duration of 4s. Generally speaking, facial expressions can be divided into two categories, with macro expressions between 1s and 5s. The former extracts 4 frames (1 frame per second), and the latter extracts 32 frames (8 frames per second), representing the fast and slow expressions of the singing face, respectively. Next, the face_locations function in the face_recognition library is used for face detection. For each detected face, the face area is segmented, and then the extracted face image array is saved as a .npy file. The above preprocessing operations ensure the consistency of the data and the standardization of the model input, which facilitates the loading and processing of the subsequent network model.

[0066] For the preprocessing of auditory data, the original audio is directly put into the processor module of Wav2Vec2.0 to preprocess the audio data and convert it into a series of audio features. Model training is not involved here.

[0067] In step S2 and step S3 of the present invention, the high-dimensional features of facial expression movement changes and voice emotion changes are first extracted using the video branch and the audio branch, respectively, and finally the audio-visual fusion module is used to combine the two emotion features obtained previously to output the final emotion score. Figure 3 As shown, the specifically constructed network model includes: a video branch network, an audio branch network, and an audio-visual fusion (AVCroatt) branch network. In this embodiment, AVCroatt is a multimodal emotion regression task.

[0068] In step S2 of this embodiment, the video branch network simultaneously processes video streams of different temporal resolutions to capture the dynamic information of macro-expressions and micro-expressions in the video. The network contains two parallel paths: (1) Slow path (Slow_emotion): This path processes a small portion of video frames (4 frames) and focuses on capturing high-quality spatial information, namely, details and morphology in the video, i.e., macro-expression feature extraction. (2) Fast path (Fast_emotion): In contrast, the fast path processes the video at a higher frame rate (32 frames), i.e., micro-expression feature extraction. It focuses on capturing the temporal changes of rapid expression movements. This path usually adopts a shallower network structure because it needs to process more frames and reduce the amount of computation. It mainly focuses on the rapid changes in facial expressions rather than the specific details of the expression appearance.

[0069] The video branch network uses 3Dcnext as the backbone module for feature extraction, such as Figure 4As shown in the figure, 3Dcnext is a combination of the ConvNextv2 backbone network and the downsample part with 3D convolution, using GRN to effectively alleviate the problem of feature collapse. With the help of its backbone network idea, 3Dcnext is responsible for extracting feature information, and downsamples the extracted information to reduce the complexity of the model while retaining important information. Specifically, 3Dcnext includes a backbone network and a downsample part, where the backbone network includes 3 3D convolution layers with a step size of 1. The first convolution layer uses 3D separable convolution with a convolution kernel size of 7*7*7, followed by layer normalization. The second and third convolution layers both use 1*1*1 point convolution, and the activation function is GELU.

[0070] The downsample part uses a 3D convolution with a convolution kernel size of 2*2*2 and a step size of 2 to downsample the visual features.

[0071] The above operation is to transform the idea of ​​2D image feature extraction to 3D video features. Subsequently, this embodiment uses a time-step convolution module (Time Stride Conv) to upsample the Fast visual features to T=4 in the time dimension and upsample them to twice the original channel in the channel dimension in the Fast branch, and then add them to the spatial information of Slow in the channel dimension to further realize the fusion of spatiotemporal features.

[0072] In step S3, this embodiment uses the Wav2Vec2.0-base model to extract the emotional features of the singing audio. Wav2Vec2.0-base is a large pre-trained audio model based on Wav2Vec2.0-base, which can ensure the full extraction of the emotional features of the input singing audio. The model was first pre-trained on the LibriSpeech audio data set for 960 hours with a sampling rate of 16kHz. Then, the pre-trained model is fine-tuned on the RAVDESS data containing emotional speech of different speakers and scenes, and finally the Wav2Vec2.0-base model weights are obtained. Therefore, this embodiment adopts the method of transfer learning, and the Wav2Vec2.0-base model is fine-tuned on the data set provided by the present invention, and then used as the audio feature extraction backbone of this method. Specifically, Wav2Vec2.0-processor is used as the front-end processor, which is responsible for converting the original audio data into a format that the model can understand. The Wav2Vec2.0-base model will then use these processed audio features to perform emotional feature extraction, input the extracted auditory emotional features into the downstream one-dimensional convolutional network, and finally obtain high-dimensional audio features to match the input feature dimension of the fusion module.

[0073] In this embodiment, in step S4, the auditory modality features and visual modality features output by the feature extraction network are represented as X a and X v In order to achieve information fusion and interaction between the two, this embodiment adopts the cross attention mechanism method, such as Figure 5 The Cross Attention part in is as follows:

[0074]

[0075]

[0076] in, is the learned weight matrix, d k are the dimensions of the query and key vectors.

[0077] The bimodal cross-attention mechanism can dynamically adjust the information flow between the two modalities. This mechanism can not only capture the emotional interdependence between the two modalities, but also highlight the most relevant emotional features between the two modalities. With the idea of ​​residuals, the loss of important feature information of the original modality is prevented. The cross-attention features are added to the original features to obtain the auditory and visual modal features after information complementation, as shown in the following formula:

[0078] F a =X a +CrossAttention(X a ,X v )

[0079] F v =X v +CrossAttention(X a ,X v )

[0080] After the above feature complementation operation, the complementary auditory and visual modal features are expressed as [F a ,F v ], and set different weight parameters [W a ,W v ], simulating the teacher's emotional scoring method, according to the students' facial expression changes (visual) and vocal emotions (auditory) have different importance on the final scoring results, and the weights are optimized through back propagation and gradient descent [W a ,W v ], after the network autonomously learns the weights of the two modes, it then multiplies the different weights with the two modes, adds the features, and inputs them into the two-layer Linear, and finally outputs the model prediction score Score, as shown in the following formula:

[0081] F = Concat(W a ·F a ,W v ·F v )

[0082] Score=Linear2(Linear1(F))

[0083] Among them, F represents the features after AVCroatt fusion.

[0084] In this embodiment, the Smooth L1Loss loss function is used in step S5 to construct the corresponding loss function, which is expressed as:

[0085]

[0086] Where: x n Score the prediction, y n is the expert scoring, β is a hyperparameter used to control the transition between L1 loss and L2 loss. In this embodiment, β is 0.1, and n represents the total number of samples in the dataset used for training, verification or testing.

[0087] In the scoring regression task, this embodiment uses the above loss function to calculate the error between the predicted score and the expert's actual score. It divides each loss by the number of labeled samples n in the current mini-batch, and uses this sum as the training target. The model updates each parameter through back propagation to reduce the loss between the actual value and the predicted value.

[0088] like Figure 6 As shown in the figure, the training loss of the proposed method on the SAVE dataset is 3.79, and the verification loss is 4.12. Finally, the average error of the singing emotion score of the Singemo method on the SAVE dataset test set is 4.03 points, which is consistent with the possible error of the teacher's emotion score.

[0089] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solution of the present invention can be modified or replaced by equivalents without departing from the purpose and scope of the technical solution, which should be included in the scope of the claims of the present invention.

Claims

1. A method for evaluating vocal singing emotion based on audio-visual fusion, characterized in that: It includes: S1. Construct a singing emotion scoring dataset and preprocess the dataset; S2, using the slow path and fast path of the video feature extraction network to simultaneously capture the dynamic information of macro-expressions and micro-expressions in the video, and fusing the feature information extracted on the slow path and the fast path to obtain the visual modality feature; S3, using the Wav2Vec2.0-base model to extract the auditory modal features of the singing audio that can represent the singer's emotions; S4. Use a bimodal cross-attention mechanism to interact with visual modal features and auditory modal features to obtain complementary visual modal features and auditory modal features, set corresponding weight parameters that can be learned autonomously, and then obtain scoring results based on the features and weights; S5. Construct the corresponding loss function and calculate the error of the predicted score based on the loss function.

2. The method for evaluating vocal singing emotion based on audio-visual fusion according to claim 1, characterized in that: In step S1, it includes: S11, firstly recording video files of several singers singing songs, and cutting the recorded video files into sub-video files of preset length; S12, vocal experts give comprehensive emotional scoring annotations based on the singer's facial expressions and vocal emotional expressions, and add label annotations to the sub-video file, wherein the label annotations include: video file serial number, singer's gender, song serial number, emotional scoring annotation, sub-video file serial number; S13, using the cv2.VideoCapture function in the OpenCV library to read the video frame by frame, calculate the frame interval, and extract the frame image from the sub-video file; according to the number of frames extracted per second, the facial expression of the singer is divided into macro-expression and micro-expression; and then the facial area of ​​the face in the frame image is segmented and saved; S14. Put the original audio directly into the processor module of Wav2Vec2.0 to preprocess the audio data and convert the audio data into audio features.

3. The method for evaluating vocal singing emotion based on audio-visual fusion according to claim 1, characterized in that: In step S2, a video branch network is used to capture visual modal features, wherein the video branch network includes a slow path for extracting macro-expression features and a fast path for extracting micro-expression features; several 3Dcnext networks are used on both the fast path and the slow path as the backbone modules for feature extraction, and the visual features extracted on the fast path are down-sampled in the time dimension through several time-step convolution modules and then fused with the features on the slow path to finally obtain spatiotemporal fusion features.

4. The method for evaluating vocal singing emotion based on audio-visual fusion according to claim 2 is characterized in that: In step S2, the structural idea of ​​3Dcnext is to combine the backbone and downsample of ConvNextv2, and transform its 2D convolution into 3D convolution. 3Dcnext is divided into the backbone network and the downsample part, where The 3Dcnext backbone network consists of three 3D convolutional layers with a step size of 1. The first convolutional layer uses 3D separable convolution with a convolution kernel size of 7*7*7, followed by layer normalization. The second and third convolutional layers both use 1*1*1 point convolution with GELU as the activation function. The downsample part uses a 3D convolution with a convolution kernel size of 2*2*2 and a step size of 2 to downsample the visual features.

5. The method for evaluating vocal singing emotion based on audio-visual fusion according to claim 1 is characterized in that: In step S3, Wav2Vec2.0-processor is used as a front-end processor to convert the original audio data into a format that can be understood by the Wav2Vec2.0-base model. The Wav2Vec2.0-base model will use the processed audio features to perform emotional feature extraction, and input the extracted auditory emotional features into the downstream one-dimensional convolutional network to finally obtain high-dimensional auditory modal features.

6. The method for evaluating vocal singing emotion based on audio-visual fusion according to claim 1, characterized in that: In step S4, it includes: S41. The auditory modality features and visual modality features output by the feature extraction network are represented as X a and X v ,The cross-attention mechanism is used to dynamically adjust the information flow between the two modalities and obtain the cross-attention features; S42, adding the cross attention feature to the original feature to complement the information, and obtaining the auditory modal feature and the visual modal feature after information complementation [F a ,F v ]; S43, for the complementary auditory modality features and visual modality features [F a ,F v ] respectively set different weight parameters [W a ,W v ], optimize the weights [W by back propagation and gradient descent a ,W v ], after autonomously learning the weights of the two modalities, the final prediction score is calculated.

7. The method for evaluating vocal singing emotion based on audio-visual fusion according to claim 6 is characterized in that: In step S41, the calculation process of the cross attention feature is: in, is the learned weight matrix, d k are the dimensions of the query and key vectors.

8. The method for evaluating vocal singing emotion based on audio-visual fusion according to claim 7 is characterized in that: In step S42, the complementary auditory modality feature and visual modality feature [Fa, Fv] are calculated as follows: F a =X a +CrossAttention(X a ,X v ) F v =X v +CrossAttention(X v ,X a ) The complementary auditory modal features and visual modal features retain the feature information of the original features.

9. The method for evaluating vocal singing emotion based on audio-visual fusion according to claim 8 is characterized in that: In step S43, different weights are multiplied by the two modes, and then the features are added and input into the two-layer Linear, and finally the model prediction score Score is output: F=Concat(W a ·F a ,W v ·F v ) Score=Linear2(Linear1(F)) Among them, F represents the features after AVCroatt fusion.

10. The method for evaluating vocal singing emotion based on audio-visual fusion according to claim 1, characterized in that: The Smooth L1Loss loss function is used to calculate the error between the predicted score and the expert's actual score, which is expressed as: Where: x n Score the prediction, y n is the expert score, β is a hyperparameter used to control the transition between L1 loss and L2 loss, and n represents the total number of samples in the dataset for training, validation, or testing.

Citation Information

Patent Citations

  • Vocal music performance scoring method and system based on neural network and audio-visual fusion

    CN115579021A

  • Dimensional sentiment analysis method based on joint cross attention mechanism

    CN116167015A

  • Micro-expression detection method based on double-time sequence flow network

    CN116486447A