A vocal performance scoring method and system based on neural networks and audio-visual fusion

Abstract

The present application relates to a kind of vocal performance scoring method and system based on neural network and audio-visual fusion, belong to vocal evaluation field.The method utilizes different neural networks to obtain expert score data in three dimensions, including audio score, emotion score and dressing score, then the scores in three dimensions are input into expert score fitting neural network, and finally the comprehensive score is obtained.The present application makes the evaluation result more real and effective, close to expert score, and the scoring process is more efficient and convenient.

Description

Technical Field

[0001] This invention belongs to the field of vocal performance evaluation and relates to a vocal performance scoring method and system based on neural networks and audiovisual fusion. Background Technology

[0002] In current arts education examinations, such as vocal examinations, examiners typically provide an objective evaluation of the candidate's singing ability and assign a corresponding score. However, this process may suffer from various factors that compromise subjectivity, accuracy, and efficiency.

[0003] Existing audio feature extraction and scoring software on the market can effectively score singers' rhythm, breath control, and tone, but the scoring dimensions are singular and lack a holistic representation of the art.

[0004] Therefore, in order to improve the accuracy of existing audio evaluation software, there is an urgent need for a vocal performance scoring method and system based on neural networks and audiovisual fusion to solve this problem. Summary of the Invention

[0005] In view of this, the purpose of this invention is to provide a vocal performance scoring method based on neural networks and audiovisual fusion. This method combines visual and auditory assessments of the singer's facial expressions and attire to perform a series of scores and judgments. Specifically, it utilizes a large amount of multi-dimensional expert scoring data, including audio scores, attire scores, and emotional scores. An expert database system is constructed using neural networks, making the evaluation results more realistic, effective, and closely aligned with expert assessments. Furthermore, the scoring process is more efficient and convenient.

[0006] To achieve the above objectives, the present invention provides the following technical solution:

[0007] Solution 1: A vocal performance scoring method based on neural networks and audiovisual fusion. This method utilizes different neural networks to obtain scoring data across three dimensions: audio score, emotional score, and attire score. These three scores are then input into an audiovisual fusion expert scoring network to obtain the overall score. The method specifically includes the following steps:

[0008] S1: Obtain the performer's audio data, preprocess it to obtain high-pitched features and rhythm features, then compare it with standard audio data to obtain the minimum distance vector and rhythm fit, and input it into the audio scoring neural network to obtain the audio score;

[0009] S2: Extract keyframes from the performance video, preprocess to obtain audio and face region images corresponding to the keyframes, extract pitch sequences and facial features from them, and input them into the audio emotion neural network and the facial keypoint detector to obtain audio emotion feature vectors and facial expression feature vectors. Then, calculate the Euclidean distance between the audio emotion feature vector and the facial emotion feature vector, i.e., feature vector similarity. Finally, input the feature vector similarity into the audiovisual fusion emotion scoring neural network to obtain the emotion score.

[0010] S3: Extract a frame from the performance video, extract the key features of the clothing, and input them into the clothing style evaluation neural network to obtain the singer's clothing style image feature vector; then obtain the Mel spectrogram from the audio data of the corresponding video, and input it into the audio style evaluation neural network to obtain the singer's music style feature vector; finally, input the singer's clothing style feature vector and music style feature vector into the audiovisual fusion dressing score neural network to obtain the dressing score.

[0011] S4: Input the audio score, emotion score, and clothing score into the audiovisual fusion expert scoring network to obtain the overall score.

[0012] Furthermore, step S1 specifically includes the following steps:

[0013] S101: The video samples of performers scored by experts are divided into training set, validation set and test set according to a 7:2:1 ratio.

[0014] S102: Preprocess the audio data, use intensity for boundary detection and noise reduction, use a bandpass filter for pre-filtering, use a high-pass filter for pre-emphasis of the speech signal, and finally perform frame segmentation and windowing on the audio data.

[0015] S103: The Sum of Harmonics (SHS) method is used to extract pitch features. The input audio is processed by frame-by-frame windowing. The short-time Fourier transform of each frame of speech signal is performed to obtain the spectrum. For each frequency, the energy of the harmonic overlay is superimposed to finally obtain the pitch of the singing data.

[0016] S104: Smooth the extracted pitch sequence;

[0017] S105: The similarity matching algorithm (DTW) is used to measure the similarity between the extracted pitch features and the standard pitch features, and to obtain the minimum distance between the feature vectors of the two melodies.

[0018] S106: Mix the solo vocals and accompaniment music provided by the music standard library according to a certain time shift, and synthesize the fast beat, slow beat and synchronized beat audio suitable for training the rhythm model;

[0019] S107: To obtain the frequency intensity curve, first, perform audio preprocessing (pre-filtering, pre-emphasis, etc.), divide the audio into frames and add windows, obtain the signal spectrum through Fourier transform, and according to the idea of ​​harmonic summation, superimpose the energy of each frequency to obtain the frequency energy curve. Through scale conversion, use semitone values ​​to represent the audio pitch to obtain the frequency intensity curve.

[0020] S108: Extract frequency intensity curves using standard accompanied singing, train and establish a rhythm model based on a hidden Markov model, with the frequency intensity sequence considered as an observable sequence.

[0021] S109: The Vibit algorithm is used for music model recognition, and the path with the highest cumulative probability is selected as the recognition result;

[0022] S110: Based on the model recognition results of all audio segments, quantify the rhythmic fit of the entire song.

[0023] S111: Input the minimum distance of the melody feature vector obtained in S105 and the rhythm fit obtained in S110 into the audio scoring neural network (supervised learning bp neural network based on expert scoring) to train and obtain the audio score.

[0024] Furthermore, step S2 specifically includes the following steps:

[0025] S201: Extract keyframes from video data to obtain static images, record the start and end times of keyframes, detect face regions from static images, and obtain face region images.

[0026] S202: Perform Gaussian filtering on each frame of the face region image, calculate the tilt angle based on the Hough transform, and finally correct the image based on the tilt angle.

[0027] S203: Perform grayscale conversion and normalization on the image;

[0028] S204: Use a facial landmark detection model library to obtain facial feature points;

[0029] S205: Identifies the singer's emotional state by analyzing information such as the degree of mouth opening, eye squinting, and eyebrow raising. It categorizes emotions into eight types: joyful, sorrowful, lyrical, passionate, angry, composed, resolute, and quiet, resulting in a facial emotion feature vector.

[0030] S206: In the audio after the experts have performed emotion classification and annotation, the data is divided according to the ratio of training data: validation data: test data = 7:2:1;

[0031] S207: Construct an audio emotion extraction neural network (a three-layer BP neural network) as an audio emotion classifier, and extract audio feature vectors: pitch, frequency, and intensity as network input;

[0032] S208: After completing the construction of the audio emotion extraction neural network, input the training data for training; after training, use the validation data for validation, and the expected requirements can be met.

[0033] S209: After the audio to be tested is extracted at the time recorded in S205, it is put into the previously trained audio emotion extraction neural network to obtain the audio emotion feature vector.

[0034] S210: Calculate the Euclidean distance between the audio emotion feature vector and the facial emotion feature vector to measure their vector similarity;

[0035] S211: Input the feature vector similarity information into the audiovisual fusion emotion scoring neural network, and obtain the emotion score according to the expert-customized standard.

[0036] Furthermore, step S3 specifically includes the following steps:

[0037] S301: Extract a frame of image content from the input video information as input information for singer's clothing style recognition and analysis, and export the audio track content from the input video information as .wav format as input information for music style analysis;

[0038] S302: In clothing style recognition and analysis, image processing and machine learning techniques are used to establish a mapping relationship between the singer's clothing image and the actual clothing style;

[0039] S303: In the image processing process, a convolutional neural network is used to evaluate clothing style. In order to effectively reduce the complexity of the network and reduce the number of parameters, the main steps in the network are: convolution, pooling, full connection and activation function to complete the construction of the neural network.

[0040] S304: In the analysis of clothing style using machine learning, a well-trained clothing style evaluation neural network is used to extract and format the pixels of the input image, and then a series of operations such as convolution and pooling are performed on each image to obtain key feature points.

[0041] S305: Identify the singer's clothing style by different classification information such as fabric, pattern, partial embroidery, bow tie, and collar flower in the image; it can be divided into 16 clothing styles, including classic, ladylike, romantic, ethnic, avant-garde, light, preppy, casual, unisex, pastoral, punk, street, minimalist, sporty, elegant, and futuristic.

[0042] S306: In music style analysis, the audio information is preprocessed and Fourier transformed to obtain frequency domain information. The frequency domain information is then superimposed to obtain the Mel spectrum. A music style evaluation neural network is used to train and classify these samples to obtain the mapping relationship between the singing music and the performer's music style.

[0043] S307: Input an optimized dataset into the music style evaluation neural network, use Fourier transform to transfer the audio data to the frequency domain, and derive the time-based evolution information of all frequencies of the song.

[0044] S308: Creates fixed-length slices of the spectrum, reducing the vocal spectrum to 256×256 resolution slices as independent samples representing musical styles;

[0045] S309: Build a classifier by cutting the songs in the training set into square spectral images, which will be used as a dataset containing tens of thousands of samples for each category;

[0046] S310: Uses TFLearn, a wrapper for the Tensorflow deep learning library, to train and classify these samples using a deep convolutional neural network;

[0047] S311: Input the .wav format music into the deep convolutional neural network trained by S310. Based on the differences in sound frequencies of different music styles, it can be divided into 11 music styles, including folk, bel canto, blues, classical, disco, hip hop, jazz, metal, pop, reggae, and rock.

[0048] S312: Input the output feature vectors of the performer's clothing style and music style into the audiovisual fusion dressing score neural network, calculate the similarity of the feature vectors according to the expert-customized standards, and obtain the matching score of the performer's clothing and music style, i.e., the dressing score.

[0049] Furthermore, in step S4, the audiovisual fusion expert scoring network employs an expert scoring mechanism. Experts score based on three dimensions: audio score, emotional score, and attire score, all using a 100-point scale. A dataset is constructed based on these three scores. The neural network learns the experts' scoring points through supervised learning. Additionally, experts score the overall performance of the musical act, again using a 100-point scale. A dataset is constructed based on these expert scores, and the audiovisual fusion expert scoring network learns the weights of the experts' scores for each of the three dimensions. This scoring method improves upon the overly mechanical nature of traditional scoring methods that rely solely on pitch and rhythm. By using expert-provided professional scoring data as the prediction target and combining it with traditional scoring points, the scores become more reliable and human-centered.

[0050] Furthermore, the audio scoring neural network, audiovisual fusion emotion scoring neural network, audiovisual fusion clothing scoring neural network, and audiovisual fusion expert scoring network can be trained independently and synchronously, and used together after training; during the training process of each network, the video samples of performers scored by experts are first divided into training set, validation set, and test set according to 7:2:1; each sample contains a video and a score of 4 percentage points, in which the score is given by experts for emotion, clothing, music, and overall evaluation respectively; for the training of each network, (1) select important parameters according to the data characteristics, including convolution kernel (size, number), attention mechanism, activation function, loss function, number of network layers, network optimizer, and learning rate, etc.; (2) synchronize according to the time axis, and The feature vectors of the image and the feature vectors of the audio are jointly input into each scoring neural network. The difference between the network's output score and the professional score given by the expert is used as the loss function. The backpropagation algorithm is used to minimize the loss function. (3) By observing the loss function curves of the training set and the validation set, when the curves of the two sets show a turning point, the training process is terminated. The neural network parameters are determined by the number of iterations at this time, and the test data is used for testing. (4) The trained neural network is tested on the test set. If the difference between the score output by the network and the score given by the expert is within 1 point, the network has passed the test and can be used for the evaluation process of other performers. Otherwise, the network is redesigned and adjusted, and the network training process is repeated.

[0051] Option 2: A vocal performance scoring system based on neural networks and audiovisual fusion, comprising a server, a camera, a microphone, a DSP decoder, and a communication network; the server is used to execute the vocal performance scoring method in Option 1.

[0052] The beneficial effects of this invention are as follows: The expert scoring fitting neural network used in this invention constructs a dataset based on the overall performance scores of vocal performances by experts, which improves the traditional scoring method that scores based solely on pitch and rhythm and is too mechanical. It uses the professional scoring data given by experts as the prediction target, making it more humane.

[0053] The method of this invention combines emotional scoring, attire scoring, and pitch scoring during vocal performances, and also adopts a combination of expert scoring, making the evaluation criteria more diversified and the evaluation results more accurate, which well meets the requirements of objective and accurate music singing scoring in the art quality assessment of primary and secondary schools.

[0054] Other advantages, objectives, and features of the invention will be set forth in part in the description which follows, and in part will be apparent to those skilled in the art from the following examination, or may be learned from practice of the invention. The objectives and other advantages of the invention can be realized and obtained through the following description. Attached Figure Description

[0055] To make the objectives, technical solutions, and advantages of the present invention clearer, the preferred embodiments of the present invention will be described in detail below with reference to the accompanying drawings, wherein:

[0056] Figure 1 A top-down view of a vocal performance;

[0057] Figure 2 This is a schematic diagram of the overall architecture;

[0058] Figure 3 This is a schematic diagram of the network training and evaluation process;

[0059] Figure 4 A diagram illustrating facial expression recognition in vocal performances;

[0060] Figure 5 A diagram illustrating the scoring of a singer's costume matching accuracy;

[0061] Figure 6 A diagram illustrating music style identification;

[0062] Figure 7 This is a framework diagram of a neural network. Detailed Implementation

[0063] The following specific examples illustrate the implementation of the present invention. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and various details in this specification can be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of the present invention. Unless otherwise specified, the following embodiments and features can be combined with each other.

[0064] Please see Figures 1 to 7 This embodiment provides a vocal performance scoring system based on neural networks and audiovisual fusion, including a server, a camera, a microphone, a DSP decoder, and a communication network. The communication network is a wired network or a wireless network (such as Wi-Fi) for data interaction. The server is a cloud server or a physical server for processing data (i.e., executing the vocal performance scoring method based on neural networks and audiovisual fusion).

[0065] Figure 1 It is an overhead view of a vocal performance, including room size design, microphone array, camera deployment, and installation of sound-absorbing materials.

[0066] Figure 2 This is a flowchart of a vocal performance scoring method based on neural networks and audiovisual fusion. The method specifically includes the following steps:

[0067] Step 1: Divide the sight-singing practice data, which includes expert scores, collected in advance by the system into 7:2:1 parts. Use 7 parts as training data, 2 parts as validation data, and 1 part as test data. Use the training data to build a model.

[0068] Step 2: Preprocess the singing data by using intensity for boundary detection and noise reduction, using a bandpass filter for pre-filtering, using a high-pass filter for pre-emphasis of the speech signal, and finally performing frame-by-frame windowing on the singing data.

[0069] Step 3: Use the sum of harmonics (SHS) method to extract pitch features. Perform frame-by-frame windowing processing on the input audio. Perform short-time Fourier transform on the speech signal of each frame to obtain the spectrum. For each frequency, superimpose the energy of the harmonics to finally obtain the pitch of the singing data.

[0070] Step 4: Smooth the extracted pitch sequence.

[0071] Step 5: Use the similarity matching algorithm (DTW) to measure the similarity between the extracted pitch features and the standard pitch features, and obtain the minimum distance between the feature vectors of the two melodies.

[0072] Step 6: Mix the solo vocals and accompaniment music provided by the music standard library according to a certain time shift to synthesize the fast-paced, slow-paced, and synchronized audio suitable for training the rhythm model.

[0073] Step 7: Obtain the frequency intensity curve. First, perform audio preprocessing (pre-filtering, pre-emphasis, etc.), divide the audio into frames and add windows, obtain the signal spectrum through Fourier transform, and obtain the frequency energy curve by superimposing the energy of each frequency according to the idea of ​​harmonic summation method. Through scale conversion, use semitone values ​​to represent audio pitch to obtain the frequency intensity curve.

[0074] Step 8: Extract frequency intensity curves using standard accompanied singing, train and establish a rhythm model based on a hidden Markov model, with the frequency intensity sequence considered as an observable sequence.

[0075] Step 9: Use the Vibit algorithm to perform music model recognition and select the path with the highest cumulative probability as the recognition result.

[0076] Step 10: Combine the model recognition results of all audio segments to quantify the rhythmic fit of the entire song.

[0077] Step 11: Using the minimum distance of the melody feature vectors obtained in Step 5 and the rhythm fit obtained in Step 10 as input values, train the neural network using a supervised learning bp neural network based on expert scoring. The percentage score is the network output.

[0078] Step 12: Use Python's PyAV and Pillow libraries to extract keyframes from the performance video to obtain static images, and record the start and end times of the keyframes. Use the Viola-Jones detector in OpenCV to detect the face region from the static images and obtain face region images.

[0079] Step 13: Perform Gaussian filtering on each frame of the face region image, calculate the tilt angle based on the Hough transform, and finally correct the image based on the tilt angle.

[0080] Step 14: Convert and normalize the image to grayscale.

[0081] Step 15: Use the .dat model library of shape_predictor_68_face_landmarks.dat in the Dlib library to obtain information of 68 facial feature points.

[0082] Step 16: Identify the singer's emotional state by observing information such as the degree of mouth opening, eye squinting, and eyebrow raising. This can be categorized into eight emotional types: joyful, sorrowful, lyrical, passionate, angry, composed, resolute, and quiet, resulting in a facial emotional feature vector.

[0083] Step 17: In the audio after the experts have labeled it with emotion classification, divide it into 7 training data, 2 validation data and 1 test data in a ratio of 7:2:1.

[0084] Step 18: Construct a 3-layer BP neural network as an audio emotion classifier, and use Python's librosa library to extract audio feature vectors: pitch, frequency, and intensity.

[0085] The input layer consists of a three-dimensional vector of emotional features, namely pitch, frequency, and intensity. Therefore, the input layer has 3 nodes.

[0086] The BP neural network classifier outputs audio emotions, with eight musical emotion types: cheerful, sad, lyrical, passionate, angry, calm, resolute, and quiet. Therefore, the output layer has 3 nodes (log28 = 3). The eight emotions are represented as follows: cheerful (0, 0, 0), sad (0, 0, 1), lyrical (0, 1, 0), passionate (0, 1, 1), angry (1, 0, 0), calm (1, 0, 1), resolute (1, 1, 0), and quiet (1, 1, 1).

[0087] Step 181: According to Kolmogorov's theorem, design a network with 7 hidden nodes. The final network structure is 3×7×3, with 3 neurons in the input layer, 7 neurons in the hidden layer, and 3 neurons in the output layer.

[0088] Step 182: The Sigmoid function is used as the activation function for both the hidden and output layers. The maximum number of iterations is set to 2000, and the expected error is 0.001.

[0089] Step 19: After completing the construction of the BP neural network model, input the training data for training. After training, use validation data to validate the model, and it should meet the expected requirements.

[0090] Step 20: After the audio to be tested is extracted at the time recorded in step 16, it is put into the previously trained BP neural network to obtain the emotional feature vector of the audio.

[0091] Step 21: Use Python to calculate the Euclidean distance between the audio emotion feature vector and the facial emotion feature vector to measure their vector similarity.

[0092] Step 22: Input the feature vector similarity information into the expert scoring network, obtain the singing emotion score according to the expert-customized standard, and output it on a percentage scale.

[0093] Step 23: Extract a frame from the input video information as input for singer's clothing style recognition and analysis. Export the audio track content from the input video information as a .wav file as input for music style analysis.

[0094] Step 24: In the clothing style recognition and analysis, establish a mapping relationship between the singer's clothing image and the actual clothing style through image processing and machine learning techniques.

[0095] Step 25: In the image processing process, a convolutional neural network is used. In order to effectively reduce the complexity of the network and the number of parameters, the network mainly adopts the following steps: convolution, pooling, full connection and activation function to complete the construction of the neural network.

[0096] Step 26: In the clothing style analysis using machine learning, a well-trained convolutional neural network model is used to extract and format the pixels of the input image. Then, a series of operations such as convolution and pooling are performed on each image to obtain key feature points.

[0097] Step 27: Identify the singer's clothing style by analyzing different classification information such as fabric, pattern, partial embroidery, and bow tie / collar ornaments in the image. The styles can be categorized into 16 types: classic, ladylike, romantic, ethnic, avant-garde, lighthearted, preppy, casual, unisex, pastoral, punk, street, minimalist, sporty, elegant, and futuristic.

[0098] Step 28: In music style analysis, the audio information is preprocessed and Fourier transformed to obtain frequency domain information. The frequency domain information is then superimposed to obtain the Mel spectrum. A deep convolutional neural network is used to train and classify these samples to obtain the mapping relationship between the sung music and the singer's musical style.

[0099] Step 29: Input an optimized dataset into a deep convolutional neural network, use Fourier transform to transfer the audio data to the frequency domain, and derive the time-based evolution information of all frequencies of the song.

[0100] Step 30: Create fixed-length slices of the spectrum, reducing the vocal spectrum to 256×256 resolution slices as independent samples representing musical styles.

[0101] Step 31: Build a classifier by cutting the songs in the training set into square spectral images, which will form a dataset containing tens of thousands of samples for each category.

[0102] Step 32: The TFLearn wrapper from the Tensorflow deep learning library was used to train and classify these samples using a deep convolutional neural network.

[0103] Step 33: Using the deep convolutional neural network trained as described above, input music in .wav format. The sound frequencies of different music styles are different, and they can be divided into 11 music styles, including folk, bel canto, blues, classical, disco, hip hop, jazz, metal, pop, reggae, and rock.

[0104] Step 34: Input the output feature vectors of clothing style and music style into the expert scoring network, calculate the similarity of feature vectors according to the expert-customized standards, and obtain the matching score of the performer's clothing and music style, outputting it on a percentage scale.

[0105] Step 35: Based on steps 11, 22 and 34, input the percentage scores obtained from each network into the expert scoring network. This network is a supervised learning network that uses a traditional backpropagation network. It is based on the expert's overall performance score of the singer's vocal performance. The inputs are audio score, facial expression score and costume score, and the output is an overall performance percentage score.

[0106] like Figure 3As shown, the audiovisual fusion expert scoring network includes two processes: training and evaluation.

[0107] During the training process, the video samples of performers scored by experts were first divided into training set, validation set and test set according to a 7:2:1 ratio. Each sample contained a video and a score of 4 percentage points, in which the experts gave scores for emotion, clothing, music and overall evaluation respectively. For network training, (1) important parameters were selected according to the characteristics of the data, including convolution kernel (size, number), attention mechanism, activation function, loss function, number of network layers, network optimizer, learning rate, etc.; (2) according to the time axis synchronization, the feature vectors of the image and the feature vectors of the audio were jointly input into the evaluation neural network, and the difference between the network output score and the number of professional scores given by the experts was used as the loss function. The backpropagation algorithm was used to minimize the loss function; (3) a multi-dimensional expert evaluation network (including audio scoring neural network, audio-visual fusion emotion scoring neural network, audio-visual fusion clothing scoring neural network) and video The expert scoring network can be trained independently and synchronously. After training, it can be used together in groups. (4) By observing the loss function curves of the training set and the validation set, when the curve trends of the two sets turn, the training process is terminated, and the neural network parameters are determined by the number of iterations at this time, and the test data is used for testing. (5) The trained neural network is tested on the test set. If the difference between the score output by the network and the score given by the expert is within 1 point, the network has passed the test and can be used for the evaluation process of other performers in the future. Otherwise, the network is redesigned and adjusted, and the network training process is repeated.

[0108] The audiovisual fusion expert scoring network module mainly comprises three parts: signal preprocessing, a multidimensional expert evaluation network, and an expert score fitting network. Signal preprocessing separates the image and audio signals, resulting in image and audio files. The multidimensional expert evaluation network includes scoring networks for emotion, attire, and audio. Supervised learning is achieved by using expert scores for these three aspects as the basis for the network's evaluation, forming a multidimensional assessment of the performer. The expert score fitting network automatically calculates the weights of the three aspect scores on the overall score using a neural network.

[0109] During the evaluation process, the performer's video is input into a pre-trained audiovisual fusion expert scoring network, and the final score can be obtained directly.

[0110] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A vocal performance scoring method based on neural networks and audiovisual fusion, characterized in that, This method utilizes different neural networks to obtain scoring data across three dimensions: audio score, emotion score, and clothing score. These scores are then input into an audiovisual fusion expert scoring network to obtain the overall score. The specific steps of this method are as follows: S1: Obtain the performer's audio data, preprocess it to obtain high-pitched features and rhythm features, then compare it with standard audio data to obtain the minimum distance vector and rhythm fit, and input it into the audio scoring neural network to obtain the audio score; S2: Extract keyframes from the performance video, preprocess to obtain audio and face region images corresponding to the keyframes, extract pitch sequences and facial features from them, and input them into the audio emotion neural network and the facial keypoint detector to obtain audio emotion feature vectors and facial expression feature vectors. Then, calculate the Euclidean distance between the audio emotion feature vector and the facial emotion feature vector, i.e., feature vector similarity. Finally, input the feature vector similarity into the audiovisual fusion emotion scoring neural network to obtain the emotion score. S3: Extract a frame from the performance video, extract the key features of the clothing, and input them into the clothing style evaluation neural network to obtain the singer's clothing style image feature vector; then obtain the Mel spectrogram from the audio data of the corresponding video, and input it into the audio style evaluation neural network to obtain the singer's music style feature vector; finally, input the singer's clothing style feature vector and music style feature vector into the audiovisual fusion dressing score neural network to obtain the dressing score. S4: Input the audio score, emotion score, and clothing score into the audiovisual fusion expert scoring network to obtain the overall score.

2. The vocal performance scoring method according to claim 1, characterized in that, Step S1 specifically includes the following steps: S101: The video samples of performers scored by experts are divided into training set, validation set and test set according to a 7:2:1 ratio. S102: Preprocess the audio data, use intensity for boundary detection and noise reduction, use a bandpass filter for pre-filtering, use a high-pass filter for pre-emphasis of the speech signal, and finally perform frame segmentation and windowing on the audio data. S103: The pitch feature is extracted using the harmonic summation method. The input audio is processed by frame-by-frame windowing. The short-time Fourier transform of each frame of speech signal is performed to obtain the spectrum. For each frequency, the energy of the harmonic overlay is superimposed to finally obtain the pitch of the singing data. S104: Smooth the extracted pitch sequence; S105: Employ a similarity matching algorithm to measure the similarity between the extracted pitch features and the standard pitch features, and obtain the minimum distance between the feature vectors of the two melodies. S106: Mix the solo vocals and accompaniment music provided by the music standard library according to a certain time shift, and synthesize the fast beat, slow beat and synchronized beat audio suitable for training the rhythm model; S107: To obtain the frequency intensity curve, first, audio preprocessing is performed, the audio is divided into frames and windowed, and the signal spectrum is obtained through Fourier transform. According to the idea of ​​harmonic summation, the energy of each frequency is superimposed to obtain the frequency energy curve. Through scale conversion, the audio pitch is represented by semitone values ​​to obtain the frequency intensity curve. S108: Extract frequency intensity curves using standard accompanied singing, train and establish a rhythm model based on a hidden Markov model, with the frequency intensity sequence considered as an observable sequence. S109: The Vibit algorithm is used for music model recognition, and the path with the highest cumulative probability is selected as the recognition result; S110: Based on the model recognition results of all audio segments, quantify the rhythmic fit of the entire song. S111: Input the minimum distance of the melody feature vectors obtained in S105 and the rhythm fit obtained in S110 into the audio scoring neural network to train and obtain the audio score.

3. The vocal performance scoring method according to claim 1, characterized in that, Step S2 specifically includes the following steps: S201: Extract keyframes from video data to obtain static images, record the start and end times of keyframes, detect face regions from static images, and obtain face region images. S202: Perform Gaussian filtering on each frame of the face region image, calculate the tilt angle based on the Hough transform, and finally correct the image based on the tilt angle. S203: Perform grayscale conversion and normalization on the image; S204: Use a facial landmark detection model library to obtain facial feature points; S205: Recognize the singer's emotions to obtain facial emotion feature vectors; S206: In the audio after the experts have performed emotion classification and annotation, the data is divided according to the ratio of training data: validation data: test data = 7:2:1; S207: Construct an audio emotion extraction neural network as an audio emotion classifier, and extract audio feature vectors: pitch, frequency, and intensity as network input; S208: After completing the construction of the audio emotion extraction neural network, input training data for training; S209: After the audio to be tested is extracted at the time recorded in S205, it is put into the previously trained audio emotion extraction neural network to obtain the audio emotion feature vector. S210: Calculate the Euclidean distance between the audio emotion feature vector and the facial emotion feature vector to measure their vector similarity; S211: Input the feature vector similarity information into the audiovisual fusion emotion scoring neural network, and obtain the emotion score according to the expert-customized standard.

4. The vocal performance scoring method according to claim 1, characterized in that, Step S3 specifically includes the following steps: S301: Extract a frame of image content from the input video information as input information for singer's clothing style recognition and analysis, and export the audio track content from the input video information as .wav format as input information for music style analysis; S302: In clothing style recognition and analysis, image processing and machine learning techniques are used to establish a mapping relationship between the singer's clothing image and the actual clothing style; S303: In the process of image processing, a neural network for evaluating clothing style is used; S304: In the analysis of clothing style using machine learning, a trained clothing style evaluation neural network is used to extract and format the pixels of the input image, and then convolution and pooling are performed on each image to obtain key feature points. S305: Identify the singer's clothing style by using different classification information of the singer's clothing in the image; S306: In music style analysis, the audio information is preprocessed and Fourier transformed to obtain frequency domain information. The frequency domain information is then superimposed to obtain the Mel spectrum. A music style evaluation neural network is used to train and classify the samples to obtain the mapping relationship between the singing music and the performer's music style. S307: Input an optimized dataset into the music style evaluation neural network, use Fourier transform to transfer the audio data to the frequency domain, and derive the time-based evolution information of all frequencies of the song. S308: Creates fixed-length slices of the spectrum, reducing the vocal spectrum to 256×256 resolution slices as independent samples representing musical styles; S309: Build a classifier by cutting the songs in the training set into square spectral images, which will be used as a dataset containing tens of thousands of samples for each category; S310: Use a deep convolutional neural network to train and classify these samples; S311: Input .wav format music into the deep convolutional neural network trained by S310, and derive the music style based on the differences in sound frequencies of different music styles; S312: Input the output feature vectors of the performer's clothing style and music style into the audiovisual fusion dressing score neural network, calculate the similarity of the feature vectors according to the expert-customized standards, and obtain the matching score of the performer's clothing and music style, i.e., the dressing score.

5. The vocal performance scoring method according to claim 1, characterized in that, In step S4, the audiovisual fusion expert scoring network adopts an expert scoring mechanism. Experts score based on three dimensions: audio score, emotion score, and attire score, all on a 100-point scale. A dataset is constructed based on the scores of the three dimensions. The neural network learns the expert's scoring points through supervised learning. Furthermore, experts score the overall performance of the music performance, which is also on a 100-point scale. A dataset is constructed based on the scores given by the experts. The audiovisual fusion expert scoring network learns the weights of the experts' scores for the three dimensions.

6. The vocal performance scoring method according to claim 1, characterized in that, The audio scoring neural network, audiovisual fusion emotion scoring neural network, audiovisual fusion clothing scoring neural network, and audiovisual fusion expert scoring network can be trained independently and synchronously, and can be used together after training. During the training process, the video samples of performers scored by experts are first divided into training set, validation set, and test set according to a ratio of 7:2:

1. Each sample contains a video and four percentage scores, in which the scores are given by experts for emotion, clothing, music, and overall evaluation. For the training of each network, (1) important parameters are selected according to the characteristics of the data, including convolution kernel, attention mechanism, activation function, loss function, number of network layers, network optimizer, and learning rate; (2) according to the time axis. Step 1: Input the feature vectors of the image and the feature vectors of the audio into each scoring neural network. Use the difference between the network's output score and the number of professional scores given by the expert as the loss function. Use the backpropagation algorithm to minimize the loss function. (3) By observing the loss function curves of the training set and the validation set, when the curves of the two sets show a turning point, terminate the training process and determine the neural network parameters based on the number of iterations at this time. Test the network with test data. (4) Test the trained neural network on the test set. If the difference between the network's output score and the score given by the expert is within 1 point, the network has passed the test. Otherwise, redesign and adjust the network and repeat the network training process.

7. A vocal performance scoring system based on neural networks and audiovisual fusion, characterized in that, It includes a server, a camera, a microphone, a DSP decoder, and a communication network; the server is used to execute the vocal performance scoring method according to any one of claims 1 to 6.

Images