Multi-modal spoken language pronunciation evaluation method based on large language model

Abstract

The invention provides a multi-modal spoken language pronunciation evaluation method based on a large language model, and belongs to the technical field of spoken language evaluation, and the method comprises the steps: obtaining the audio data of a spoken language tester; performing feature coding processing on the audio data through an acoustic encoder to obtain audio representation information, and mapping the obtained audio representation information into a public space through a modal adaptation layer; setting an audio evaluation instruction, separating the audio evaluation instruction into a plurality of phrases through a word segmentation device, converting the phrases into high-dimensional text features through a word embedding module, and mapping the high-dimensional text features into a public space; splicing the audio representation information in the public space with the high-dimensional text features, carrying out multi-modal feature combination, and constructing an evaluation sequence; and the big language model carries out evaluation prediction on the evaluation sequence to obtain a corresponding pronunciation accuracy evaluation result and a fluency evaluation result. The method does not need to depend on an automatic voice recognition system, and the evaluation efficiency of the audio data is greatly improved while the evaluation process is simplified.

Description

Technical Field

[0001] The present invention relates to the technical field of spoken language assessment, and in particular to a multimodal spoken language pronunciation assessment method based on a large language model. Background Art

[0002] Speaking is an important part of learning a foreign language. Fluent and accurate pronunciation can effectively improve the communication and exchange of information.

[0003] In traditional technology, the evaluation of oral pronunciation is based on an automatic speech recognition system. The automatic speech recognition system obtains feature information based on the time series of the oral tester's speech data and pronunciation text, and builds a phoneme-level scoring network to evaluate the pronunciation of the oral tester. The above method is heavily dependent on the automatic speech recognition system, and the evaluation based on the phoneme level leads to a slow evaluation process and cumbersome steps to obtain the evaluation results.

[0004] Therefore, a multimodal spoken pronunciation evaluation method based on a large language model is proposed. Summary of the invention

[0005] In order to solve the above technical problems, the present invention provides a multimodal spoken pronunciation evaluation method based on a large language model, which is used to solve the problems of slow oral evaluation process and complicated steps in traditional technologies.

[0006] An embodiment of the present invention provides a multimodal spoken pronunciation evaluation method based on a large language model, comprising:

[0007] Obtain audio data of oral testers;

[0008] Performing feature encoding processing on the audio data through an acoustic encoder to obtain audio representation information, and mapping the obtained audio representation information into a common space through a modal adaptation layer;

[0009] Setting an audio evaluation instruction, separating the audio evaluation instruction into a plurality of phrases by a word segmenter, and converting the phrases into high-dimensional text features by a word embedding module, and mapping them into a common space;

[0010] The audio representation information in the public space is spliced ​​with the high-dimensional text features to merge multimodal features and construct an evaluation sequence;

[0011] The large language model obtains corresponding pronunciation accuracy evaluation results and fluency evaluation results by evaluating and predicting the evaluation sequence.

[0012] Preferably, an embodiment of the present invention provides a multimodal oral pronunciation evaluation method based on a large language model, the steps of: obtaining audio data of an oral tester; comprising:

[0013] Get the original audio signal;

[0014] The high frequency part of the original audio signal is pre-emphasized by a pre-emphasis filter to obtain a pre-emphasized audio signal; the pre-emphasized audio signal is framed by using the duration of the preset sampling point as the frame length, and each speech frame signal is windowed by using a window function to obtain a pre-processed audio signal;

[0015] Calculating the short-time energy value of the speech frame signal in the preprocessed audio signal, and constructing an energy value sequence corresponding to the preprocessed audio signal;

[0016]

[0017] Among them, E t is the short-time energy value of the speech frame signal s(t) at time t in the preprocessed audio signal, s(t+d) is the frame length offset signal of the speech frame signal s(t), d is the frame length variable, l is the frame length of the speech frame, and ω(d) is the window function corresponding to the speech frame signal s(t);

[0018] The method further comprises: setting a first decision threshold and a second decision threshold respectively according to the energy value sequence, performing speech endpoint detection on the preprocessed audio signal based on the energy value sequence, and acquiring audio data; optionally, comprising:

[0019] When the short-time energy value is lower than the first decision threshold, the speech frame signal corresponding to the short-time energy value is marked as a silent point; when the short-time energy value is higher than the first decision threshold and lower than the second decision threshold, the speech frame signal corresponding to the short-time energy value is marked as a transition point; when the short-time energy value is higher than the second decision threshold, the speech frame signal corresponding to the short-time energy value is marked as a speech point;

[0020] Analyze the speech frame signal according to the marking information, detect the duration of the speech point and the transition point, and identify the speech frame signal as noise when the duration is less than the shortest speech time; and identify the speech frame signal as valid speech when the duration is greater than the shortest speech time;

[0021] The noise in the preprocessed audio signal and the speech frame signal corresponding to the silence point are filtered out to obtain audio data.

[0022] Preferably, an embodiment of the present invention provides a multimodal spoken pronunciation evaluation method based on a large language model, the steps of: performing feature encoding processing on the audio data through an acoustic encoder to obtain audio representation information; and mapping the obtained audio representation information into a common space through a modal adaptation layer; comprising:

[0023] Constructing an initial acoustic encoder, and training the initial acoustic encoder using an automatic speech recognition data set to obtain an acoustic encoder;

[0024] Analyzing the global correlation features and hidden features of the audio data through an acoustic encoder to obtain audio representation information;

[0025] Constructing an initial modal adaptation layer, and training the initial modal adaptation layer through an automatic speech recognition data set to obtain a modal adaptation layer;

[0026] The modal features of the vectors in the audio representation information are analyzed through the modal adaptation layer, and the deep audio feature vectors, hidden audio feature vectors and high-dimensional feature vectors in the audio representation information are mapped into a common space through the corresponding modal feature extractors.

[0027] Preferably, an embodiment of the present invention provides a multimodal spoken pronunciation evaluation method based on a large language model, the steps of: constructing an initial modal adaptation layer, training the initial modal adaptation layer through an automatic speech recognition data set, and obtaining the modal adaptation layer; comprising:

[0028] Obtain sample representation information corresponding to sample speech data in an automatic speech recognition dataset;

[0029] Performing modality recognition on the sample representation information through an initial modality adaptation layer and processing it through a modality feature extractor to obtain sample mapping information;

[0030] Evaluating the mapping loss of the modality feature extractor by a cross entropy function;

[0031]

[0032] in, is the mapping loss function of the modal feature extractor of modality m, L v Represent information X for the vth sample v The corresponding sample label, softmax() is the normalized exponential function, T m is the modal feature extractor of modality m, V is the number of samples of sample representation information in the automatic speech recognition dataset;

[0033] Analyzing the similarity between the sample mapping information and the standard mapping information corresponding to the sample representation information, and evaluating the learning loss of the modal feature extractor;

[0034]

[0035] in, is the learning loss function of the modal feature extractor of modality m, T m (Xv ) is the modal feature extractor T m Characterize the sample information X v The sample mapping information to be processed, X is the sample representation information v Corresponding standard mapping information, θ is the scale factor, || ||2 is the 2-norm;

[0036] A comprehensive loss function is constructed according to the mapping loss and learning loss of the modal feature extractor; the comprehensive loss function is optimized according to the automatic speech recognition data set to obtain a modal adaptation layer.

[0037] Preferably, an embodiment of the present invention provides a multimodal spoken pronunciation evaluation method based on a large language model, wherein the large language model evaluates and predicts the evaluation sequence to obtain corresponding pronunciation accuracy evaluation results and fluency evaluation results, including:

[0038] Get the oral scoring dataset;

[0039] Dividing the oral scoring data set into an oral scoring training set and an oral scoring test set according to a preset ratio;

[0040] Build an initial large language model;

[0041] The initial large language model is trained by using the oral scoring training set, the control parameters of the initial large language model are optimized, and the evaluation accuracy of the initial large language model is analyzed by using the test set, and when the preset evaluation accuracy is met, the large language model is obtained;

[0042] The large language model obtains the evaluation task by reading the high-dimensional text features in the evaluation sequence, performs evaluation prediction on the evaluation sequence, and obtains the pronunciation accuracy evaluation results and the fluency evaluation results.

[0043] Preferably, an embodiment of the present invention provides a multimodal spoken pronunciation evaluation method based on a large language model, the steps of: training an initial large language model through a spoken language scoring training set, optimizing control parameters of the initial large language model, and analyzing the evaluation accuracy of the initial large language model through a test set, and obtaining a large language model when a preset evaluation accuracy is met; comprising:

[0044] Obtain oral training samples and corresponding standard evaluation results in the oral scoring training set;

[0045] The initial large language model performs a random masking operation on the spoken language training sample, inputs the unmasked portion of the spoken language training sample into the Transformer layer, obtains spoken language sample features, and obtains sample evaluation results through the decision maker;

[0046] Analyzing the error between the sample evaluation result and the standard evaluation result by evaluating the loss function;

[0047]

[0048] Among them, L e is the evaluation loss function, D is the number of feature blocks in the spoken language training sample, and D m is the number of masked feature blocks in the spoken training sample, R' dk is the sample evaluation value obtained by the initial large language model based on the d-th feature block and the masked k-th feature block in the spoken training sample, sg() is the stop gradient operation, R dk is the sample evaluation value obtained by the initial large language model based on the d-th feature block and the unmasked k-th feature block in the spoken training sample, and R is the standard evaluation result;

[0049] The loss value obtained by the evaluation loss function is reduced by adjusting the control parameters of the initial large language model until the training of all spoken language training samples in the spoken language scoring training set is completed.

[0050] Preferably, an embodiment of the present invention provides a multimodal spoken pronunciation evaluation method based on a large language model, comprising:

[0051] The oral training samples in the oral scoring training set are obtained by actually recording the oral speech data of the oral test takers; the standard evaluation results are sample pronunciation accuracy evaluation results and sample fluency evaluation results obtained by manual evaluation based on the oral training samples.

[0052] Compared with traditional technologies, the beneficial effects of the present invention are: a multimodal oral pronunciation evaluation method based on a large language model, which does not rely on an automatic speech recognition system, performs feature encoding on audio data based on an acoustic encoder, merges audio representation information with high-dimensional text features for multimodal feature, and constructs an evaluation sequence. The pronunciation accuracy and fluency of audio data can be evaluated through a large language model. Compared with the phoneme-level scoring grid in traditional technologies, the evaluation process is simplified while the evaluation efficiency of audio data is greatly improved. In addition, the above method does not require forced speech alignment, thereby effectively improving the accuracy of the evaluation results.

[0053] Other features and advantages of the present invention will be described in the following description, and partly become apparent from the description, or understood by practicing the present invention. The purpose and other advantages of the present invention can be realized and obtained by the structures particularly pointed out in the written description, claims, and drawings.

[0054] The technical solution of the present invention is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] Figure 1 A schematic diagram of the flow of a multimodal spoken pronunciation evaluation method based on a large language model provided by the present invention. DETAILED DESCRIPTION

[0056] The preferred embodiments of the present invention are described below in conjunction with the accompanying drawings. It should be understood that the preferred embodiments described herein are only used to illustrate and explain the present invention, and are not used to limit the present invention.

[0057] Embodiment 1:

[0058] The embodiment of the present invention provides a multimodal spoken pronunciation evaluation method based on a large language model, referring to Figure 1 ,include:

[0059] Obtain audio data of oral testers;

[0060] The audio data is feature encoded by an acoustic encoder to obtain audio representation information, and the obtained audio representation information is mapped to a common space by a modal adaptation layer;

[0061] Set audio evaluation instructions, separate the audio evaluation instructions into several phrases through a word segmenter, and convert the phrases into high-dimensional text features through a word embedding module and map them into a common space;

[0062] The audio representation information in the public space is spliced ​​with high-dimensional text features to merge multimodal features and construct an evaluation sequence;

[0063] The large language model evaluates and predicts the test sequence to obtain the corresponding pronunciation accuracy and fluency test results.

[0064] In the above embodiments, the audio data of the oral tester is feature encoded through an acoustic encoder to obtain audio representation information, which is mapped to a public space through a modal adaptation layer; and an audio evaluation instruction is set, the audio evaluation instruction is separated into several phrases through a word segmenter, and the phrases are converted into high-dimensional text features through a word embedding module and mapped to a public space; the audio representation information in the public space is spliced ​​with the high-dimensional text features, multi-modal features are merged, and an evaluation sequence is constructed; the large language model obtains corresponding pronunciation accuracy evaluation results and fluency evaluation results by evaluating and predicting the evaluation sequence.

[0065] In the above embodiments, the special task character "Assessment" can be set as an audio assessment instruction, and the audio assessment instruction can be separated into several phrases through the Tokenizer. The word embedding module can be implemented as an Embedding layer to convert the phrases into high-dimensional text features and map them to the common space.

[0066] In the above embodiments, the audio representation information in the public space is the speech modal feature, the high-dimensional text feature is the text modal feature, and the multi-modal features are merged to construct the evaluation sequence.

[0067] The beneficial effects of the above technology are: feature encoding processing is performed on audio data through an acoustic encoder, and it is mapped to a public space through a modal adaptation layer, the audio representation information in the public space is spliced ​​with high-dimensional text features, multi-modal features are merged, an evaluation sequence is constructed, and the corresponding pronunciation accuracy evaluation results and fluency evaluation results are obtained through a large language model, and the pronunciation situation is fed back to the oral tester; compared with traditional technologies, the above method does not need to rely on an automatic speech recognition system, and feature encoding of audio data is performed based on an acoustic encoder to construct an evaluation sequence. The pronunciation accuracy and fluency of audio data can be evaluated through a large language model. Compared with the phoneme-level scoring grid in traditional technologies, while simplifying the evaluation process, the evaluation efficiency of audio data is greatly improved, and the above method does not require forced speech alignment, thereby effectively improving the accuracy of the evaluation results.

[0068] Embodiment 2:

[0069] The embodiment of the present invention provides a multimodal oral pronunciation evaluation method based on a large language model, the steps of: obtaining audio data of an oral tester; comprising:

[0070] Get the original audio signal;

[0071] The high frequency part of the original audio signal is pre-emphasized by a pre-emphasis filter to obtain a pre-emphasized audio signal; the pre-emphasized audio signal is framed by using the duration of the preset sampling point as the frame length, and each speech frame signal is windowed by using a window function to obtain a pre-processed audio signal;

[0072] Calculate the short-time energy value of the speech frame signal in the preprocessed audio signal and construct an energy value sequence corresponding to the preprocessed audio signal;

[0073]

[0074] Among them, E tis the short-time energy value of the speech frame signal s(t) at time t in the preprocessed audio signal, s(t+d) is the frame length offset signal of the speech frame signal s(t), d is the frame length variable, l is the frame length of the speech frame, and ω(d) is the window function corresponding to the speech frame signal s(t);

[0075] The method further comprises: setting a first decision threshold and a second decision threshold respectively according to the energy value sequence, performing speech endpoint detection on the preprocessed audio signal based on the energy value sequence, and acquiring audio data; optionally, comprising:

[0076] When the short-time energy value is lower than the first decision threshold, the speech frame signal corresponding to the short-time energy value is marked as a silent point; when the short-time energy value is higher than the first decision threshold and lower than the second decision threshold, the speech frame signal corresponding to the short-time energy value is marked as a transition point; when the short-time energy value is higher than the second decision threshold, the speech frame signal corresponding to the short-time energy value is marked as a speech point;

[0077] Analyze the speech frame signal according to the marking information, detect the duration of speech points and transition points, and identify the speech frame signal as noise when the duration is less than the shortest speech time; and identify the speech frame signal as valid speech when the duration is greater than the shortest speech time;

[0078] The noise and speech frame signals corresponding to the silent points in the preprocessed audio signal are filtered out to obtain audio data.

[0079] In the above embodiments, the high frequency part of the original audio signal is pre-emphasized by a pre-emphasis filter to obtain a pre-emphasized audio signal; the pre-emphasized audio signal is framed by using the duration of the preset sampling point as the frame length, and each speech frame signal is windowed by using a window function to obtain a pre-processed audio signal.

[0080] The beneficial effects of the above technology are: by processing the original audio signal through the pre-emphasis filter, the resolution of the high-frequency part of the signal can be effectively improved, and the influence of lip radiation can be filtered out; by performing frame processing on the pre-emphasized audio signal, the pre-emphasized audio signal can be converted into a smooth signal, which is convenient for the subsequent steps to calculate the short-time energy value of the speech frame signal. Since there will be a signal interval between adjacent frames after the frame processing, the truncation effect of the speech frame signal is effectively reduced through windowing processing with a window function.

[0081] In the above embodiments, an energy value sequence is constructed by calculating the short-time energy value of the speech frame signal in the preprocessed audio signal, and a first decision threshold and a second decision threshold are set to perform speech endpoint detection on the preprocessed audio signal to obtain audio data.

[0082] In the above embodiments, silence points, transition points and speech points are detected based on the short-time energy value of the speech frame signal based on the first decision threshold and the second decision threshold, and when the duration of the speech point and the transition point is less than the shortest speech time, the speech frame signal is identified as noise and filtered out; when the duration is greater than the shortest speech time, the speech frame signal is identified as valid speech and retained, and the speech frame signal corresponding to the silence point is filtered out.

[0083] The beneficial effects of the above technology are as follows: by analyzing the short-time energy value of the speech frame signal, the silent signal and the noise signal in the pre-processed audio signal can be screened out, and while obtaining the audio data, it is convenient for the subsequent steps to extract and evaluate the features of the audio data.

[0084] Embodiment 3:

[0085] The embodiment of the present invention provides a multimodal spoken pronunciation evaluation method based on a large language model, comprising the steps of: performing feature encoding processing on audio data through an acoustic encoder to obtain audio representation information; and mapping the obtained audio representation information to a common space through a modal adaptation layer; comprising:

[0086] Construct an initial acoustic encoder, train the initial acoustic encoder using an automatic speech recognition dataset, and obtain an acoustic encoder;

[0087] The global correlation features and hidden features of the audio data are analyzed through an acoustic encoder to obtain audio representation information;

[0088] Construct an initial modal adaptation layer, train the initial modal adaptation layer using an automatic speech recognition dataset, and obtain a modal adaptation layer;

[0089] Through the modal adaptation layer, the modal features of the vectors in the audio representation information are analyzed, and the deep audio feature vectors, hidden audio feature vectors and high-dimensional feature vectors in the audio representation information are mapped into a common space through the corresponding modal feature extractors.

[0090] In the above embodiments, the acoustic encoder encodes the audio feature information to obtain the audio representation information, and analyzes the modal features of the vectors in the audio representation information through the modal adaptation layer to map the audio representation information into a common space.

[0091] In the above embodiments, the initial acoustic encoder and the initial modal adaptation layer are trained by using the automatic speech recognition data set to obtain the corresponding acoustic encoder and modal adaptation layer.

[0092] In the above embodiment, the automatic speech recognition data set can be implemented as 1.4w hours of English automatic speech recognition data.

[0093] In the above embodiments, the acoustic encoder may call an existing encoder model to perform feature encoding processing on the audio data.

[0094] In one embodiment, the acoustic encoder may be built based on the Whisper model, and the acquired audio representation information is Mel-spectrogram feature information.

[0095] In one embodiment, the acoustic encoder may be constructed based on the Hubert model, and the acquired audio representation information is filter bank feature information.

[0096] In one embodiment, the acoustic encoder can be built based on the Data2vec2.0 model, and the acquired audio representation information is Mel-spectrogram feature information.

[0097] The beneficial effects of the above technology are: feature extraction of audio feature information is performed through the acoustic encoder, thereby obtaining audio representation information, and mapping the audio representation information to the common space through the modal adaptation layer, thereby constructing an evaluation sequence for oral scoring.

[0098] Embodiment 4:

[0099] The embodiment of the present invention provides a multimodal spoken pronunciation evaluation method based on a large language model, comprising the steps of: constructing an initial modal adaptation layer, training the initial modal adaptation layer through an automatic speech recognition data set, and obtaining the modal adaptation layer; comprising:

[0100] Obtain sample representation information corresponding to sample speech data in an automatic speech recognition dataset;

[0101] The sample representation information is modally identified through the initial modal adaptation layer and processed through the modal feature extractor to obtain sample mapping information;

[0102] The mapping loss of the modality feature extractor is evaluated by the cross entropy function;

[0103]

[0104] in, is the mapping loss function of the modal feature extractor of modality m, L v Represents the information X for the vth sample v The corresponding sample label, softmax() is the normalized exponential function, T m is the modal feature extractor of modality m, V is the number of samples of sample representation information in the automatic speech recognition dataset;

[0105] Analyze the similarity between the sample mapping information and the standard mapping information corresponding to the sample representation information, and evaluate the learning loss of the modal feature extractor;

[0106]

[0107] in, is the learning loss function of the modal feature extractor of modality m, T m (X v ) is the modal feature extractor T m Characterize the sample information X v The sample mapping information to be processed, X is the sample representation information v Corresponding standard mapping information, θ is the scale factor, || ||2 is the 2-norm;

[0108] According to the mapping loss and learning loss of the modal feature extractor, a comprehensive loss function is constructed; according to the automatic speech recognition dataset, the comprehensive loss function is optimized to obtain the modal adaptation layer.

[0109] In the above embodiments, the initial modal adaptation layer performs modal identification on the sample representation information, selects the corresponding modal feature extractor for processing, and obtains the sample mapping information; the mapping loss and learning loss of the modal feature extractor are evaluated by the cross entropy function and the learning loss function respectively, and a comprehensive loss function is constructed to optimize the initial modal adaptation layer and obtain the modal adaptation layer.

[0110] In the above embodiments, the mapping loss of the modal feature extractor is obtained by using the cross entropy function, that is, the feature extraction performance of the modal feature extractor is optimized.

[0111] In the above embodiments, the similarity between the sample mapping information and the standard mapping information corresponding to the sample representation information is analyzed by learning the loss function, thereby optimizing the learning performance of the modal feature extractor on the sample speech data.

[0112] In the above embodiments, the initial modal adaptation layer is trained according to the sample representation information to reduce the comprehensive loss function and obtain the modal adaptation layer.

[0113] The beneficial effects of the above technology are: the initial modal adaptation layer is trained according to the sample representation information, the control parameters of the initial modal adaptation layer are optimized, and the acquired modal adaptation layer can extract features according to the audio representation information and map them to the common space.

[0114] Embodiment 5:

[0115] The embodiment of the present invention provides a multimodal spoken pronunciation evaluation method based on a large language model, the steps of: the large language model evaluates and predicts the evaluation sequence to obtain corresponding pronunciation accuracy evaluation results and fluency evaluation results, including:

[0116] Get the oral scoring dataset;

[0117] According to the preset ratio, the oral scoring data set is divided into an oral scoring training set and an oral scoring test set;

[0118] Build an initial large language model;

[0119] The initial large language model is trained through the oral scoring training set, the control parameters of the initial large language model are optimized, and the evaluation accuracy of the initial large language model is analyzed through the test set. When the preset evaluation accuracy is met, the large language model is obtained;

[0120] The large language model obtains the evaluation task by reading the high-dimensional text features in the evaluation sequence, performs evaluation prediction on the evaluation sequence, and obtains the pronunciation accuracy evaluation results and fluency evaluation results.

[0121] In the above embodiment, the oral scoring data set may be implemented as including 5000 oral scoring data of oral test takers, of which 2500 are used as the oral scoring training set and 2500 are used as the oral scoring test set.

[0122] In the above embodiment, the initial large language model is trained through the oral scoring training set, the control parameters of the initial large language model are optimized, and the evaluation accuracy of the initial large language model is analyzed through the test set. When the preset evaluation accuracy is met, the large language model is obtained.

[0123] In the above embodiments, the large language model reads the high-dimensional text features in the evaluation sequence, performs speech evaluation on the evaluation sequence, randomly masks the audio representation information in the evaluation sequence, performs feature prediction through the Transformer layer, obtains the audio feature information, and obtains the corresponding pronunciation accuracy evaluation results and fluency evaluation results through the decision maker.

[0124] The beneficial effects of the above technology are: constructing an initial large language model, and training and testing the model through a spoken language scoring training set and a spoken language scoring test set, thereby realizing the acquisition of a large language model; the large language model processes the evaluation sequence to realize the acquisition of pronunciation accuracy evaluation results and fluency evaluation results.

[0125] Embodiment 6:

[0126] The embodiment of the present invention provides a multimodal spoken pronunciation evaluation method based on a large language model, comprising the steps of: training an initial large language model through a spoken language scoring training set, optimizing control parameters of the initial large language model, and analyzing the evaluation accuracy of the initial large language model through a test set, and obtaining the large language model when a preset evaluation accuracy is met; comprising:

[0127] Obtain oral training samples and corresponding standard evaluation results in the oral scoring training set;

[0128] The initial large language model performs random masking operations on the spoken training samples, inputs the unmasked parts of the spoken training samples into the Transformer layer, obtains the spoken sample features, and obtains the sample evaluation results through the decision maker;

[0129] By evaluating the loss function, the error between the sample evaluation results and the standard evaluation results is analyzed;

[0130]

[0131] Among them, L e is the evaluation loss function, D is the number of feature blocks in the spoken language training sample, and D m is the number of masked feature blocks in the spoken training sample, R' dk is the sample evaluation value obtained by the initial large language model based on the d-th feature block and the masked k-th feature block in the spoken training sample, sg() is the stop gradient operation, R dk is the sample evaluation value obtained by the initial large language model based on the d-th feature block and the unmasked k-th feature block in the spoken training sample, and R is the standard evaluation result;

[0132] By adjusting the control parameters of the initial large language model, the loss value obtained by the evaluation loss function is reduced until the training of all spoken training samples in the spoken scoring training set is completed.

[0133] In the above embodiment, the initial large language model performs a random masking operation on the spoken training samples in the spoken scoring training set, inputs the unmasked part of the spoken training samples into the Transformer layer for feature prediction, obtains the spoken sample features, and obtains the sample evaluation results through the decision maker; through the evaluation loss function, analyzes the error between the sample evaluation results and the standard evaluation results, and optimizes the control parameters of the initial large language model to reduce the loss value obtained by the evaluation loss function, until the training of all spoken training samples in the spoken scoring training set is completed.

[0134] In the above embodiment, the evaluation loss function calculates the loss value of the spoken training sample by analyzing the error between the sample evaluation value corresponding to each feature block in the spoken training sample and the standard evaluation result, and stops the gradient operation through sg(), which effectively avoids overfitting of the initial model and causes model collapse.

[0135] The beneficial effect of the above technology is that the spoken language training samples are processed by the initial large language model to obtain the sample evaluation results, and the errors with the standard evaluation results are analyzed by the evaluation loss function to optimize the control parameters of the initial large language model, thereby realizing the training of the initial large language model.

[0136] Embodiment 7:

[0137] The embodiment of the present invention provides a multimodal spoken pronunciation evaluation method based on a large language model, comprising:

[0138] The oral training samples in the oral scoring training set are obtained by actually recording the oral voice data of the oral test takers; the standard evaluation results are the sample pronunciation accuracy evaluation results and sample fluency evaluation results obtained by manual evaluation based on the oral training samples.

[0139] In the above embodiments, the oral training samples are oral speech data of oral test takers, and the corresponding standard evaluation results are obtained by manually annotating the sample pronunciation accuracy evaluation results and the sample fluency evaluation results.

[0140] The beneficial effect of the above technology is that by actually recording spoken voice data and manually annotating the standard evaluation results, it is possible to build a spoken scoring training set, and then to train the initial large language model.

[0141] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.

Claims

1. A multimodal spoken pronunciation evaluation method based on a large language model, characterized in that: include: Obtain audio data of oral test takers; Performing feature encoding processing on the audio data through an acoustic encoder to obtain audio representation information, and mapping the obtained audio representation information into a common space through a modal adaptation layer; Setting an audio evaluation instruction, separating the audio evaluation instruction into a plurality of phrases by a word segmenter, and converting the phrases into high-dimensional text features by a word embedding module, and mapping them into a common space; The audio representation information in the public space is spliced ​​with the high-dimensional text features to merge multimodal features and construct an evaluation sequence; The large language model obtains corresponding pronunciation accuracy evaluation results and fluency evaluation results by evaluating and predicting the evaluation sequence.

2. The multimodal spoken pronunciation evaluation method based on a large language model according to claim 1, characterized in that: The step of obtaining the audio data of the oral tester includes: Get the original audio signal; The high frequency part of the original audio signal is pre-emphasized by a pre-emphasis filter to obtain a pre-emphasized audio signal; the pre-emphasized audio signal is framed by using the duration of the preset sampling point as the frame length, and each speech frame signal is windowed by using a window function to obtain a pre-processed audio signal; Calculating the short-time energy value of the speech frame signal in the preprocessed audio signal, and constructing an energy value sequence corresponding to the preprocessed audio signal; Among them, E t is the short-time energy value of the speech frame signal s(t) at time t in the preprocessed audio signal, s(t+d) is the frame length offset signal of the speech frame signal s(t), d is the frame length variable, l is the frame length of the speech frame, and ω(d) is the window function corresponding to the speech frame signal s(t); The method further comprises: setting a first decision threshold and a second decision threshold respectively according to the energy value sequence, performing speech endpoint detection on the preprocessed audio signal based on the energy value sequence, and acquiring audio data; optionally, comprising: When the short-time energy value is lower than the first decision threshold, the speech frame signal corresponding to the short-time energy value is marked as a silent point; when the short-time energy value is higher than the first decision threshold and lower than the second decision threshold, the speech frame signal corresponding to the short-time energy value is marked as a transition point; when the short-time energy value is higher than the second decision threshold, the speech frame signal corresponding to the short-time energy value is marked as a speech point; Analyze the speech frame signal according to the marking information, detect the duration of the speech point and the transition point, and identify the speech frame signal as noise when the duration is less than the shortest speech time; and identify the speech frame signal as valid speech when the duration is greater than the shortest speech time; The noise in the preprocessed audio signal and the speech frame signal corresponding to the silence point are filtered out to obtain audio data.

3. The multimodal spoken pronunciation evaluation method based on a large language model according to claim 1, characterized in that: The step of: performing feature encoding processing on the audio data through an acoustic encoder to obtain audio representation information; and mapping the obtained audio representation information to a common space through a modal adaptation layer; comprises: Constructing an initial acoustic encoder, and training the initial acoustic encoder using an automatic speech recognition data set to obtain an acoustic encoder; Analyzing the global correlation features and hidden features of the audio data through an acoustic encoder to obtain audio representation information; Constructing an initial modal adaptation layer, and training the initial modal adaptation layer through an automatic speech recognition data set to obtain a modal adaptation layer; The modal features of the vectors in the audio representation information are analyzed through the modal adaptation layer, and the deep audio feature vectors, hidden audio feature vectors and high-dimensional feature vectors in the audio representation information are mapped into a common space through the corresponding modal feature extractors.

4. The multimodal spoken pronunciation evaluation method based on a large language model according to claim 3, characterized in that: The step of constructing an initial modal adaptation layer, training the initial modal adaptation layer through an automatic speech recognition data set, and obtaining the modal adaptation layer comprises: Obtain sample representation information corresponding to sample speech data in an automatic speech recognition dataset; Performing modality recognition on the sample representation information through an initial modality adaptation layer and processing it through a modality feature extractor to obtain sample mapping information; Evaluating the mapping loss of the modality feature extractor by a cross entropy function; in, is the mapping loss function of the modal feature extractor of modality m, L v Represents the information X for the vth sample v The corresponding sample label, softmax() is the normalized exponential function, T m is the modal feature extractor of modality m, V is the number of samples of sample representation information in the automatic speech recognition dataset; Analyzing the similarity between the sample mapping information and the standard mapping information corresponding to the sample representation information, and evaluating the learning loss of the modal feature extractor; in, is the learning loss function of the modal feature extractor of modality m, T m (X v ) is the modal feature extractor T m Characterize the sample information X v The sample mapping information to be processed, X is the sample representation information v Corresponding standard mapping information, θ is the scale factor, || ||2 is the 2-norm; A comprehensive loss function is constructed according to the mapping loss and learning loss of the modal feature extractor; the comprehensive loss function is optimized according to the automatic speech recognition data set to obtain a modal adaptation layer.

5. The multimodal spoken pronunciation evaluation method based on a large language model according to claim 1, characterized in that: The step: the large language model evaluates and predicts the evaluation sequence to obtain corresponding pronunciation accuracy evaluation results and fluency evaluation results, including: Get the oral scoring dataset; Dividing the oral scoring data set into an oral scoring training set and an oral scoring test set according to a preset ratio; Build an initial large language model; The initial large language model is trained by using the oral scoring training set, the control parameters of the initial large language model are optimized, and the evaluation accuracy of the initial large language model is analyzed by using the test set, and when the preset evaluation accuracy is met, the large language model is obtained; The large language model obtains the evaluation task by reading the high-dimensional text features in the evaluation sequence, performs evaluation prediction on the evaluation sequence, and obtains the pronunciation accuracy evaluation results and the fluency evaluation results.

6. The multimodal spoken pronunciation evaluation method based on a large language model according to claim 5, characterized in that: The step of training the initial large language model through the oral scoring training set, optimizing the control parameters of the initial large language model, and analyzing the evaluation accuracy of the initial large language model through the test set, and obtaining the large language model when the preset evaluation accuracy is met; includes: Obtain oral training samples and corresponding standard evaluation results in the oral scoring training set; The initial large language model performs a random masking operation on the spoken language training sample, inputs the unmasked portion of the spoken language training sample into the Transformer layer, obtains spoken language sample features, and obtains sample evaluation results through the decision maker; Analyzing the error between the sample evaluation result and the standard evaluation result by evaluating the loss function; Among them, L e is the evaluation loss function, D is the number of feature blocks in the spoken language training sample, and D m is the number of masked feature blocks in the spoken training sample, R' dk is the sample evaluation value obtained by the initial large language model based on the d-th feature block and the masked k-th feature block in the spoken training sample, sg() is the stop gradient operation, R dk is the sample evaluation value obtained by the initial large language model based on the d-th feature block and the unmasked k-th feature block in the spoken training sample, and R is the standard evaluation result; The loss value obtained by the evaluation loss function is reduced by adjusting the control parameters of the initial large language model until the training of all spoken language training samples in the spoken language scoring training set is completed.

7. The multimodal spoken pronunciation evaluation method based on a large language model according to claim 6, characterized in that: include: The oral training samples in the oral scoring training set are obtained by actually recording the oral speech data of the oral tester; The standard evaluation results are sample pronunciation accuracy evaluation results and sample fluency evaluation results obtained by manually evaluating the oral training samples.

Images