Open scenario english speech evaluation method and system based on semantic perception

Abstract

The application discloses a semantic perception-based open-scene English speech evaluation method and system, relates to the technical field of speech evaluation, and comprises the following steps: performing feature extraction on audio data to obtain frame-level acoustic features; performing recognition on the audio data to obtain transcription text; inputting the transcription text and the subject text corresponding to the audio data into a language model for processing to obtain word-level representation of the transcription text and word-level representation of the subject text, and then obtaining corresponding sentence-level text representation; calculating the similarity of the sentence-level text representation of the transcription text and the subject text to obtain a correlation score; inputting the acoustic features and the word-level representation of the transcription text into a multi-layer Transformer decoder for feature fusion to obtain word-level pronunciation representation, and then obtaining a semantic perception score; and obtaining an evaluation result based on the correlation score and the semantic perception score. Through a deep learning model, acoustic features and language features are extracted, semantic perception fusion is performed, and accurate evaluation of open-scene English speech is realized.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of speech evaluation, and in particular to an open-scene English speech evaluation method and system based on semantic perception. BACKGROUND

[0002] The statements in this section merely provide background information related to the present disclosure and do not necessarily constitute the prior art.

[0003] English pronunciation assessment (PA) plays an important role in modern society, bringing great convenience to communication and exchange in daily life, and creating a more realistic and richer language learning environment for language learners, greatly improving learning efficiency. Speech assessment is an important research direction in the field of computer-assisted pronunciation training (CAPT), which is committed to timely, accurate and convenient scoring of specific aspects of pronunciation of non-native learners (L2 learners), so as to help learners improve their oral English ability, and has broad application prospects in the field of education and language learning. The realization of automatic assessment of English oral English will greatly reduce the difficulty of organizing the test, improve the scoring efficiency, and make large-scale oral English test a reality, entering the era of computer intelligent assessment.

[0004] In early related research, speech assessment mainly relied on teachers' subjective evaluation of students. With the development of technology, rule-based automatic assessment systems that can evaluate pronunciation accuracy based on pre-set speech models have emerged, but these systems lack flexibility and cannot adapt to individual pronunciation differences. The improvement of computer computing power and the progress of machine learning technology enable speech assessment to use more complex algorithms to improve the accuracy and flexibility of assessment.

[0005] Thanks to the breakthroughs of deep learning in the field of natural language processing and speech processing, the development of speech assessment technology has evolved from basic pronunciation recognition to complex speech analysis and feedback systems, gradually meeting the requirements of actual application scenarios. Most current speech assessment systems are based on hidden Markov models, using their likelihood score values and other related information as the basis for scoring, not only to evaluate the accuracy of pronunciation, but also to assess and feedback on the intonation, fluency, rhythm and other aspects of speech, helping language learners to improve their oral English ability comprehensively, and achieving good performance in related tasks.

[0006] Open scenario English speech evaluation is an important branch of speech evaluation field, mainly including scenarios such as oral translation, oral repetition, picture description, topic expression and the like. Open scenario speech evaluation only uses audio and does not rely on corresponding transcription text, and it is difficult to directly evaluate the oral expression ability of learners, so relevant research is relatively scarce. Although the existing research has proposed a system based on multiple self-supervised learning (SSL) models and a solution supporting two different scenarios of reading and opening, it fails to fully exploit the semantic information of the text; the LSTM-RNN method based on the attention mechanism, although it avoids feature engineering and achieves good performance, ignores the analysis of the acoustic features of pronunciation; and the innovative method based on SSL speech models such as wav2vec2.0 and HuBERT, although it performs well in large-scale unlabeled audio pre-training and makes progress in fluency scoring and zero-shot pronunciation evaluation downstream tasks, its evaluation dimension still has limitations. SUMMARY

[0007] In order to overcome the shortcomings of the prior art, the present application provides an open scenario English speech evaluation method and system based on semantic perception, which extracts acoustic features and language features through a deep learning model, and innovatively performs semantic perception fusion to realize accurate evaluation of open scenario English speech.

[0008] To achieve the above object, one or more embodiments of the present application provide the following technical solutions:

[0009] In a first aspect, the present application provides an open scenario English speech evaluation method based on semantic perception, comprising:

[0010] obtaining audio data to be evaluated;

[0011] inputting the audio data into a speech evaluation model for processing to obtain an evaluation result; wherein the construction process of the speech evaluation model comprises:

[0012] extracting features from the audio data to obtain frame-level acoustic features;

[0013] identifying the audio data to obtain a transcription text, inputting the transcription text and a topic text corresponding to the audio data into a language model for processing to obtain a word-level representation of the transcription text and a word-level representation of the topic text, obtaining a sentence-level text representation of the transcription text and a sentence-level text representation of the topic text after mean pooling operation, and calculating the similarity between the sentence-level text representation of the transcription text and the sentence-level text representation of the topic text to obtain a correlation score;

[0014] The frame-level acoustic feature and the word-level representation of the transcription text are input into a multi-layer Transformer decoder for feature fusion to obtain a word-level pronunciation representation, and a sentence-level pronunciation representation is obtained after a mean pooling operation, and then a semantic perception score is obtained;

[0015] Based on the correlation score and the semantic perception score, a final evaluation result is obtained.

[0016] Further, the speech evaluation model comprises an acoustic feature extraction module and a language feature extraction module arranged in parallel, and a semantic perception fusion module.

[0017] Further, the acoustic feature extraction module receives audio data, pre-processes the audio data by using a speech endpoint detection technology, obtains pre-processed audio data, and inputs the pre-processed audio data into a self-supervised speech pre-training model to obtain frame-level acoustic features; the frame-level acoustic features are filtered by using a mask technology to obtain filtered acoustic features.

[0018] Further, the language feature extraction module extracts features by using a language model, the language model comprises a first branch and a second branch arranged in parallel, and the first branch and the second branch have the same structure and are sequentially connected with a BERT encoder and a mean pooling layer, and the BERT encoders of the first branch and the second branch share parameters.

[0019] Further, the language model adopts a weakly supervised fine-tuning strategy, and based on the constructed positive and negative samples and weakly supervised labels, a cosine embedding loss is used as a loss function to fine-tune the language model.

[0020] Further, the cosine similarity of the sentence-level text representation of the transcription text and the sentence-level text representation of the theme text is calculated to obtain a correlation score, which is represented as:

[0021]

[0022] wherein, the correlation score is represented as, the sentence-level text representation of the transcription text is represented as, the sentence-level text representation of the theme text is represented as, the transpose is represented as.

[0023] Further, the semantic perception fusion module comprises an acoustic feature projection layer and a language feature projection layer arranged in parallel, a multi-layer Transformer decoder, and a mean pooling layer in sequence.

[0024] In a second aspect, the present application provides an open scenario English speech evaluation system based on semantic perception, comprising:

[0025] a data acquisition module configured to acquire audio data to be evaluated;

[0026] a model evaluation module configured to input the audio data into a speech evaluation model for processing to obtain an evaluation result, wherein a construction process of the speech evaluation model comprises:

[0027] performing feature extraction on the audio data to obtain frame-level acoustic features;

[0028] performing recognition on the audio data to obtain a transcription text, inputting the transcription text and a topic text corresponding to the audio data into a language model for processing to obtain a word-level representation of the transcription text and a word-level representation of the topic text, obtaining a sentence-level text representation of the transcription text and a sentence-level text representation of the topic text through mean pooling operation, and calculating a similarity between the sentence-level text representation of the transcription text and the sentence-level text representation of the topic text to obtain a correlation score;

[0029] inputting the frame-level acoustic features and the word-level representation of the transcription text into a multi-layer Transformer decoder for feature fusion to obtain a word-level pronunciation representation, obtaining a sentence-level pronunciation representation through mean pooling operation, and further obtaining a semantic perception score;

[0030] obtaining a final evaluation result based on the correlation score and the semantic perception score.

[0031] In a third aspect, the present application provides a computer readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps in the semantic perception based open scenario English speech evaluation method according to the first aspect.

[0032] In a fourth aspect, the present application provides a computer device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein the processor implements the steps in the semantic perception based open scenario English speech evaluation method according to the first aspect when executing the program.

[0033] The above one or more technical solutions have the following beneficial effects:

[0034] The application realizes accurate evaluation of open scene English speech by respectively extracting acoustic features and language features through a deep learning model and innovatively performing semantic perception fusion. Specifically, in the acoustic feature extraction stage, a pre-trained SSL acoustic model HuBERT is used as a pronunciation feature extraction module to extract frame-level acoustic features for subsequent acoustic aspect representation of pronunciation evaluation; in the language feature extraction stage, a fine-tuned SSL language model BERT is used as a text feature extraction module to extract word-level text features in the audio transcription text for subsequent language aspect representation of pronunciation evaluation; in the semantic perception stage, a novel semantic perception fusion module is designed to fully fuse the extracted acoustic features and language features and the like multi-modal features. The module performs soft alignment between the text features and the acoustic features to generate semantic fusion pronunciation representation, thereby better completing the evaluation task.

[0035] The application proposes a multi-modal feature extraction and semantic perception method for open scene English speech evaluation, fully excavates acoustic features and language features in audio data, realizes text and audio unification, and better completes the evaluation task.

[0036] The application uses an ASR model Whisper to obtain the audio transcription text of the learner when audio transcription is performed, which is helpful for excavating text information.

[0037] The application proposes a novel weakly supervised fine-tuning method, uses a cosine embedding loss function to weakly supervise the fine-tuning of the language model, which is helpful for solving the domain difference problem existing in the pre-trained model and improving the performance of the model in the downstream task. BRIEF DESCRIPTION OF DRAWINGS

[0038] The drawings accompanying the specification of the application form part of the application and serve to provide a further understanding of the application. The exemplary embodiments of the application and their description serve to explain the application without constituting an improper limitation thereof.

[0039] Figure 1 is a flowchart of the open scene English speech evaluation method based on semantic perception of the embodiment of the application;

[0040] Figure 2 is a framework diagram of the acoustic feature extraction module of the embodiment of the application;

[0041] Figure 3 is a structure diagram of the acoustic model HuBERT of the embodiment of the application;

[0042] Figure 4 is a framework diagram of the language feature extraction module of the embodiment of the application;

[0043] Figure 5 is a structure diagram of the language model of the embodiment of the application;

[0044] Figure 6 is a semantic perception fusion module framework diagram of an embodiment of the present application. DETAILED DESCRIPTION

[0045] It should be noted that the following detailed description is exemplary in nature and is intended to provide further description of the application. Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs.

[0046] It should be noted that the terms used herein are only intended to describe specific embodiments and are not intended to limit exemplary embodiments according to the present application. As used herein, the singular form is intended to include the plural form unless the context clearly indicates otherwise, and it should also be understood that when the terms "comprise" and / or "include" are used in the specification, there is a presence of the features, steps, operations, devices, components and / or combinations thereof.

[0047] The embodiments in the present application and the features in the embodiments can be combined with each other without conflict.

[0048] Embodiment one

[0049] As shown in Figure 1 The present embodiment discloses an open-scene English speech evaluation method based on semantic perception, which comprises the following steps:

[0050] S1: obtaining audio data to be evaluated;

[0051] S2: inputting the audio data to a speech evaluation model for processing to obtain an evaluation result; the speech evaluation model comprises side-by-side acoustic feature extraction module and language feature extraction module, semantic perception fusion module.

[0052] (I) acoustic feature extraction module

[0053] An acoustic feature extraction model is used in the acoustic feature extraction module for acoustic feature extraction. Unlike text, which naturally presents in relatively discrete units such as words and sentences, speech units are continuous, and not all acoustic features are meaningful, which requires the acoustic feature extraction model to perform high-quality acoustic feature extraction and filtering of meaningless features.

[0054] The acoustic feature extraction model takes the original audio wave (audio data) as input, extracts features from the audio data, and outputs frame-level acoustic features for subsequent acoustic representation of pronunciation evaluation, including a pre-processing module, a fine-tuning module, an acoustic model module, and a filtering module.

[0055] (1) Preprocessing module

[0056] The preprocessing module is used for preprocessing the input audio data. Some speakers actually take a long time to think when answering the question, and the average silence duration can reach 2.3 seconds / time, so it is shown as a silent state in the audio data, which not only increases the data size, but also interferes with the accurate extraction of acoustic features. In order to avoid this phenomenon, the Voice Activity Detection (VAD) technology is innovatively used to segment the audio data, accurately identify and remove the silent part, thereby improving the efficiency and accuracy of subsequent processing.

[0057] The VAD technology can effectively distinguish between speech segments and silent segments by analyzing the energy and zero-crossing rate of the audio signal, ensuring that the effective speech information in the audio is retained while the unnecessary silent part is removed. This preprocessing step lays a good data foundation for subsequent acoustic feature extraction.

[0058] (2) Fine-tuning module

[0059] The fine-tuning module is used to fine-tune the acoustic model HuBERT to better adapt to the needs of open-scene English speech evaluation. In the pre-training stage, HuBERT learns the underlying features of the audio signal through unsupervised learning, but these features may not be completely suitable for specific pronunciation evaluation tasks. Therefore, the CTC loss is used to fine-tune HuBERT, and the CTC loss formula is as follows:

[0060]

[0061] where, is the input sequence, is the corresponding target sequence, is the output label sequence predicted by the model at each time step of the input sequence, i.e., the path, is the set of all paths that can be mapped to the target sequence .

[0062] During fine-tuning, the HuBERT freezes the entire model weight except the convolutional waveform encoder, removes the projection layer and replaces it with a randomly initialized Softmax layer, which not only preserves the general feature learning ability of the pre-trained model, but also allows for precise optimization for specific tasks, ensuring the adaptability and accuracy of the model in open-scene speech evaluation.

[0063] (3) Acoustic model module

[0064] The acoustic model is a self-supervised speech pre-training model HuBERT. Unlike the BERT model in NLP, HuBERT does not have a "dictionary" of speech units to refer to during the pre-training phase. In the representation learning phase based on mask prediction, HuBERT provides pseudo labels for each frame with unsupervised clustering algorithms such as k-means, adopts the same mask generation strategy as wav2vec2.0, and combines the cross-entropy loss of the masked area and the loss of the unmasked area to drive the model learning. Specifically, the acoustic model will randomly mask part of the audio frames during training, and then predict the pseudo labels of the masked frames through the context information, so as to learn the context-related features of the audio signal.

[0065] At the model architecture level, HuBERT is consistent with the wav2vec2.0 architecture, and includes a convolutional waveform encoder, a masker embedding layer, a BERT encoder, and a projection layer in turn. HuBERT extracts features from preprocessed audio data and outputs frame-level acoustic features, which can be represented by the following formula:

[0066]

[0067] wherein, represents the preprocessed audio data, represents the frame-level acoustic feature, is the size of the dictionary, is the number of time steps.

[0068] At the same time, HuBERT can identify and classify the time steps of the blank mark and the space symbol, and then mark the value of this part as 0 through the mask to achieve filtering.

[0069] (4) Filtering module

[0070] The application innovatively proposes a filtering module, which filters the frame-level acoustic features extracted by the acoustic model using the mask technology. For pronunciation evaluation, part of the acoustic features is meaningless, and this module marks all time steps classified as blank marks and space symbols as zero values after the acoustic model module to filter and facilitate subsequent operation and calculation.

[0071]

[0072] wherein, represents the filtered acoustic feature, represents the mask matrix, represents the frame-level acoustic feature, represents element-by-element multiplication.

[0073] (2) Language feature extraction module

[0074] The language feature extraction model is a deep learning model, which can better extract features reflecting language content and semantic information from the audio data, and provide language level basis for subsequent speech evaluation.

[0075] The language feature extraction model sequentially comprises a speech recognition module, a language model module and a topic similarity score calculation module. The initial input of the model includes audio and text modal data. The corresponding transcription text of the audio data and the original topic text corresponding to the audio data are obtained through the speech recognition module, and are jointly input into the language model module. Finally, the word-level language features and the text topic similarity score are output.

[0076] (1) Speech recognition module

[0077] The speech recognition module uses an automatic speech recognition (ASR) model Whisper to recognize the reply audio of the learner and obtains the corresponding transcription text, so as to extract language features. Compared with the traditional speech recognition process using a combination of an acoustic model, a language model and a decoder, Whisper completes multiple tasks with a unified model, greatly simplifying the processing process.

[0078] The speech recognition module receives the audio data preprocessed by the acoustic feature extraction module, performs speech recognition through the Whisper model, and generates the transcription text of the learner's reply. This transcription text not only contains the language content in the speech, but also provides a basis for subsequent language feature extraction and semantic perception calculation.

[0079] (2) Language model module

[0080] The language model comprises a first branch and a second branch in parallel, both of which have the same structure of a BERT encoder and a mean pooling layer connected in turn, and the BERT encoders of the first branch and the second branch share parameters. That is, the language model module is composed of a twin pre-trained BERT network and a mean pooling layer, and the twin BERT network comprises two identical BERT subnetworks which share the same parameters. This parameter sharing structure enables the model to utilize the same semantic understanding mechanism when processing sentence pairs, thereby more effectively comparing the relationship between the two sentences.

[0081] The input of the language model module is the transcription text generated by the speech recognition module and the topic text corresponding to the audio data, and then the BERT encoder is used to obtain the word-level representation of the transcription text and the topic text. The specific operation definition is as follows:

[0082]

[0083]

[0084] wherein, denotes the tokenized transcript text, denotes the tokenized subject text, denotes the word-level representation of the transcript text, denotes the word-level representation of the subject text, denotes the feature dimension, denotes the number of words in the transcript text, denotes the number of words in the subject text.

[0085] Then, the sentence-level text representations and are obtained using the mean-pooling operation and , which integrates the local semantic information in the text into a global semantic representation, thereby providing more comprehensive feature support for subsequent semantic analysis and evaluation, denoted as:

[0086]

[0087] .

[0088] The application also innovatively proposes a language model weakly supervised fine-tuning strategy, which fine-tunes by constructing weakly supervised labels on the target dataset, solves the domain difference problem existing in the pre-trained model, and further improves the performance of the language model in the open scene English speech evaluation task.

[0089] Specifically, first, based on the existing annotations, set the forward threshold and the reverse threshold to construct positive and negative samples and their weakly supervised labels, and then use the constructed samples and pseudo labels to fine-tune the model. For learners whose expert scores are higher than , their replies are not only considered reasonable in content, but also have a high probability of being related to the given topic. Therefore, for learners whose expert scores are greater than , assign the content score label 1 and the relevance label 1 to their reply transcript text as positive samples; on the contrary, for learners whose expert scores are less than , the content of their replies is considered unreasonable and unrelated to the topic, and the content score label 0 and the relevance label -1 are assigned to the sample as negative samples. Subsequently, using these constructed positive and negative samples and weakly supervised label topic-reply pairs, the language model BERT is fine-tuned with the cosine embedding loss as the loss function, and the loss function is defined as follows:

[0090]

[0091] wherein, is the assigned weakly supervised relevance label, represents the minimum interval between negative sample pairs, and is set to 0 in this embodiment.

[0092] (3) A topic similarity score calculation module

[0093] The topic similarity score calculation module calculates the relevance between the transcribed text and the topic text by using cosine similarity, thereby ensuring that the learner's reply is relevant to the topic. Specifically, the cosine similarity between the sentence-level text representation of the transcribed text and the topic text is calculated as the relevance score between the learner's reply and the given topic. The higher the score, the closer the learner's answer is to the topic, and vice versa, indicating that the answer has lower relevance to the topic. The specific operation process is defined as follows:

[0094]

[0095] wherein, represents the relevance score, represents the sentence-level text representation (language feature) of the transcribed text, represents the sentence-level text representation of the topic text, represents transposition.

[0096] (Three) Semantic perception fusion module

[0097] The audio data naturally contains acoustic features and language features. Here, the two features are innovatively fused to replace the previous strategy of using only a single feature, thereby improving the accuracy of model evaluation.

[0098] The semantic perception fusion module sequentially includes the acoustic feature projection layer and the language feature projection layer, the Transformer decoder layer and the mean pooling layer, the Transformer decoder layer sequentially includes the cross-attention layer, the addition and normalization layer, the feedforward neural network layer, and the addition and normalization layer.

[0099] Word-level representation of transcribed text (semantics feature) and filtered acoustic features Soft alignment is achieved through cross-attention to obtain word-level pronunciation representation wherein, is used as the query vector input to the Transformer decoder, and the acoustic feature The key vector and value vector are used as inputs, respectively. This attention-based feature fusion method can decode detailed pronunciation information and derive reliable pronunciation representations. The dataset used in this invention only has relevant annotations at the sentence level; therefore, it is necessary to transform word-level features into sentence-level features. Subsequently, the model processes the word-level pronunciation representations obtained through multiple feature fusions. Mean pooling is applied to obtain a statement-level pronunciation representation. , is represented as:

[0100]

[0101]

[0102]

[0103] in, This represents the output of the layer above the Transformer decoder. This represents the semantic features after projection. This represents the output of the last layer of the Transformer decoder, i.e. .

[0104] Ultimately, the model represents speech-level pronunciation. Above this, add a regression module (regression prediction head) to generate pronunciation scores with semantic awareness features. (Semantic perception score).

[0105] semantic perception score and correlation score By performing a learnable weighted sum, the final predicted score of the model is obtained, expressed as:

[0106]

[0107] in, This indicates the predicted score (evaluation result).

[0108] During the training phase of the speech evaluation model, all pre-trained model parameters are frozen to prevent performance degradation caused by the introduction of new datasets. The overall loss is as follows:

[0109]

[0110] in, For mean square error loss, It is the sample size. The model is for the first The predicted value for each sample, It is the first The true value of the sample, i.e. the expert scoring result. By calculating the deviation between the predicted value and the true value, the model parameters are trained and adjusted, thereby guiding the recognition model to output correct results.

[0111] Embodiment two

[0112] The embodiment discloses an open-scene English speech evaluation system based on semantic perception, comprising:

[0113] A data acquisition module configured to acquire audio data to be evaluated;

[0114] A model evaluation module configured to input the audio data to a speech evaluation model for processing to obtain an evaluation result; wherein the construction process of the speech evaluation model comprises:

[0115] Feature extraction is performed on the audio data to obtain frame-level acoustic features;

[0116] Transcription text is obtained by recognizing the audio data, and the transcription text and the subject text corresponding to the audio data are input to a language model for processing to obtain word-level representations of the transcription text and the subject text, and sentence-level text representations of the transcription text and the subject text are obtained after mean pooling operation, and a correlation score is calculated by calculating the similarity between the sentence-level text representations of the transcription text and the subject text.

[0117] The frame-level acoustic features and the word-level representations of the transcription text are input to a multi-layer Transformer decoder for feature fusion to obtain word-level pronunciation representations, and sentence-level pronunciation representations are obtained after mean pooling operation, and then a semantic perception score is obtained;

[0118] Based on the correlation score and the semantic perception score, a final evaluation result is obtained.

[0119] Embodiment three

[0120] The purpose of the embodiment is to provide a computing device, comprising a memory, a processor and a computer program stored on the memory and executable on the processor, wherein the processor executes the program to realize the steps of the method of embodiment one.

[0121] Embodiment four

[0122] The purpose of the embodiment is to provide a computer-readable storage medium, a computer-readable storage medium having a computer program stored thereon, wherein the program is executed by a processor to execute the steps of the method of embodiment one.

[0123] The steps and methods involved in the apparatuses of Embodiments 3 and 4 above correspond to those in Embodiment 1. For specific implementation details, please refer to the relevant description section of Embodiment 1. The term "computer-readable storage medium" should be understood as a single medium or multiple media including one or more instruction sets; it should also be understood as including any medium capable of storing, encoding, or carrying an instruction set for execution by a processor and enabling the processor to perform any of the methods in this invention.

[0124] Those skilled in the art will understand that the modules or steps of the present invention described above can be implemented using general-purpose computer devices. Optionally, they can be implemented using computer-executable program code, thereby allowing them to be stored in a storage device for execution by a computer device, or they can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. The present invention is not limited to any particular combination of hardware and software.

[0125] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

[0126] While the specific embodiments of the present invention have been described above in conjunction with the accompanying drawings, this is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art without creative effort based on the technical solutions of the present invention are still within the scope of protection of the present invention.

Claims

1. A method for open-scene English speech evaluation based on semantic perception, characterized in that, The method comprises the following steps: obtaining audio data to be evaluated; inputting the audio data into a speech evaluation model for processing; wherein the speech evaluation model comprises an acoustic feature extraction module and a language feature extraction module, and a semantic perception fusion module; the acoustic feature extraction module is used for feature extraction of audio data to obtain frame-level acoustic features; the language feature extraction module is used for recognizing audio data to obtain a transcription text, inputting the transcription text and a subject text corresponding to the audio data into a language model for processing to obtain a word-level representation of the transcription text and a word-level representation of the subject text, obtaining a sentence-level text representation of the transcription text and a sentence-level text representation of the subject text after mean pooling operation, and calculating the similarity of the sentence-level text representation of the transcription text and the sentence-level text representation of the subject text to obtain a correlation score; the semantic perception fusion module is used for inputting the frame-level acoustic features and the word-level representation of the transcription text into a multi-layer Transformer decoder for feature fusion to obtain a word-level pronunciation representation, obtaining a sentence-level pronunciation representation after mean pooling operation, and further obtaining a semantic perception score; based on the correlation score and the semantic perception score, a final evaluation result is obtained.

2. The method of claim 1, wherein the open scene English speech evaluation based on semantic perception is characterized by, The acoustic feature extraction module receives audio data, pre-processes the audio data by using a speech endpoint detection technology, obtains pre-processed audio data, and inputs the pre-processed audio data into a self-supervised speech pre-training model to obtain frame-level acoustic features; the frame-level acoustic features are filtered by using a mask technology to obtain filtered acoustic features.

3. The method of claim 1, wherein the open scene English speech evaluation based on semantic perception is characterized by, The language feature extraction module extracts features by using a language model, the language model comprises a first branch and a second branch arranged in parallel, both of which have the same structure and are sequentially connected BERT encoders and mean pooling layers, and the BERT encoders of the first branch and the second branch share parameters.

4. The method of claim 3, wherein the open scene English speech evaluation based on semantic perception is characterized by, The language model adopts a weakly supervised fine-tuning strategy, and is fine-tuned based on constructed positive and negative samples and weakly supervised labels by taking a cosine embedding loss as a loss function.

5. The method of claim 1, wherein the semantic-aware open-durational English speech assessment method is characterized by, The cosine similarity of the sentence-level text representation of the transcription text and the sentence-level text representation of the subject text is calculated to obtain a correlation score, which is represented as: wherein, denotes a correlation score, denotes a sentence-level textual representation of the transcribed text, denotes a sentence-level textual representation of the topic text, denotes a transpose.

6. The method of claim 1, wherein the semantic-aware open-durational English speech assessment method is characterized by, The semantic perception fusion module sequentially comprises an acoustic feature projection layer and a language feature projection layer in parallel, a layer Transformer decoder and a mean pooling layer.

7. An open-scene English speech evaluation system based on semantic awareness, characterized by, The method comprises the following steps: a data acquisition module configured to obtain audio data to be evaluated; a model evaluation module configured to input the audio data into a speech evaluation model for processing; wherein the speech evaluation model comprises an acoustic feature extraction module and a language feature extraction module, and a semantic perception fusion module; the acoustic feature extraction module is used for feature extraction of audio data to obtain frame-level acoustic features; the language feature extraction module is used for recognizing audio data to obtain a transcription text, inputting the transcription text and a subject text corresponding to the audio data into a language model for processing to obtain a word-level representation of the transcription text and a word-level representation of the subject text, obtaining a sentence-level text representation of the transcription text and a sentence-level text representation of the subject text after mean pooling operation, and calculating the similarity of the sentence-level text representation of the transcription text and the sentence-level text representation of the subject text to obtain a correlation score; The semantic perception fusion module is configured to input the frame-level acoustic features and the word-level representation of the transcription text into a multi-layer Transformer decoder for feature fusion to obtain a word-level pronunciation representation, perform mean pooling operation on the word-level pronunciation representation to obtain a sentence-level pronunciation representation, and further obtain a semantic perception score. Based on the correlation score and the semantic perception score, a final evaluation result is obtained.

8. A computer-readable storage medium having stored thereon a computer program, characterized in that, The program, when executed by a processor, implements the semantic perception based open scenario English speech evaluation method according to any one of claims 1-6.

9. A computer device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor, when executing the program, implements the steps in the semantic perception based open scenario English speech evaluation method according to any one of claims 1-6.

Images