English listening and speaking assessment method, device, equipment and storage medium

Through the multi-dimensional scoring model, serialized encoding and speech feature extraction of English listening and speaking evaluation technology is solved, and the recognition error problem of evaluating non-standard accents and openness problems in the prior art is solved, achieving higher accuracy and robustness , suitable for a wide range of English listening and speaking evaluation scenarios.

CN118675550BActive Publication Date: 2024-11-05CHENGDU JIAFAANTAI EDUCATION TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411162786.7
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-08-23
Publication Date
2024-11-05
Estimated Expiration
2044-08-23

AI Technical Summary

Technical Problem

The existing English listening and speaking evaluation technology has identification errors when dealing with non-standard accents, speech speed variations and background noise, resulting in scoring deviations, affecting the accuracy and reliability of the evaluation, especially in the evaluation of open-minded problems.

Method used

Using a multi-dimensional scoring model, serialized encoding and speech feature extraction are performed by obtaining listening material text, question text and voice comments to be tested, combined with pre-trained text feature network, acoustic feature network and fusion feature network, output scores of each dimension evaluation. And a weighted average is performed based on predefined weights to obtain a comprehensive score.

Benefits of technology

The accurate evaluation of speech is achieved without standard text, which improves the accuracy and robustness of the evaluation, has a wider range of application, avoids recognition errors in automatic speech recognition, and enhances the accuracy of the evaluation.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118675550B_ABST
    Figure CN118675550B_ABST
Patent Text Reader

Abstract

The present application discloses an English listening and speaking assessment method, device, equipment and storage medium, including: obtaining a listening material text, a question text and a speech to be assessed; serializing the listening material text and the question text, encoding the serialized listening material text and the question text to obtain a coding sequence; encoding the speech to be assessed to obtain a speech feature vector sequence; inputting the coding sequence and the speech feature vector sequence into a pre-trained multi-dimensional scoring model to obtain the evaluation scores of each dimension of the speech to be assessed; and performing weighted averaging on the evaluation scores of each dimension according to the pre-defined weights of each dimension to obtain a comprehensive score of the speech to be assessed. The method provided in the embodiment of the present application can accurately assess speech even without a standard text, and can solve the evaluation problem of open questions.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the technical field of automatic assessment, and in particular to an English listening and speaking assessment method, device, equipment and storage medium. Background Art

[0002] English listening and speaking assessment aims to evaluate an individual's ability to understand and express spoken English, which is of great significance for education, career development and international communication. It not only measures the mastery of language knowledge, but also evaluates practical communication skills, including pronunciation, fluency, vocabulary use and grammatical correctness.

[0003] Traditionally, English listening and speaking assessments have mostly relied on standard text materials, such as reading a specified paragraph or answering questions based on a preset script. This method is convenient for building a unified scoring benchmark, but it faces limitations, especially when it comes to open-ended questions. The lack of fixed standard answer texts significantly increases the difficulty of assessment.

[0004] In order to solve the evaluation problem of open-ended answers, Automatic Speech Recognition (ASR) technology has been widely introduced, aiming to convert spoken language into text, and then apply natural language processing technology for analysis and scoring. However, ASR technology still has recognition errors, especially when dealing with complex situations such as non-standard accents, speech rate variations, and background noise. These errors will be directly transmitted to the subsequent evaluation process, resulting in scoring bias, affecting the accuracy and reliability of the evaluation, and thus limiting the effectiveness of the evaluation and the expansion of application scenarios. Therefore, exploring more accurate and robust English listening and speaking evaluation technology has become an important direction for current research and practice. Summary of the invention

[0005] The embodiments of the present application provide an English listening and speaking assessment method, apparatus, device and storage medium to at least solve the technical problem in the related art that it is difficult to accurately conduct English listening and speaking assessment.

[0006] According to one aspect of an embodiment of the present application, there is provided an English listening and speaking assessment method, comprising:

[0007] Obtain the listening material text, question text and the audio to be evaluated;

[0008] Serializing the listening material text and the question text, and encoding the serialized listening material text and the question text to obtain a coding sequence;

[0009] Encode the speech to be evaluated to obtain a speech feature vector sequence;

[0010] Input the coding sequence and the speech feature vector sequence into the pre-trained multi-dimensional scoring model to obtain the evaluation scores of each dimension of the speech to be evaluated;

[0011] According to the pre-defined weights of each dimension, the evaluation scores of each dimension are weighted averaged to obtain a comprehensive score for the speech to be evaluated.

[0012] In one embodiment, the listening material text and the question text are serialized, and the serialized listening material text and the question text are encoded to obtain an encoded sequence, including:

[0013] Use the preset template to splice the listening material text and the question text to obtain the serialized listening material text and question text;

[0014] The serialized listening material text and question text are input into the word segmenter for encoding to obtain the encoding sequence.

[0015] In one embodiment, the speech to be evaluated is encoded to obtain a speech feature vector sequence, including:

[0016] Calculate the original Mel-spectrogram feature vector sequence of the speech to be evaluated;

[0017] Preprocessing the original Mel spectrum feature vector sequence to obtain a Mel spectrum feature vector sequence;

[0018] The Mel spectrum feature vector sequence is input into the acoustic encoder to obtain the speech feature vector sequence.

[0019] In one embodiment, the multi-dimensional scoring model includes:

[0020] The text feature network consists of a word embedding layer, which receives the encoding sequence and outputs a sequence of text feature vectors;

[0021] The acoustic feature network is composed of multiple linear layers and activation functions, which is used to receive a speech feature vector sequence and output an aligned speech feature vector sequence;

[0022] A fusion feature network, composed of a position embedding layer and a feature identifier embedding layer, is used to receive the text feature vector sequence and the aligned speech feature vector sequence, and output a fusion feature vector sequence;

[0023] An encoding network, composed of multiple transformer layers with an autoregressive attention mechanism, is used to receive the fused feature vector sequence and output a latent vector sequence;

[0024] A language network, composed of a linear layer and a softmax activation function, is used to receive the latent vector sequence and output a predicted coding sequence;

[0025] The scoring network, composed of multiple linear layers and activation functions, is used to receive the latent vector sequence and output the predicted scores of each evaluation dimension.

[0026] In one embodiment, the scoring network includes:

[0027] The word pronunciation evaluation network is used to receive the latent vector sequence and output the predicted pronunciation score sequence;

[0028] Fluency evaluation network, used to receive latent vector sequences and output fluency evaluation scores;

[0029] Content evaluation network, which is used to receive latent vector sequences and output content evaluation scores;

[0030] The rhythm evaluation network is used to receive the latent vector sequence and output the rhythm evaluation score.

[0031] In one embodiment, before inputting the coding sequence and the speech feature vector sequence into the pre-trained multi-dimensional scoring model, the method further includes:

[0032] Obtain training data, which includes listening material text, question text, assessment speech, assessment speech target text, and normalized scores of each dimension;

[0033] Randomly divide the training data into training data set and validation data set;

[0034] Based on the training data set and the validation data set, the gradient descent method is used to train the multi-dimensional scoring model.

[0035] In one embodiment, the loss function of the multi-dimensional scoring model is:

[0036]

[0037] in, represents the preset hyperparameter, M represents the number of scoring dimensions, and m represents the current scoring dimension. Represents the prediction results of each dimension output by the scoring network, represents the normalized score of each dimension, N represents the length of the target text encoding sequence, V represents the vocabulary, represents the true probability value of the jth word at the i-th position, represents the predicted probability value of the jth word at the i-th position.

[0038] According to another aspect of the embodiment of the present application, an English listening and speaking assessment device is provided, comprising:

[0039] The acquisition module is used to obtain the listening material text, question text and the voice to be evaluated;

[0040] A text encoding module is used to serialize the listening material text and the question text, and encode the serialized listening material text and the question text to obtain a coding sequence;

[0041] A speech coding module is used to encode the speech to be evaluated to obtain a speech feature vector sequence;

[0042] Each dimension evaluation module is used to input the coding sequence and the speech feature vector sequence into the pre-trained multi-dimensional scoring model to obtain the evaluation score of each dimension of the speech to be evaluated;

[0043] The comprehensive evaluation module is used to perform weighted average of the evaluation scores of each dimension according to the pre-defined weights of each dimension to obtain a comprehensive score for the speech to be evaluated.

[0044] According to another aspect of the embodiments of the present application, there is also provided an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor is configured to execute the English listening and speaking assessment method through the computer program.

[0045] According to another aspect of the embodiments of the present application, a computer-readable storage medium is provided, in which a computer program is stored, wherein the computer program is configured to execute the above-mentioned English listening and speaking assessment method when it is run.

[0046] The technical solution provided by the embodiments of the present application may have the following beneficial effects:

[0047] The method provided in the embodiment of the present application only needs to input the listening material text, the question text and the speech to be evaluated into a multi-dimensional scoring model to obtain the scores of each dimension and the comprehensive score of the speech to be evaluated. There is no need to obtain a standard text. Even in the absence of a standard text, the speech can be accurately evaluated. The evaluation problem of open-ended questions can be solved. The model can infer the content that the speech to be evaluated wants to express based on the listening material text and the question text, and then combine the content and pronunciation to give an accurate score. The method has a wider scope of application and can avoid the problem of recognition errors in automatic speech recognition. The method of the present application has higher accuracy and strong robustness. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0049] Figure 1 is a flow chart of an optional English listening and speaking assessment method according to an embodiment of the present application;

[0050] Figure 2 is a structural schematic diagram of a multi-dimensional scoring model according to an embodiment of the present application;

[0051] Figure 3is a schematic diagram of an English listening and speaking assessment device according to an embodiment of the present application;

[0052] Figure 4 It is a schematic diagram of the structure of an optional electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0053] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.

[0054] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0055] The following is combined with Figure 1 The English listening and speaking assessment method of the embodiment of the present application is described in detail. Figure 1 As shown, the method mainly includes the following steps:

[0056] S101 obtains the listening material text, the question text and the voice to be evaluated.

[0057] In an exemplary scenario, the English listening and speaking test includes playing a listening material and answering questions based on the listening material. The listening material text specifically includes the reading material in the listening test or text with similar functions. The question text refers to the specific question in the listening test or text with similar functions. The user answers the question based on the question heard and obtains the voice to be evaluated.

[0058] The method of the present application only needs to obtain the listening material text, the question text and the voice to be evaluated, and does not need to obtain the standard text corresponding to the answer to the question. It can solve the evaluation problem of open-ended questions.

[0059] S102 serializes the listening material text and the question text, and encodes the serialized listening material text and the question text to obtain a coding sequence.

[0060] In one embodiment, the listening material text and the question text are serialized, and the serialized listening material text and the question text are encoded to obtain an encoded sequence, including:

[0061] The listening material text and the question text are spliced ​​using a preset template to obtain serialized listening material text and question text.

[0062] Specifically, serialization refers to arranging or organizing these texts in a certain order so that they can be processed by programs or understood by users. The present embodiment uses a preset splicing template "listening material: <listening material text>\ntitle: <title text>" to splice the listening material text and the title text.

[0063] In one embodiment, the serialized text is “Listening material: Would you like some more rice?\nTopic: Answer based on what you hear”.

[0064] Furthermore, the serialized listening material text and question text are input into a word segmenter for encoding to obtain an encoding sequence.

[0065] In an optional embodiment, a Qwen word segmenter is used as a text encoding method, and the serialized text is input into the word segmenter to obtain an encoding sequence.

[0066] S103 encodes the speech to be evaluated to obtain a speech feature vector sequence.

[0067] In one embodiment, encoding the speech to be evaluated to obtain a speech feature vector sequence includes: calculating an original Mel-spectrogram feature vector sequence of the speech to be evaluated.

[0068] Specifically, the original speech signal, usually waveform data in the time domain, is converted into Mel-frequency cepstral coefficients. Mel frequency is a nonlinear frequency scale that mimics the human ear's ability to perceive sounds of different frequencies. First, the speech signal is pre-emphasized to enhance the energy of the high-frequency part. Then, the signal is converted from the time domain to the frequency domain through the fast Fourier transform (FFT). Next, the signal in the frequency domain is mapped to the Mel-frequency scale, and the output energy of each Mel filter is extracted through a filter bank. Finally, the logarithm of these energy values ​​is taken, and the original Mel-spectrum feature vector is obtained through the discrete cosine transform (DCT).

[0069] Furthermore, the original mel spectrum feature vector sequence is preprocessed to obtain a mel spectrum feature vector sequence.

[0070] The original Mel spectrum feature may contain noise and instability, so preprocessing is required to improve the stability and robustness of the feature. The preprocessing in the example of this application includes: first taking the logarithm of the feature; then truncating the lower bound of the feature data so that the minimum value is greater than or equal to a certain specific value. In this example, this specific value is the maximum value of the feature logarithm minus 8. Then scale the feature value after the above processing to a specific interval, the interval in this example is:

[0071]

[0072] The feature sequence obtained after this series of processing is the Mel spectrum feature vector sequence.

[0073] Furthermore, the mel-spectrogram feature vector sequence is input into an acoustic encoder to obtain a speech feature vector sequence.

[0074] The preprocessed Mel-spectrogram feature sequence is used as input and fed into the acoustic encoder. The acoustic encoder can be a deep learning model such as a recurrent neural network (RNN), a long short-term memory network (LSTM), or a convolutional neural network (CNN). The role of the encoder is to further extract and abstract speech features and generate more advanced and abstract feature representations that can better capture the semantic and structural information in the speech signal.

[0075] In one embodiment, the acoustic encoder used is a whisper-large encoder, and the mel-spectrogram feature vector sequence is input into the encoder to obtain a speech feature vector sequence. In this example, the dimension of the speech feature is 1280 dimensions.

[0076] S104 inputs the coding sequence and the speech feature vector sequence into a pre-trained multi-dimensional scoring model to obtain the evaluation scores of each dimension of the speech to be evaluated.

[0077] In one embodiment, before inputting the coding sequence and the speech feature vector sequence into the pre-trained multi-dimensional scoring model, the method further includes:

[0078] Obtain training data, which includes listening material text, question text, assessment speech, assessment speech target text and normalized scores of each dimension.

[0079] The training data in this embodiment consists of listening test questions, recording data and manual scoring. The manual scoring dimensions in this embodiment include:

[0080] Word Pronunciation: The pronunciation score for each word.

[0081] Fluency: The fluency score of the entire speech (fluency refers to the rationality of speaking speed, effective duration and pauses).

[0082] Content: The score is based on whether the content of the answer in the voice is in line with the requirements of the listening question and whether the answer is complete.

[0083] Prosody: The prosody score of the entire speech.

[0084] The scores of the above dimensions are normalized to the range of 0 to 1.

[0085] Here is a sample from this example:

[0086] {"id": 10081,"speak_content": "Would you like some more rice?","question": "Answer according to what you heard.","answer_audio": " / data / 10081.wav","answer_text": "yes please""score": {"words": [{"word": "yes","score": 1.0},{"word":"please","score": 0.9}],"fluency": 0.7,"content": 1.0,"rhythm": 0.7}}

[0087] In this code, "speak_content" is a field in the JSON object, and its value is "Would you like some more rice?". This is a sentence asking someone if they need more rice. The question "question" is: "Answer based on what you heard.". "answer_audio" is a field, and its value is " / data / 10081.wav". This value indicates the path to the audio file of the answer. The value of the "answer_text" field is "yes please". This indicates that when asked "Would you like some more rice?", the respondent selected this phrase as a response. "word": "yes", "score": 1.0 indicates that in the text of the answer "yes please", the word "yes" has a score of 1.0. "word": "please", "score": 0.9 indicates that in the text of the answer "yes please", the word "please" has a score of 0.9. "fluency": 0.7 indicates that the fluency score of the answer is 0.7. "content": 1.0 means the answer's content score is 1.0. "rhythm": 0.7 means the answer's rhythm score is 0.7.

[0088] The listening material text and question text in the training sample are serialized and encoded to obtain a text encoding sequence. The encoding sequence of the above sample after encoding by the word segmenter is as follows:

[0089] [111751, 100643, 5122, 27989, 498, 1075, 1045, 803, 19653, 5267,106367, 5122, 100345, 104188, 100363, 71817, 102104].

[0090] The speech to be evaluated is encoded to obtain a speech feature vector sequence. The target text of the evaluation speech is encoded into a target text encoding sequence through a word segmenter. As mentioned above, the Qwen word segmenter is used as the text encoding method in this example. Here, the target text of the evaluation speech is input into the word segmenter to obtain an encoding sequence, which is the target text encoding sequence. The encoding result of this example sample is [9693, 4486].

[0091] The acquired and processed training data is randomly divided into a training data set and a validation data set. Based on the training data set and the validation data set, the multi-dimensional scoring model is trained using the gradient descent method. During the training process, the validation data set is used to verify the score of the current model weight for each training step; the model with the highest model weight score is finally selected as the final model weight used.

[0092] In an optional implementation, the loss function of the multi-dimensional scoring model is:

[0093]

[0094] in, represents the preset hyperparameter, M represents the number of scoring dimensions, and m represents the current scoring dimension. Represents the prediction results of each dimension output by the scoring network, represents the normalized score of each dimension, N represents the length of the target text encoding sequence, V represents the vocabulary, represents the true probability value of the jth word at the i-th position, represents the predicted probability value of the jth word at the i-th position.

[0095] like Figure 2 As shown in the figure, the multi-dimensional scoring model structure includes:

[0096] The text feature network is composed of the word embedding layer of the open source qwen-14B-base. It is used to receive the encoding sequence and output the text feature vector sequence.

[0097] The acoustic feature network consists of two linear layers. The relu activation function is used between the linear layers. The sizes of the two linear layers are 1280x5120 and 5120x5120 respectively. The initial weights are initialized randomly. It is used to receive a speech feature vector sequence and output an aligned speech feature vector sequence.

[0098] The fusion feature network is used to receive the text feature vector sequence and the aligned speech feature vector sequence, and output the fusion feature vector sequence. It consists of a position embedding layer and a feature identifier embedding layer, the position embedding layer dimension is 5120, and the feature identifier embedding layer dimension is 5120. The network accepts the feature sequence formed by connecting the text feature vector sequence and the aligned speech feature vector sequence, and then calculates the position feature sequence according to the position information, and then inputs the feature identifier embedding layer according to the feature type to which the position belongs to obtain the feature identifier feature sequence. The three feature sequences are added together to obtain the final fusion feature vector sequence. The initial weight of the network is initialized in a random manner.

[0099] The encoding network is used to receive the fused feature vector sequence and output the latent vector sequence. In this example, it is composed of 40 qwen-14B-base transformer layers, which have their own multi-head attention mechanism.

[0100] The language network is used to receive the latent vector sequence and output the predicted coding sequence; in this example, it is composed of the corresponding linear layer of qwen-14b-base and the softmax activation function. The predicted coding sequence output by the language network is used in model training.

[0101] The scoring network is used to receive the latent vector sequence and output the predicted score of each evaluation dimension. In one embodiment, the scoring network includes: a word pronunciation evaluation network, which is used to receive the latent vector sequence and output the predicted pronunciation score sequence; a fluency evaluation network, which is used to receive the latent vector sequence and output the fluency evaluation score; a content evaluation network, which is used to receive the latent vector sequence and output the content evaluation score; and a rhythm evaluation network, which is used to receive the latent vector sequence and output the rhythm evaluation score.

[0102] The scoring network in this example contains 4 sub-networks, namely the word pronunciation evaluation network, the fluency evaluation network, the content evaluation network and the rhythm evaluation network. Among them, the word pronunciation evaluation network consists of two linear layers, which are activated by the relu activation function between the linear layers. The two linear layers are 5120x5120 and 5120x1 respectively, and finally activated by the sigmoid activation function. The initial weights are initialized in a random manner. The network receives the predicted latent vector sequence and outputs the predicted pronunciation score sequence. Each score of the sequence is the score of the corresponding word. The fluency evaluation network, the content evaluation network and the rhythm evaluation network have the same structure, but different weights. Structurally, it consists of two linear layers, which are activated by relu in the middle. The linear layers are 5120x1280 and 1280x1 respectively, and the final output is also activated by the sigmoid activation function. These three networks receive the last vector of the predicted latent vector sequence and output the predicted score of the corresponding dimension.

[0103] Optionally, the number of subgrids in the scoring network is not specifically limited in the present embodiment of the application, and can be flexibly selected according to needs. For example, only a fluency evaluation network and a content evaluation network are set. Other evaluation networks can also be set, such as adding a grammar evaluation subnetwork.

[0104] In terms of model training, more than 100,000 training data are used for training, including about 100,000 training data and 5,000 verification data. The specific training parameters are as follows:

[0105] Batch size: 32.

[0106] Learning rate: 1e-4.

[0107] Learning rate scheduling: linear decay with warm-up, decay rate 0.05, warm-up steps 1000.

[0108] epoch:2。

[0109] Optimizer: AdamW, β1 is set to 0.9, β is set to 0.95, and weight decay coefficient is 0.01.

[0110] Number of verification steps: 1000.

[0111] After obtaining the trained multi-dimensional scoring model, the coding sequence and speech feature vector sequence are input into the pre-trained multi-dimensional scoring model to obtain the evaluation scores of each dimension of the speech to be evaluated, including the word pronunciation score, fluency score, content score and rhythm score of the speech to be evaluated.

[0112] The multi-dimensional scoring model provided in the embodiment of the present application can obtain the evaluation scores of the speech to be evaluated in various dimensions such as word pronunciation, fluency, content and rhythm without inputting standard text, which is convenient for analyzing the mastery of English listening and speaking of candidates from different angles.

[0113] S105 performs weighted averaging of the evaluation scores of each dimension according to the predefined weights of each dimension to obtain a comprehensive score of the speech to be evaluated.

[0114] In one embodiment, the evaluation scores of each dimension are weighted averaged according to the predefined weights of each dimension to obtain a comprehensive score. In this example, the pronunciation of each word is evaluated with equal weight to calculate the pronunciation of the word of the speech. Then, the word pronunciation is weighted with 0.4 weight, fluency with 0.2 weight, content with 0.3 weight, and rhythm with 0.1 weight to calculate the comprehensive score.

[0115] The specific values ​​of the weights in this application can be adjusted according to the actual application scenario, and the embodiments of this disclosure do not make specific limitations.

[0116] The method provided in the embodiment of the present application only needs to input the listening material text, the question text and the speech to be evaluated into a multi-dimensional scoring model to obtain the scores of each dimension and the comprehensive score of the speech to be evaluated. There is no need to obtain a standard text. Even in the absence of a standard text, the speech can be accurately evaluated. The evaluation problem of open-ended questions can be solved. The model can infer the content that the speech to be evaluated wants to express based on the listening material text and the question text, and then combine the content and pronunciation to give an accurate score. The method has a wider scope of application and can avoid the problem of recognition errors in automatic speech recognition. The method of the present application has higher accuracy and strong robustness.

[0117] According to another aspect of the embodiment of the present application, an English listening and speaking assessment device for implementing the above-mentioned English listening and speaking assessment method is also provided. Figure 3 As shown, the device comprises:

[0118] The acquisition module 301 is used to acquire the listening material text, the question text and the speech to be evaluated;

[0119] A text encoding module 302 is used to serialize the listening material text and the question text, and encode the serialized listening material text and the question text to obtain a coding sequence;

[0120] The speech encoding module 303 is used to encode the speech to be evaluated to obtain a speech feature vector sequence;

[0121] The dimension evaluation module 304 is used to input the coding sequence and the speech feature vector sequence into the pre-trained multi-dimensional scoring model to obtain the evaluation score of each dimension of the speech to be evaluated;

[0122] The comprehensive evaluation module 305 is used to perform weighted average of the evaluation scores of each dimension according to the predefined weights of each dimension to obtain a comprehensive score of the speech to be evaluated.

[0123] It should be noted that the English listening and speaking assessment device provided in the above embodiment only uses the division of the above functional modules as an example when executing the English listening and speaking assessment method. In actual applications, the above functional distribution can be completed by different functional modules as needed, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above. In addition, the English listening and speaking assessment device provided in the above embodiment and the English listening and speaking assessment method embodiment belong to the same concept, and the implementation process thereof is detailed in the method embodiment, which will not be repeated here.

[0124] According to another aspect of the embodiments of the present application, an electronic device corresponding to the English listening and speaking assessment method provided in the above embodiments is also provided to execute the above English listening and speaking assessment method.

[0125] Please refer to Figure 4 , which shows a schematic diagram of an electronic device provided by some embodiments of the present application. Figure 4 As shown, the electronic device includes: a processor 400, a memory 401, a bus 402 and a communication interface 403, and the processor 400, the communication interface 403 and the memory 401 are connected via the bus 402; the memory 401 stores a computer program that can be run on the processor 400, and when the processor 400 runs the computer program, the English listening and speaking assessment method provided in any of the aforementioned embodiments of the present application is executed.

[0126] The memory 401 may include a high-speed random access memory (RAM), and may also include a non-volatile memory, such as at least one disk memory. The communication connection between the system network element and at least one other network element is realized through at least one communication interface 403 (which may be wired or wireless), and the Internet, wide area network, local area network, metropolitan area network, etc. may be used.

[0127] The bus 402 may be an ISA bus, a PCI bus, or an EISA bus, etc. The bus may be divided into an address bus, a data bus, a control bus, etc. The memory 401 is used to store programs, and the processor 400 executes the programs after receiving the execution instructions. The English listening and speaking assessment method disclosed in any of the embodiments of the present application may be applied to the processor 400, or implemented by the processor 400.

[0128] The processor 400 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the hardware integrated logic circuit or software instructions in the processor 400. The above processor 400 can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a readily available programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in the embodiments of the present application can be implemented or executed. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. The steps of the method disclosed in the embodiments of the present application can be directly embodied as a hardware decoding processor to be executed, or the hardware and software modules in the decoding processor can be executed. The software module can be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory 401, and the processor 400 reads the information in the memory 401 and completes the steps of the above method in combination with its hardware.

[0129] The electronic device provided in the embodiment of the present application and the English listening and speaking assessment method provided in the embodiment of the present application are based on the same inventive concept and have the same beneficial effects as the methods adopted, operated or implemented therein.

[0130] According to another aspect of the embodiments of the present application, a computer-readable storage medium corresponding to the English listening and speaking assessment method provided in the aforementioned embodiments is also provided, on which a computer program (i.e., a program product) is stored. When the computer program is executed by a processor, it will execute the English listening and speaking assessment method provided in any of the aforementioned embodiments.

[0131] It should be noted that examples of computer-readable storage media may also include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other optical or magnetic storage media, which are not listed here one by one.

[0132] The computer-readable storage medium provided in the above-mentioned embodiments of the present application and the English listening and speaking assessment method provided in the embodiments of the present application are based on the same inventive concept and have the same beneficial effects as the methods adopted, run or implemented by the application programs stored therein.

[0133] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0134] The above embodiments only express several implementation methods of the present invention, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present invention. It should be pointed out that, for a person of ordinary skill in the art, several modifications and improvements can be made without departing from the concept of the present invention, and these all belong to the protection scope of the present invention. Therefore, the protection scope of the present invention patent shall be subject to the attached claims.

Claims

1. An English listening and speaking assessment method, characterized in that: include: Obtain the listening material text, question text and the audio to be evaluated; Serializing the listening material text and the question text, and encoding the serialized listening material text and the question text to obtain a coding sequence; Encoding the speech to be evaluated to obtain a speech feature vector sequence; The coding sequence and the speech feature vector sequence are input into a pre-trained multi-dimensional scoring model to obtain the evaluation scores of each dimension of the speech to be evaluated; the multi-dimensional scoring model includes: The text feature network consists of a word embedding layer, which receives the encoding sequence and outputs a sequence of text feature vectors; The acoustic feature network is composed of multiple linear layers and activation functions, which is used to receive a speech feature vector sequence and output an aligned speech feature vector sequence; A fusion feature network, composed of a position embedding layer and a feature identifier embedding layer, is used to receive the text feature vector sequence and the aligned speech feature vector sequence, and output a fusion feature vector sequence; An encoding network, composed of multiple transformer layers with an autoregressive attention mechanism, is used to receive the fused feature vector sequence and output a latent vector sequence; A language network, composed of a linear layer and a softmax activation function, is used to receive the latent vector sequence and output a predicted coding sequence; A scoring network, composed of multiple linear layers and activation functions, is used to receive the latent vector sequence and output the predicted score of each evaluation dimension; According to the predefined weights of each dimension, the evaluation scores of each dimension are weighted averaged to obtain a comprehensive score of the speech to be evaluated.

2. The English listening and speaking assessment method according to claim 1, characterized in that: The listening material text and the question text are serialized, and the serialized listening material text and the question text are encoded to obtain an encoding sequence, including: Using a preset template to splice the listening material text and the question text to obtain serialized listening material text and question text; The serialized listening material text and question text are input into a word segmenter for encoding to obtain an encoding sequence.

3. The English listening and speaking assessment method according to claim 1, characterized in that: The speech to be evaluated is encoded to obtain a speech feature vector sequence, including: Calculating the original Mel-spectrogram feature vector sequence of the speech to be evaluated; Preprocessing the original Mel spectrum feature vector sequence to obtain a Mel spectrum feature vector sequence; The Mel-spectrogram feature vector sequence is input into an acoustic encoder to obtain a speech feature vector sequence.

4. The English listening and speaking assessment method according to claim 1, characterized in that: The scoring network comprises: A word pronunciation evaluation network, used for receiving the latent vector sequence and outputting a predicted pronunciation score sequence; A fluency evaluation network, used for receiving the latent vector sequence and outputting a fluency evaluation score; A content evaluation network, used to receive the latent vector sequence and output a content evaluation score; The rhythm evaluation network is used to receive the latent vector sequence and output a rhythm evaluation score.

5. The English listening and speaking assessment method according to claim 1, characterized in that: Before inputting the coding sequence and the speech feature vector sequence into the pre-trained multi-dimensional scoring model, the method further includes: Acquire training data, wherein the training data includes a listening material text, a question text, an evaluation voice, an evaluation voice target text, and a normalized score of each dimension; Randomly divide the training data into a training data set and a validation data set; Based on the training data set and the validation data set, the multi-dimensional scoring model is trained using the gradient descent method.

6. The English listening and speaking assessment method according to claim 5, characterized in that: The loss function of the multi-dimensional scoring model is: in, represents the preset hyperparameter, M represents the number of scoring dimensions, and m represents the current scoring dimension. Represents the prediction results of each dimension output by the scoring network, represents the normalized score of each dimension, N represents the length of the target text encoding sequence, V represents the vocabulary, represents the true probability value of the jth word at the i-th position, represents the predicted probability value of the jth word at the i-th position.

7. An English listening and speaking assessment device, characterized in that: include: The acquisition module is used to obtain the listening material text, question text and the voice to be evaluated; A text encoding module, used for serializing the listening material text and the question text, and encoding the serialized listening material text and the question text to obtain a coding sequence; A speech encoding module, used for encoding the speech to be evaluated to obtain a speech feature vector sequence; Each dimension evaluation module is used to input the coding sequence and the speech feature vector sequence into a pre-trained multi-dimensional scoring model to obtain the evaluation scores of each dimension of the speech to be evaluated; the multi-dimensional scoring model includes: The text feature network consists of a word embedding layer, which receives the encoding sequence and outputs a sequence of text feature vectors; The acoustic feature network is composed of multiple linear layers and activation functions, which is used to receive a speech feature vector sequence and output an aligned speech feature vector sequence; A fusion feature network, composed of a position embedding layer and a feature identifier embedding layer, is used to receive the text feature vector sequence and the aligned speech feature vector sequence, and output a fusion feature vector sequence; An encoding network, composed of multiple transformer layers with an autoregressive attention mechanism, is used to receive the fused feature vector sequence and output a latent vector sequence; A language network, composed of a linear layer and a softmax activation function, is used to receive the latent vector sequence and output a predicted coding sequence; A scoring network, composed of multiple linear layers and activation functions, is used to receive the latent vector sequence and output the predicted score of each evaluation dimension; The comprehensive evaluation module is used to perform weighted averaging of the evaluation scores of each dimension according to the pre-defined weights of each dimension to obtain a comprehensive score of the speech to be evaluated.

8. An electronic device, characterized in that: It comprises a processor and a memory storing program instructions, wherein the processor is configured to execute an English listening and speaking assessment method as claimed in any one of claims 1 to 6 when executing the program instructions.

9. A computer-readable medium, characterized in that Computer-readable instructions are stored thereon, and the computer-readable instructions are executed by a processor to implement an English listening and speaking assessment method as described in any one of claims 1 to 6.