Multi-modal interpretation training evaluation method and evaluation device

Abstract

The invention relates to the technical field of computers, provides a voice recognition scheme, and particularly relates to a multi-modal interpretation training evaluation method and evaluation device. According to the embodiment provided by the invention, the translation semantic correctness and the emotion tendency of the text data are determined by performing standard text matching and emotion tendency score calculation on the text data, the multiple features are extracted from the audio data, and the translation fluency of the audio data is determined based on the expressions of the multiple features; and determining a corresponding semantic score, an emotion score and a fluency score according to the translation semantic correctness, the emotion tendency and the translation fluency, and evaluating the translation behavior of the trainee according to the semantic score, the emotion score and the fluency score. According to the method provided by the embodiment of the invention, the translation process and result can be automatically evaluated based on a natural language processing technology and a voice processing technology, and compared with the prior art, the method can comprehensively evaluate the result from three aspects of accurate semantic expression, emotional tendency and fluency expression.

Description

Technical Field

[0001] This application relates to the field of computer technology, and is a speech recognition solution, specifically relating to a multi-modal interpretation training evaluation method and an evaluation device. Background Art

[0002] Currently, in the field of translation teaching, especially in the field of interpretation teaching, the error correction of the translation results made by learners and the scoring of training results are mainly carried out through one-on-one guidance by teachers, and rely on the experience of teachers for error correction and scoring. Although in the current translation field, an interpretation ability characteristic scale is established to evaluate as objectively as possible, the effect that the characteristic scale can achieve also requires teachers to strictly abide by the usage rules and evaluation criteria. Therefore, the result evaluation through the characteristic scale still cannot avoid subjective judgment and will cause a relatively high time cost. Summary of the Invention

[0003] In view of the above technical problems, the present invention provides a multi-modal interpretation training evaluation method and an evaluation device. By obtaining the expression results of the text features and speech features of the interpretation speech information, and combining the expression results of the two features to determine the final evaluation result, it can comprehensively and automatically evaluate the result from three aspects of accurate semantic expression, emotional tendency and fluency expression compared with the prior art. To achieve the above purpose, the technical solutions adopted by the present invention are as follows: In a first aspect, a multi-modal interpretation training evaluation method is provided. The method includes: generating a corresponding source text to be translated and a corresponding historical corpus according to a training task, and sending the source text to be translated and receiving speech data to be evaluated that matches the source text to be translated; the historical corpus includes historical corpus data corresponding to the training task generated during the historical interpretation training process, and also includes real-time corpus data generated in real time as the training task progresses; extracting the text features and speech features of the speech data to form corresponding text data and audio data; the extraction of the text features includes: extracting the initial text features of the speech data, and performing error correction and update processing on the initial text features based on the historical corpus database to obtain target text features, and forming a target text based on the target text features as the text data; determining a semantic score based on the matching degree between the text data and the standard text, determining a fluency score based on the feature distribution result corresponding to the audio data, and determining an emotional score based on the emotional similarity between the text data and the standard text; and determining a scoring result of the speech data to be evaluated based on the semantic score, the fluency score and the emotional score.

[0004] In some specific implementation manners, extracting text features of the speech data to form corresponding text data includes: extracting initial text features of the speech data to form a corresponding initial text sequence, updating the initial text features according to a similarity distribution between the initial text features and corpus data in the historical corpus, generating target text features, and generating corresponding text data based on the target text features.

[0005] In some specific implementation manners, updating the initial text features according to a similarity distribution between the initial text features and corpus data in the historical corpus includes: obtaining similarities between the initial text features and multiple pieces of the historical corpus data and multiple pieces of the real-time corpus data respectively, screening the multiple similarities to determine an initial similarity distribution result, and updating the initial similarity distribution result based on update weights of the historical corpus data and the real-time corpus data.

[0006] In some specific implementation manners, the method further includes: obtaining a query vector corresponding to the initial text features, and key-value pairs corresponding to semantic features in each of multiple pieces of the historical corpus data and multiple pieces of the real-time corpus data, and returning at least three closest key-value pairs based on each token in each query vector.

[0007] In some specific implementation manners, determining a matching degree between the text data and the standard text includes: obtaining a matching ratio of segmented words that match between the text data and the standard text, and determining the matching degree and a corresponding semantic score based on the matching ratio.

[0008] In some specific implementation manners, obtaining a matching ratio of segmented words that match between the text data and the standard text includes: obtaining an alignment vector between the text data and the standard text, fusing the alignment vector with a word embedding vector of the text data, and performing pooling on a fusion result to determine segmented words in the text data that match the standard text, and counting a proportion of the matching segmented words.

[0009] In some specific implementation manners, obtaining an alignment vector between the text data and the standard text includes: performing block processing on the text data and the standard text, determining context information corresponding to segmented words within a block and upper and lower block information between blocks, and obtaining the alignment vector between the text data and the standard text based on the context information corresponding to the segmented words and the upper and lower block information corresponding between blocks.

[0010] In some specific implementation manners, the speech features include energy features, fundamental frequency change features, and syllable boundary clarity features; extracting the speech features of the speech data includes: sequentially obtaining the energy distribution state, the fundamental frequency change distribution state, and the syllable boundary clarity distribution state corresponding to multiple frames of data in the speech data.

[0011] In some specific implementation manners, determining a fluency score based on the feature distribution result corresponding to the audio data includes: respectively obtaining sub-scores corresponding to the energy distribution state, the fundamental frequency change distribution state, and the syllable boundary clarity distribution state, and performing weighted summation on the sub-scores based on corresponding weights to obtain a total score as the fluency score.

[0012] In a second aspect, there is provided a multi-modal interpretation training evaluation device, the device including: a data acquisition module, configured to generate a corresponding source text to be translated and a corresponding historical corpus according to a training task, and send the source text to be translated and receive the speech data to be evaluated that matches the source text to be translated; a data processing module, configured to extract text features and speech features of the speech data to form corresponding text data and audio data; the extraction of the text features includes: extracting initial text features of the speech data, and performing error correction and update processing on the initial text features based on a historical corpus database to obtain target text features, and forming a target text based on the target text features as the text data; an evaluation module, configured to determine a semantic score based on the matching degree between the text data and a standard text, determine a fluency score based on the feature distribution result corresponding to the audio data, determine an emotion score based on the emotional similarity between the text data and the standard text; and determine a scoring result of the speech data to be evaluated based on the semantic score, the fluency score, and the emotion score.

[0013] In the technical solution provided by the embodiments of the present application, by obtaining the text data and audio data of the speech data of a trainer, and respectively performing standard text matching and emotional tendency score calculation on the text data to determine the translation semantic correctness and emotional tendency, extracting multiple features from the audio data and determining its translation fluency based on the expressions of the multiple features, and determining corresponding semantic scores, emotion scores, and fluency scores according to the translation semantic correctness, emotional tendency, and translation fluency, and evaluating the translation behavior of the trainer according to the semantic score, emotion score, and fluency score. The method provided by the embodiments of the present application can realize the automatic evaluation of the translation process and result based on natural language processing technology and speech processing technology, and can comprehensively evaluate the result from three aspects of semantic accurate expression, emotional tendency, and fluency expression compared with the prior art. Description of the Drawings

[0014] To more clearly illustrate the technical solutions in the embodiments of the present application, the following will briefly introduce the drawings required for the description of the embodiments. Obviously, the drawings in the following description are only some embodiments of the present application. For those of ordinary skill in the art, without creative efforts, other drawings can be obtained based on these drawings.

[0015] The methods, systems, and / or programs in the drawings will be further described according to exemplary embodiments. These exemplary embodiments will be described in detail with reference to the drawings. These exemplary embodiments are non-limiting exemplary embodiments, where the example numbers represent similar mechanisms in the various views of the drawings.

[0016] Figure 1 It is a schematic structural diagram of the teaching system provided by the embodiments of the present application.

[0017] Figure 2 It is a schematic structural diagram of the server provided by the embodiments of the present application.

[0018] Figure 3 It is a schematic flowchart of the training evaluation method provided by the embodiments of the present application.

[0019] Figure 4 It is a schematic structural diagram of the evaluation device provided by the embodiments of the present application.

[0020] Figure 5 It is a schematic structural diagram of the terminal device provided by the embodiments of the present application. Detailed implementation manners

[0021] To better understand the above technical solutions, the following will make a detailed description of the technical solutions of the present application through the drawings and specific embodiments. It should be understood that the embodiments of the present application and the specific features in the embodiments are detailed descriptions of the technical solutions of the present application, rather than limitations on the technical solutions of the present application. Without conflict, the technical features in the embodiments of the present application and the embodiments can be combined with each other.

[0022] In the following detailed description, many specific details are set forth by way of example in order to provide a thorough understanding of the relevant teachings. However, for those skilled in the art, it is obvious that the present application can be practiced without these details. In other instances, well-known methods, procedures, systems, components, and / or circuits have been described at a relatively high level without detail in order to avoid unnecessarily obscuring aspects of the present application.

[0023] In this application, a flowchart is used to illustrate the execution process performed by the system according to the embodiments of this application. It should be clearly understood that the execution process of the flowchart may not be executed in sequence. On the contrary, these execution processes may be executed in the reverse order or simultaneously. Additionally, at least one other execution process may be added to the flowchart. One or more execution processes may be deleted from the flowchart.

[0024] Before further elaborating on the embodiments of the present invention, the nouns and terms involved in the embodiments of the present invention are described. The nouns and terms involved in the embodiments of the present invention are applicable to the following explanations.

[0025] (1) Responsive to, which is used to indicate the conditions or states on which the executed operations depend. When the dependent conditions or states are met, one or more of the executed operations may be real-time or may have a set delay; without special instructions, there is no restriction on the execution order of the multiple executed operations.

[0026] (2) Based on, which is used to indicate the conditions or states on which the executed operations depend. When the dependent conditions or states are met, one or more of the executed operations may be real-time or may have a set delay; without special instructions, there is no restriction on the execution order of the multiple executed operations.

[0027] An embodiment of this application provides a multi-modal interpretation training evaluation method applied to a teaching system for automatically evaluating the results of the interpretation process of a trainee.

[0028] In this embodiment, for the overall structure of the teaching system 10, reference may be made to Figure 1 As shown, it includes at least a server 11, a student terminal 12, and a teacher terminal 13. Among them, the student terminal and the teacher terminal may be mobile terminals, including but not limited to smart phones, tablet computers, and other mobile devices capable of hosting application programs. In this embodiment, a smart phone is preferably selected. In this embodiment, for the structure of the server 11, reference may be made to Figure 2, a data processing unit 111 and a database 112 are provided, and content is displayed through a display interface on the student side and the teacher side respectively. In the interpretation training scenario of this system, the student side selects the interpretation project to be trained and the corresponding audio data. After being retrieved from the database, it is sent to the terminal device, and the voice of the trainer is obtained and recorded through the voice recording hardware in the student side. The obtained voice data is transmitted to the data processing unit in real time, and the interpretation evaluation result of the trainer is obtained through the processing of the interpretation training evaluation method configured in the data processing unit; and the interpretation evaluation result and the recorded interpretation voice file are stored in the database and the corresponding teacher side. Among them, the interpretation evaluation result includes the scoring result, and also includes the text display of the interpretation result and the display of important marks. The teacher side is used to manually update the interpretation evaluation result and synchronize it to the student side.

[0029] In this embodiment, the database includes a text database, a historical corpus database, and an emotional corpus database. Among them, the text database stores the original text audio data related to the training task. Different training tasks are configured with corresponding original text audio data, which are retrieved according to the training task. The historical corpus database is the historical corpus data corresponding to this training task, and also includes the real-time corpus data generated in real time as the training task progresses. The emotional corpus data stores the text features related to emotional words and the corresponding emotional polarity scores. For this emotional corpus database, an existing and open-source database in the prior art can be used, such as the Hownet emotional dictionary, the GeneralInquirer emotional dictionary, etc.

[0030] In the prior art, for online teaching, especially for interpretation teaching, due to the scarcity of teacher resources, there are high scenario limitations and technical limitations during the training of students. Teachers cannot quickly give timely feedback and scoring evaluations on the training situation of students, resulting in higher training and teaching costs. With the development of computer technology, especially the development of speech recognition technology, the speech-text conversion of the interpretation results of trainers can be based on computer speech recognition technology.

[0031] However, it should be noted that due to the characteristics of translation, especially interpretation, the current technical solutions can only convert the voice of the interpreter's result into text, and cannot judge the interpretation result. Especially in the method of automatic oral evaluation, the evaluation method at the voice level is basically mature, such as whether the pronunciation of the trainer is accurate. However, there are not many evaluation methods at the content level, such as whether the content expressed by the trainer is complete, whether the words used are accurate, whether the sentences are meaningful, and whether the emotional expression of the sentences is accurate. Moreover, for the interpretation scenario, different types of languages, different pronunciation habits and accents of different trainers will cause problems of recognition errors. Therefore, a method for evaluation is provided to improve the recognition of the interpreter's result of the trainer in the interpretation scenario and evaluate and score the interpretation result according to the judgment criteria at the semantic level.

[0032] In order to achieve the above technical effects, an interpretation training evaluation method is configured in the data processing unit for the teaching system in this embodiment. By determining the real pronunciation result based on the context association and database association of the obtained real-time interpretation data of the trainer to generate interpretation text data, the recognition accuracy is improved. The semantic relevance is determined by comparing the context semantics with the standard translation text, and the fluency is determined according to its voice spectrum characteristics. Based on the multi-modal data fusion method, the interpretation result and the interpretation process of the trainer are comprehensively scored.

[0033] Specifically, for the interpretation training evaluation method in this embodiment, please refer to FIG. 2, which includes the following steps: Step S31. Generate the corresponding original text to be translated and the corresponding historical corpus according to the training task, and send the original text to be translated and receive the voice data to be evaluated that matches the original text to be translated.

[0034] In this embodiment, the training task is selected and obtained by the trainer at the student side. A plurality of training tasks are configured on the display interface of the student side. The target training task is determined by selection at the student side, and this training task is sent to the data processing unit. The data processing unit retrieves the corresponding original text to be translated and the historical corpus from the database based on the link or label corresponding to this training task.

[0035] Among them, the original text to be translated will be played to the trainer through the voice hardware of the terminal device at the student side by the trainer's selection. The historical corpus is the historical corpus data generated during the historical interpretation training process related to this training task, and also includes the real-time corpus data generated in real time based on the development of this training task. The training task is constructed by multiple labels. For example, for a training task, a scenario label can be selected, and the original text to be translated corresponding to the above label is generated from the database. Among them, the historical corpus database is the historical process data corresponding to this label, that is, the training task, and is used for subsequent speech recognition and text conversion.

[0036] Specifically, when the trainer's speech data is obtained, real-time text conversion is performed through automatic speech recognition technology. However, in the actual use process, due to factors such as the complexity and diversity of speech, the speaker's accent, speech rate changes, and background noise interference, it is difficult to accurately recognize words with similar pronunciations, resulting in semantic understanding deviations. For example, for "Zhejiang" based on the standard pronunciation is "zhejiang", while the recognition result may be "zhejiang", and for a standard speech "What's the weather like in Taizhou, Zhejiang today" based on automatic speech recognition will be distorted into "What's the weather like in Zhejiang Taizhou today". For the English speech data "in this monotonous life of mine that was a pleasant event", due to the similar pronunciations of "mind" and "mine", especially for continuous pronunciations, "mine" will be incorrectly recognized as "mind". Therefore, in order to avoid the impact of recognition errors on the subsequent evaluation accuracy, in this embodiment, it is necessary to correct the errors in the speech-text conversion to obtain the corrected recognition result.

[0037] However, it should be noted that for the speech recognition error correction process, not only the correctness of individual words needs to be considered, but also the text within the entire document needs to be considered. This kind of error correction not only needs to consider the correctness of individual sentences, but also needs to consider other sentences in the entire document, so as to improve the overall accuracy and readability of the text. Specifically, a larger range of context-based connections are needed for error correction. Not only the correctness of error correction vocabulary and grammar needs to be considered, but also the semantic relationship between sentences in the document and the consistency of the overall document structure need to be determined. Generally, the context attention weight corresponding to each word is usually obtained, and whether the word is incorrect and error correction processing are performed according to its attention weight. However, in this embodiment, in order to make the context semantics more rich, by configuring the historical corpus database, the context attention weight between the currently converted word and the corpus data in the historical corpus database is determined, so as to improve the error correction ability.

[0038] In this embodiment, the historical corpus database has been configured before the task is executed, and comprehensive information of the entire document or dataset can be accessed when the task is executed. Moreover, as the training task progresses, the process data generated in each training task can be accumulated, and these accumulated data are the basis for storage. For the historical corpus database in this embodiment, the data processing unit can access the entire document, and collect real-time corpus data and related context information from the data, including all sentences before and after the current sentence. This enables a better understanding of the semantic coherence and context relationship within the document during the speech recognition process, thereby improving the ability to process and analyze information in the document.

[0039] Step S32. Extract the text features and speech features of the speech data to form corresponding text data and audio data; the extraction of the text features includes: extracting the initial text features of the speech data, and performing error correction and update processing on the initial text features based on the historical corpus database to obtain target text features, and forming a target text based on the target text features as the text data.

[0040] Extract the text features and speech features of the obtained speech data of the trainer, which are used as the input data for subsequent training evaluation. Among them, for the text feature extraction process, first perform text conversion to form initial text features, and use the historical corpus database in step S31 to perform error correction and update processing on the initial text features during the text conversion process to obtain target text features, thereby forming a target text, and the formed target text is the text data.

[0041] Among them, for the above text conversion and text update process, first convert the speech into text through the ASR model in the prior art. The structure of the ASR model can adopt the prior art, and through the ASR model, the input speech data can be converted into real-time text to obtain the initial real-time text of the speech data, and the initial real-time text is sorted into an initial real-time text sequence based on the time series ; where represents the i-th word in the sequence.

[0042] Perform error correction processing on the obtained initial real-time text sequence based on end-to-end error correction and the historical corpus database to obtain the target text.

[0043] Among them, the processing logic of the error correction process is to determine the similarity distribution between the initial text features corresponding to the initial text and the expected data in the historical corpus, and update the initial text features according to the similarity distribution to generate target text features, and generate corresponding text data based on the target text features. Among them, for the determination of the similarity distribution in this embodiment, it is determined based on the attention score, that is, by determining the attention score of the initial text features in the input initial text sequence relative to each corpus data in the historical corpus database, and determining whether the current initial text feature is incorrect according to the corresponding attention score, and taking the text with the highest attention score as the target text feature.

[0044] In this embodiment, the error correction of the initial text not only considers individual sentences but also other sentences in the entire document to improve the overall accuracy and readability of the text. Specifically, for the error correction in this embodiment, a greater range of context awareness and the ability to handle more complex errors are required. Therefore, for the error correction process in this embodiment, a wider range of context information needs to be utilized, including not only the correctness of vocabulary and grammar but also the semantic relationships between sentences in the document and the consistency of the overall document structure.

[0045] Therefore, for the update in this embodiment, that is, the error correction process, the comparison information sources include not only the context relationships in the current initial text sequence but also the entire dataset including the current entire document, namely the historical corpus data.

[0046] Specifically, by obtaining the first type of similarity and the second type of similarity between the initial text features and multiple historical corpus data and multiple real-time corpus data in the corpus database, and screening the multiple first type of similarities and the second type of similarities to determine the corresponding initial first type of similarity and initial second type of similarity, and updating the initial first type of similarity and initial second type of similarity based on the update weights corresponding to the first type of similarity and the second type of similarity, taking the result with the highest similarity as the target similarity and updating and correcting the initial text features with the corresponding text features. In this embodiment, the update weights for historical corpus data and real-time corpus data are 0.4 and 0.6 respectively. When the similarities between the historical corpus data and the real-time corpus data and the initial text are the same and they have different text features, the real-time corpus data is preferably selected. That is, context information is preferentially selected as the semantic perception guidance.

[0047] Specifically, the above processing process is implemented by a processing model provided in this embodiment. This model includes an encoding and decoding unit. The encoder is constructed based on Transformer and a multi-head attention mechanism is configured in this encoder to obtain the first attention matrix corresponding to each word in the current input initial text sequence. It can be understood that the encoder is used to obtain the context semantic information in the current sequence corresponding to each word in the input initial text sequence. And the decoder is used to obtain the context semantic information in the historical corpus data and the real-time corpus data corresponding to each word in the initial text sequence. Among them, the context semantic information is output based on the attention matrix, and the attention matrix is the final output similarity distribution.

[0048] It should be noted that the context semantic information corresponding to the real-time corpus data in the historical corpus is different from the context semantic information in the encoder. The context semantic information in the historical corpus includes the context semantic information in the current text sequence and also the context semantic information corresponding to the processed text sequences; while the context semantic information in the encoder is only the context semantic information of the current text sequence.

[0049] Among them, the decoder is also constructed using the Transformer structure. And because the decoder is used to obtain the attention matrix corresponding to the current initial text, the historical corpus data, and the real-time corpus data, and the amount of data in the historical corpus is large. In order to quickly process the text features, in this embodiment, a k-nearest neighbor search mechanism is configured during the calculation in the decoder to reduce the search cost.

[0050] Specifically, for the encoder, the initial text sequence is first processed based on a tokenizer and converted into a series of tokens, where the tokens may be words, sub-words, or characters. Then, each token is mapped to a unique ID, and these IDs form a sequence. The token IDs are converted into corresponding embedding vectors, which include word embeddings and positional encodings to provide the model with information about the word order. The embedded sequence passes through the encoder layer to generate query vectors, key vectors, and value vectors, and through multiple different linear mappings based on multi-head attention, the query vectors, key vectors, and value vectors are mapped into different dimensional spaces to form different groups of query vectors, key vectors, and value vectors. The similarity between elements is obtained through the dot product of the query vectors and key vectors in the query vector group and key vector group, and is normalized by the softmax function to obtain the output weights; finally, the weighted sum of all the value vectors output for each text is obtained to get the output vector, that is, the first output attention matrix.

[0051] And, the decoder is also configured with a Transformer module. In the decoder, it is used to receive each token ID in the query vector obtained from the encoder, and through similarity matching, the k closest key-value pairs are determined according to each token ID to find the corpus data closest to the historical corpus.

[0052] In this embodiment, the screening of the key-value pairs is based on k1 key-value pairs returned by the k-nearest neighbor search operation for each token ID relative to the historical corpus data; and k2 key-value pairs feedback by the k-nearest neighbor search operation for each token ID relative to the real-time corpus data. Then, the similarities of the k1 key-value pairs and the k2 key-value pairs are updated respectively by the first type of weight and the second type of weight to obtain the similarities corresponding to the above key-value pairs, and the top k key-value pairs are screened. Among them, k1, k2, and k are all 3.

[0053] Through the above screening, each query vector and its corresponding three key-value pairs can be obtained. Then the dot product between each query vector and its retrieved key is calculated to obtain the attention matrix. Then the weighted sum of the attention matrix and the retrieved value is calculated to finally form the second attention matrix of the decoder.

[0054] The first attention matrix and the second attention matrix obtained after the above processing are weightedly merged through the gated hyperparameter g to balance the local context information and the external context information to obtain the target attention matrix, and the input speech data is sorted and organized based on the target attention matrix to obtain the text data corresponding to the final target text features.

[0055] The embodiment of the present application combines the contextual semantics with the semantics corresponding to the constructed corpus to correct the recognition errors in the current converted text to obtain text data that conforms to the contextual semantics, thereby improving the accuracy of the speech-to-speech converted text and thus improving the accuracy of subsequent evaluations.

[0056] From the above, it can be understood that the text feature in this embodiment refers to the meaning that can be used to represent the text, while the speech feature in this embodiment refers to the meaning that can be used to represent the speech. Because speech has various types and forms in feature representation, the speech feature in this embodiment is used to represent the change of the current speech and to perform fluency evaluation later.

[0057] Therefore, in order for the subsequent speech features to intuitively reflect the expression of the sound in terms of fluency, in this embodiment, energy features, fundamental frequency change features, and syllable boundary clarity features are used for the speech features.

[0058] Specifically, for the extraction of the above three features, it is first necessary to perform window framing processing on the speech data to obtain multiple frame signals.

[0059] The energy feature is obtained by calculating the sum of the squares of the amplitudes corresponding to each frame signal, and combining the calculation results of all frame signals to determine the energy distribution state corresponding to the speech data. This state is used to evaluate the dwell time and frequency in the speech data.

[0060] The acquisition of the fundamental frequency change feature is achieved by performing a fast Fourier transform on the framed signal, separating the discrete one-dimensional speech signal data more efficiently according to different frequency bands, obtaining the spectral representation form of the speech signal, then performing a vanishing point projection on the left and right windows after the fast Fourier transform, calculating the scaling coefficient, and using this scaling coefficient to characterize the change rate of the current frame. Among them, the vanishing point inner product refers to the process of performing projections on the -T0 and T0 moments using a finite number of points τ, and then calculating the dot product sum of the scaled points. It is to perform dot product summation on the data within the two windows after scaling according to a ratio, obtaining the compressed windows FL and FR or the compressed FR and FL. Based on this operation for all frame data, the scaling coefficients corresponding to all frames can be obtained, and the proportional relationship of the scaling coefficients between frames is used to characterize the fundamental frequency change feature between frames.

[0061] The acquisition of the syllable boundary clarity distribution state also performs a fast Fourier transform on the framed signal, then calculates the logarithmic energy spectrum through a Mel filter bank, extracts 12 - 13 dimensional MFCC features through discrete cosine transform and captures dynamic changes through first-order and second-order differences. Calculate the local mean and local mean of the MFCC sequence sliding window, determine the initial boundary when the local mean exceeds the preset threshold, and compare the difference between the local mean and variance corresponding to adjacent initial boundaries. Obtain the differences between adjacent initial boundaries in all frames of the speech data, and count the distribution of differences less than the preset difference threshold. This distribution is the syllable boundary clarity distribution state.

[0062] In the embodiment of the present application, the above processing process is used to obtain three characteristic states corresponding to the speech data, which are used to indicate the continuity, intonation variability, and clarity of the speech data, providing a relatively complete feature expression for subsequent evaluation.

[0063] Step S33. Determine the semantic score based on the matching degree between the text data and the standard text, determine the fluency score based on the characteristic distribution result corresponding to the audio data, and determine the emotion score based on the emotion similarity between the text data and the standard text; and determine the scoring result of the speech data to be evaluated based on the semantic score, the fluency score, and the emotion score.

[0064] In this embodiment, the determination of the evaluation result is implemented based on three dimensions: text accuracy, expression fluency, and emotion similarity. Among them, for text accuracy, the matching degree between the obtained text data and the standard text is used to determine the corresponding semantic score according to the matching degree; for expression fluency, it is determined by the distribution states of the multiple speech features obtained above; for the emotion score, it is determined by the emotion tendency similarity between the obtained text data and the standard text.

[0065] Among them, the matching degree and sentiment score are determined based on text data. The matching degree refers to the matching degree between the text data and the standard text, and the sentiment score refers to the difference between the text data and the sentiment score corresponding to the standard text.

[0066] Specifically, for the matching degree calculation and sentiment score determination in this embodiment, the text data and the standard text are first segmented into sentences. Since the translation logic in the interpretation process is based on the same sentence for translation, the relationship between sentences is clear and presented in a one-to-one correspondence in the final text presentation. Therefore, in this embodiment, the calculation of the text matching degree is implemented based on the matching degree calculation between sentences, and the sentiment scores corresponding to the text data and the standard text are also determined based on sentences.

[0067] Specifically, sentence pairs corresponding to the text data and the standard text are constructed, and the sentence pairs are divided into multiple blocks of the same length, and the corresponding matching degree and sentiment score are determined based on the blocks.

[0068] Among them, for the determination of the match, first, the local dependency relationship within each block is captured by applying intra-block attention within each block to obtain the meaning representation of the block. Then, the outputs of all blocks are processed through inter-block attention to obtain the global dependency relationship of the sentence. Finally, cross-attention is used to obtain the block-level interaction information of the sentence pair, where the block-level interaction information is used to represent the similarity probability distribution corresponding to each block between the two sentences, and the similarity of the corresponding sentence is determined by statistically analyzing this similarity distribution.

[0069] Among them, the similarity probability distribution refers to the distribution of multiple similarity probabilities corresponding to all blocks of the standard text for the current block of the text data. The calculation of the sentence similarity refers to obtaining the maximum similarity probability corresponding to each block of the text data and statistically analyzing the proportion of the blocks with a similarity probability greater than the preset similarity probability threshold in the sentence. For example, when a sentence of the text data has five blocks, the maximum similarity probabilities of the first, second, third, fourth, and fifth blocks are 75%, 60%, 99%, 85%, and 95% respectively. Among them, the similarity probability threshold is 80%, then the number of blocks greater than the preset similarity probability threshold is 3, accounting for 60% of the sentence, so the similarity of the current sentence is 60%.

[0070] For the text similarity, the similarities of the above multiple sentences are also averaged to obtain the final text similarity. And it should be noted that the blocks corresponding to a similarity lower than the similarity probability threshold are highlighted in the corresponding text data and displayed on the student side and the teacher side.

[0071] Among them, the inter-block attention in this embodiment is multi-dimensional additive attention, and its feature extraction and processing process can be implemented based on the attention mechanism. The difference is that for a block , each current word in the block is represented as q. First, the correlation between the current word vector and each intra-block vector is calculated for the current word to form a score vector, and the dimension of the score vector is the same as that of the word embedding vector. Subsequently, the probability distribution is obtained using softmax. Among them, the calculation process of the multi-dimensional additive attention is represented based on the following formula: , where W, , are parameter matrices, and b are bias terms, is the score vector of the current query vector and the intra-block context vector.

[0072] Specifically, in order to obtain the vector representation of the block, the intra-block global attention is used for the output of the intra-block attention within each block, and the calculation process is represented based on the following formula: ; where, is the output vector of the intra-block attention corresponding to the word in the block.

[0073] In order to obtain the global dependence representation of each block, the inter-block attention is used for each block, and its calculation process is the same as that of the intra-block attention. The difference is that the input object of the inter-block attention is the output result of each block for the German sentence pair. Finally, the gated mechanism is used to combine the outputs of the intra-block global attention and the inter-block attention to obtain the local and global dependence representation of the sentence pair based on the block, that is, the similarity distribution of the block of the final text data relative to the block of the standard text.

[0074] Through the above method, the text similarity of the text can be obtained, and the corresponding semantic score can be obtained based on the text similarity. Specifically, it is determined by the mapping score in the interval where the text similarity is located, and the mapping score of the interval is a pre-set scoring rule and will not be elaborated in this embodiment.

[0075] For the sentiment score, first, the target text features obtained in step S32 are combined based on sentence splitting to obtain the text feature sequence corresponding to each sentence of the text data; similarly, for the standard text, the text feature sequence corresponding to each sentence of the standard text is also determined through the feature extraction method in step S32. Then, the above two text feature sequences are respectively matched based on the sentiment corpus database to obtain the sentiment analysis score corresponding to each text feature sequence.

[0076] Specifically, the emotional corpus database can be understood as an emotional dictionary. By matching the similarity of text features, the emotional words in each text feature sequence are determined, and based on the emotional scores corresponding to each emotional word in the emotional dictionary, the emotional polarity scores in the corresponding text feature sequence and the sum of the emotional polarity scores corresponding to each sentence are determined. Finally, based on the sum of the emotional polarity scores corresponding to the sentences, the total emotional score corresponding to the entire text can be obtained.

[0077] Through the above, the first total emotional score and the second total emotional score corresponding to the text data and the standard text can be determined respectively. However, it should be noted that the above two total scores are used to represent the emotional tendencies of the two texts, while the emotional scores known in this embodiment represent the similarity between the text data and the standard text. Therefore, in this embodiment, the first total emotional score and the second total emotional score also need to be mapped in terms of emotional scores to realize the difference in emotional tendencies between the two texts.

[0078] Specifically, in this embodiment, a difference-score gradient mapping relationship is set. By obtaining the difference corresponding to the first total emotional score and the second total emotional score, and based on the difference-score gradient mapping relationship, the score corresponding to the current difference is determined, which is the emotional score referred to in this embodiment.

[0079] For the fluency score, by separately obtaining the sub-scores corresponding to the energy distribution state, fundamental frequency change distribution state, and syllable boundary clarity distribution state determined in step S32, and weighted summing the sub-scores based on the corresponding weights, the total score is the fluency score.

[0080] Specifically, the energy distribution state is used to evaluate the pause duration and pause frequency in the speech data. Therefore, the calculation of the sub-score corresponding to the energy distribution state is determined by obtaining the time length of the energy distribution state in the speech data that is less than the state threshold, that is, determining the energy distribution in the speech data and comparing it with the state threshold, and determining the accumulated time of the energy distribution corresponding to less than the state threshold as the time length. And in this embodiment, a mapping relationship between the time length and the score is configured, and the corresponding sub-score is determined based on the current time length.

[0081] The fundamental frequency change distribution state is used to characterize the changes in the speech intonation of the trainer. Generally, for interpretation, the intonation is in a stable state, and the fundamental frequency change distribution state can indirectly indicate the abnormal changes in intonation caused by uncertainty and disfluency in the trainer's interpretation process. For the determination of this sub-score, by determining the proportion of the number of fundamental frequency change rates exceeding the preset threshold in the overall audio data, and based on this proportion and the preset score mapping relationship, the corresponding sub-score is determined.

[0082] For the syllable boundary clarity distribution state used to characterize the interpreter's expression state during training, generally, the requirement for interpretation is clear and standard. Through the syllable boundary clarity distribution state, expression problems such as elision and unclearness in the trainer's audio data can be determined. Specifically, for the determination of this sub-score, by determining the proportional relationship of the differences less than the preset difference threshold in the distribution state, and based on this ratio and the preset score mapping relationship, the corresponding sub-score is determined.

[0083] For the above-mentioned sub-scores obtained, they are updated through their corresponding weights and summed to obtain the final fluency score. Among them, the weights corresponding to the above-mentioned sub-scores are 0.4, 0.3, and 0.3 respectively.

[0084] In this embodiment, through the above method, the semantic score and fluency score corresponding to the interpreter's result of the trainer can be determined, and the above two scores are used as the final scoring result. In another implementable manner, the above two scores can also be summed respectively to obtain the final score. How to calculate the scores will not be elaborated in this embodiment and can be set correspondingly according to the evaluation habits of the corresponding teacher.

[0085] It should be noted that for the abnormal words and audio features in the text data and audio data during the above processing process, corresponding highlighting is performed through the display terminals of the student side and the teacher side. Among them, the abnormal refers to the words with a similarity lower than the preset similarity threshold during text matching, and the audio data corresponding to the features in the audio features that are lower than the preset threshold or higher than the preset threshold.

[0086] The embodiment of the present application provides a multi-modal interpreter training evaluation method. By obtaining the text data and audio data of the trainer's speech data, and respectively performing standard text matching and emotional tendency score calculation on the text data to determine the translation semantic correctness and emotional tendency, extracting multiple features from the audio data and determining its translation fluency based on the expressions of multiple features, and determining the corresponding semantic score, emotional score, and fluency score according to the translation semantic correctness, emotional tendency, and translation fluency, and evaluating the trainer's translation behavior according to the semantic score, emotional score, and fluency score. The method provided by the embodiment of the present application can realize the automatic evaluation of the translation process and result based on natural language processing technology and speech processing technology, and can comprehensively evaluate the result from three aspects of semantic accurate expression, emotional tendency, and fluency expression compared with the prior art.

[0087] In another implementable manner, referring to Figure 4 , for this embodiment, a virtual device, namely the evaluation device 40, is also configured to execute the process of step S31 - step S33. The evaluation device 40 includes: A data acquisition module 41, configured to generate a corresponding original text to be translated and a corresponding historical corpus according to a training task, and send the original text to be translated and receive speech data to be evaluated that matches the original text to be translated; A data processing module 42, configured to extract the text features and speech features of the speech data to form corresponding text data and audio data; An evaluation module 43, configured to determine a semantic score based on the matching degree between the text data and a standard text, determine a fluency score based on the distribution result corresponding to the audio data, determine an emotion score based on the emotional similarity between the text data and the standard text, and determine a scoring result of the speech data to be evaluated based on the semantic score, the fluency score, and the emotion score.

[0088] Refer to Figure 5 Moreover, the above method can also be integrated into the provided terminal device 500. Due to relatively large differences that may occur in the device due to configuration or performance, it may include one or more processors 501 and a memory 502. One or more application programs or data may be stored in the memory 502. Among them, the memory 502 may be short-term storage or persistent storage. The application programs stored in the memory 502 may include one or more modules (not shown in the figure), and each module may include a series of computer-executable instructions in the terminal device. Further, the processor 501 may be set to communicate with the memory 502, and execute a series of computer-executable instructions in the memory 502 on the terminal device. The terminal device may also include one or more power supplies 503, one or more wired or wireless network interfaces 504, one or more input / output interfaces 505, one or more keyboards 506, etc.

[0089] In a specific embodiment, the terminal device includes a memory and one or more programs, where one or more programs are stored in the memory, and one or more programs may include one or more modules, and each module may include a series of computer-executable instructions in the terminal device, and is configured to be executed by one or more processors. The one or more programs include the following computer-executable instructions: Generate a corresponding original text to be translated and a corresponding historical corpus according to a training task, and send the original text to be translated and receive speech data to be evaluated that matches the original text to be translated; Extract the text features and speech features of the speech data to form corresponding text data and audio data; Determine a semantic score based on the matching degree between the text data and the standard text, determine a fluency score based on the distribution result corresponding to the audio data, determine an emotion score based on the emotional similarity between the text data and the standard text, and determine a scoring result of the speech data to be evaluated based on the semantic score, the fluency score, and the emotion score.

[0090] Optionally, the processor can execute various functions by running or executing software programs stored in the memory and calling data stored in the memory, such as executing the Figure 3 method shown above.

[0091] In a specific implementation, as an embodiment, the processor may include one or more microprocessors.

[0092] The memory is used to store software programs for executing the solutions of this application and is controlled by the processor for execution. The specific implementation manner can refer to the above method embodiments and will not be elaborated here.

[0093] In this application, "at least one" means one or more, and "a plurality" means two or more. "At least one of the following items (pieces)" or similar expressions refer to any combination of these items, including any combination of single items (pieces) or plural items (pieces). For example, at least one of a, b, or c can represent: a, b, c, a - b, a - c, b - c, or a - b - c, where a, b, c can be single or multiple.

[0094] It should be understood that in various embodiments of this application, the magnitudes of the sequence numbers of the above processes do not mean the order of execution. The execution order of each process should be determined according to its function and internal logic, and should not constitute any limitation to the implementation process of the embodiments of this application.

[0095] Those of ordinary skill in the art can realize that the units and algorithm steps of each example described in combination with the embodiments disclosed herein can be implemented by electronic hardware, or by a combination of computer software and electronic hardware. Whether these functions are executed in a hardware or software manner depends on the specific application and design constraints of the technical solution. Professional technicians can use different methods for each specific application to implement the described functions, but such implementation should not be considered to exceed the scope of this application.

[0096] Those skilled in the art can clearly understand that for the convenience and brevity of description, the specific working processes of the systems, devices, and units described above can refer to the corresponding processes in the foregoing method embodiments and will not be elaborated here.

[0097] When the above-mentioned functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art or a part of this technical solution can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in various embodiments of this application. The aforementioned storage medium includes: various media such as USB flash drives, mobile hard disks, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical discs that can store program codes.

[0098] As described above, the above are only the specific implementation manners of this application, but the protection scope of this application is not limited thereto. Any person skilled in the art within the technical scope disclosed in this application can easily think of changes or substitutions, which should all be covered by the protection scope of this application. Therefore, the protection scope of this application shall be subject to the protection scope of the claims.

Claims

1. A multi-modal interpretation training evaluation method, characterized in that, The method includes: Generating corresponding source text to be translated and a corresponding historical corpus according to a training task, and sending the source text to be translated and receiving speech data to be evaluated that matches the source text to be translated; the historical corpus includes historical corpus data corresponding to the training task generated during the historical interpretation training process, and also includes real-time corpus data generated in real time as the training task progresses; Extracting the text features and speech features of the speech data to form corresponding text data and audio data; the extraction of the text features includes: extracting the initial text features of the speech data, and performing error correction and update processing on the initial text features based on the historical corpus database to obtain target text features, and forming a target text as the text data based on the target text features; Determining a semantic score based on the matching degree between the text data and a standard text, determining a fluency score based on the feature distribution result corresponding to the audio data, and determining an emotion score based on the emotional similarity between the text data and the standard text; and determining a scoring result of the speech data to be evaluated based on the semantic score, the fluency score, and the emotion score.

2. The multi-modal interpretation training evaluation method according to claim 1, wherein Performing error correction and update on the initial text features based on the historical corpus database to obtain target text features, including: extracting the initial text features of the speech data and forming a corresponding initial text sequence, and updating the initial text features according to the similarity distribution between the initial text features and the corpus data in the historical corpus to generate target text features.

3. The multi-modal interpretation training evaluation method according to claim 2, characterized in that Updating the initial text features according to the similarity distribution between the initial text features and the corpus data in the historical corpus, including: obtaining the similarities between the initial text features and multiple pieces of the historical corpus data and multiple pieces of the real-time corpus data respectively, screening the multiple similarities to determine an initial similarity distribution result, and then updating the initial similarity distribution result based on the update weights of the historical corpus data and the real-time corpus data.

4. The multi-modal interpretation training evaluation method according to claim 3, wherein The method further includes: obtaining a query vector corresponding to the initial text features, and key-value pairs corresponding to each semantic feature in multiple pieces of the historical corpus data and multiple pieces of the real-time corpus data, and returning at least three closest key-value pairs based on each token in each query vector.

5. The multi-modal interpretation training evaluation method according to claim 4, characterized in that The determination of the matching degree between the text data and the standard text includes: obtaining the matching ratio of the segmented words that match between the text data and the standard text, and determining the matching degree and the corresponding semantic score based on the matching ratio.

6. The multi-modal interpretation training evaluation method according to claim 5, wherein Obtaining the matching ratio of the segmented words that match between the text data and the standard text includes: obtaining an alignment vector between the text data and the standard text, fusing the alignment vector with the word embedding vector of the text data, and performing pooling on the fusion result to determine the segmented words in the text data that match the standard text, and counting the proportion of the matching segmented words.

7. The multi-modal interpretation training evaluation method according to claim 6, wherein, Obtaining the alignment vector between the text data and the standard text includes: performing chunking processing on the text data and the standard text, determining the context information corresponding to the word segmentation within the chunks and the upper and lower chunk information between the chunks, and obtaining the alignment vector between the text data and the standard text based on the context information corresponding to the word segmentation and the upper and lower chunk information corresponding between the chunks.

8. The multi-modal interpretation training evaluation method according to claim 1, wherein The speech features include energy features, fundamental frequency change features, and syllable boundary clarity features; Extracting the speech features of the speech data includes: sequentially obtaining the energy distribution state, fundamental frequency change distribution state, and syllable boundary clarity distribution state corresponding to multiple frames of data in the speech data.

9. The multi-modal interpretation training evaluation method according to claim 8, characterized in that, Determining the fluency score based on the feature distribution result corresponding to the audio data includes: respectively obtaining the sub-scores corresponding to the energy distribution state, fundamental frequency change distribution state, and syllable boundary clarity distribution state, and performing weighted summation on the sub-scores based on the corresponding weights to obtain the total score as the fluency score.

10. A multi-modal interpretation training evaluation device, characterized in that, The device includes: A data acquisition module, configured to generate a corresponding source text to be translated and a corresponding historical corpus according to a training task, and send the source text to be translated and receive the speech data to be evaluated that matches the source text to be translated; A data processing module, configured to extract the text features and speech features of the speech data to form corresponding text data and audio data; the extraction of the text features includes: extracting the initial text features of the speech data, and performing error correction and update processing on the initial text features based on the historical corpus database to obtain target text features, and forming a target text based on the target text features as the text data; An evaluation module, configured to determine the semantic score based on the matching degree between the text data and the standard text, determine the fluency score based on the feature distribution result corresponding to the audio data, and determine the emotion score based on the emotion similarity between the text data and the standard text; and determine the scoring result of the speech data to be evaluated based on the semantic score, the fluency score, and the emotion score.

Images